API Feedback And Next Development Phase

[ghubstan]

I. Introduction

A pause in API work gives me a chance look at feedback from recent beta testing, and for a consensus among users and devs about the next development phase.

I think the broad goals are build out the API to expose the features available in the existing desktop application to the new gRPC CLI, and make those features available to new user interfaces and bots via gPRC. There has also been a lot of back and forth about one or more Bisq devs doing the work of building a new desktop UI and a web + mobile client, or not. Either way, the choice of using gRPC for client-server data transfer makes it possible for the API to serve clients written in a sufficient variety of programming languages with this caveat: gRPC uses the HTTP/2 transport protocol, which most browsers do not support. By default (I think), we made a decision to move away from HTTP/1.1 and supporting RESTful web clients in the near term. More will be said about this further down the document and in the ensuing discussion.

Another point to bring up early in the conversation is an agreement between @cbeams, @sqrrm, and myself: any new Bisq web client project should be scoped as a prototype, which means it will be small, experimental, and may be discarded in favor of one or more other prototype dev efforts. Chris Beams also suggested that in the short term, we should create a neatly packed UI prototype and demo it running in a local Umbrel installation; that may entice a skilled UI developer to commit to delivering a new web client.

Narrower subjects for discussion are outlined below. Technical issues are mixed in with non-technical, end user related topics. The technical discussions can be taken elsewhere, but I think some of them need to be mentioned.

Note: I try to organize what may be a many threaded discussion within a conventional outline format. The outline headers could be copied above your comments.

II. Bisq dev effort to build new UI(s) requires significant reorganization of existing code

A. Cons

1. No tangible benefit to end users

   I mention this early to bring attention to the fact that a significant amount of UI related development will not provide visible or financial benefits to end users, and may seem excessive in terms of BSQ compensation and time. But duplicating code from the desktop app in new client side modules will wind up costing more in the long run.

2. Compensating devs for code refactoring work hard to justify

   Bisq developers try to work on what the team deems valuable and necessary, often measured by what users can see and benefit from. Developers are also motivated to do work affording quick and visible benefits. Refactoring work can be drudgery.

B. Pros

1. It has to be done. I don't think any Bisq dev team member disagrees.

2. Enables migration from JavaFX desktop application to modular API (that the desktop application will use)

   The benefit of careful, step-by-step refactoring of some existing Bisq UI desktop code for each new API feature is a system of smaller modules, capable of supporting clients (including web/mobile) written in several programming languages.

   Technical Note: I think defaulting to Gradle for organizing new API modules needs to be discussed among devs. The future upgrade to JDK 15+ makes it worthwhile to at least take another look at Java modules. We may stick with Gradle as the build system, but it might not be the best tool for defining the finer grained modules -- what are now Gradle build file sub-projects.

III. How does gRPC affect decisions about new clients

A. API's client server data transport protocol is HTTP/2

1. Extra work is needed to make the API server vend Json to RESTful HTTP/1.1 web & mobile clients.

   Support for RESTful clients can be achieved With the help of an externally run HTTP reverse proxy in front of an API server, and coding effort to transform Bisq's protobuf messages into Json.

2. No additional server side work is need to send protobuf messages to HTTP/2 gRPC clients.

   As mentioned in item A.1, a proxy is still needed in front of the API server, but if the client works with gRPC, no additional work on the server is need to support new desktop and web/mobile clients.

   Note: When I say "no additional work on the server is needed", I mean that an existing API method can be invoked from a gRPC client without the server having to transform data into another data format expected by the client. Everything sent from the server is a gRPC protobuf message, no matter what kind of gRPC client sent the request.

B. Any new API security mechanism should be supported by gRPC HTTP/2

The current CLI security mechanism is the simplest possible: a cleartext --password= option. We might want to discuss implementing something stronger, but not as heavy as macaroons, which Chris Beams and I have experimented with.

IV. gRPC Web / Mobile UI

JavaScript clients can make gRPC calls to the API server using grpc-web. I have gone through a tutorial to create a commonjs client to invoke some API service methods (pretty easy), and I read in the grpc-web docs that google closure tools are also compatible (but I don't know anything about google closure tools).

As mentioned in the introduction, there seems to be consensus among devs and end users that a new web client is needed. But development plans should be formed by most core dev team members -- not one dev -- before proceeding with any UI prototype development.

V. gRPC CLI

A. The API's CLI should remain the reference client

It is my opinion that most methods exposed to end users should work in the existing gRPC CLI, even when UI development drives the work. This would be a loose rule to keep the CLI in sync with the API and web UI development. Some API methods may not be appropriate for the CLI, for example, registerdisputeagents.

B. New API methods should be supported in the CLI as soon as they are made available to a new web client

I'm being repetitive, but I think that if and when work on a new UI requires parallel efforts in the API server, do not forget to update the gRPC CLI as well. I am not saying web UI development should ever have to wait for methods to be implemented in the CLI, but the CLI should never get left far behind the UI in terms of supported features. It is understood that some web UI features may rely on bi-directional streaming, for example, and will be incompatible with the session-less CLI.

VI. Bots

I have experimented with a couple of java bots, and think the API might include a java bot support module in the future. However, releasing any java bot support this early in the API's development is not a great idea. There are many asynchronous activities that occur in offer creation and during a trade; a bot programmer may make assumptions about offer / trade preparation and state that are not true (yet). It is also prudent to let the API flush out old problems in the core layers and get them fixed before encouraging bot deployment with too much enthusiasm.

VII. API still missing basic functionality, can easily reproduce bloom filter problem (Technical)

A. Unimplemented offer related methods

editoffer, enableoffer, disableoffer, support for trigger price option in createoffer and editoffer

B. Unimplemented trade related methods

gettrades (return user's trades), maybe add add an optional --short-id option to gettrades, and make it clear in docs/help that getrrades --short-id=<short-id> may return more than one trade

C. Bloom filter configuration a problem for larger wallets

The size of the bloom filter plus infrequent filter update broadcasts become a problem after a certain number of wallet transactions are created. For example, deposit transactions cannot be found. API tests can easily reproduce the problem, and increasing the frequency of filter updates removes a strict trade count limit within a single API session. This may help devs find a solution.

VII. Conclusion

Much this doc is based on my own opinion and observations after working mostly alone on the API, and I realize the inclusion of technical issues here may be annoying. I am trying to get both API user feedback and involvement from other devs about the next development phase from the same discussion. Call me lazy.

Hypothetically, without any other input on how to proceed with the API work, and with an experienced colleague with good web client skills (specifically with gRPC on Node.js), I would work closely with the UI developer to build a web client prototype.

What do you think?

[ghubstan]

Nothing looks right immediately after posting it. Please offer opinions on anything API related, not just what's defined in the outline.

[sqrrm]

Thanks for the write up.

UI

I think using the UI to drive API dev is a good idea, although that doesn't mean other use cases should be ignored. I would also like a web and mobile UI but at this point it's not clear how much should be done as part of the DAO and what could be just left to anyone who feels so inclined.

The benefits of being Bisq DAO sanctioned is that users feel more confident in using these UIs, and they would perhaps be more full feature. The drawback is that we would have to support them and the efforts to support multiple UIs, the current client, web and mobile, could become too much and splinter the focus, making all UIs worse in the process.

I think there is a benefit to code refactor, but that could be gained by just support the client UI as we do now and exposing the api for external devs to use and build their own.

Bots

I don't think there should be any DAO sanctioned bot. Anyone who wants to write one can do so and reap the benefits themselves. The exception being if we find a use case for a burningman bot or some other automation of roles that has been discussed elsewhere. Proper documentation on how to use the API should be enough I think.

Inconclusion

I'm not sure what direction is best here. I'm leaning towards starting on a simple web or mobile UI that can't trade but that could receive notifications and see the market.

Keeping the CLI as reference sounds good.

[flix1]

I don't know if this is the right place to write this, but it would be great for the next API release to include BSQ trading. Altcoin trading in general, since it does not pass through fiat institutions, banks, could be almost 100% automated. XMR would also be a good candidate.

[ghubstan]

It's the right place for me... BSQ trade support was almost done before I posted this discussion. Your comment reminded me I needed to finish it ;-) I submitted #5412 about a 1/2 hr ago, and added a comment to the PR showing how to use the API's CLI to do a BSQ trade. (I have done one BSQ trade on mainnet.)

Doing the work to support XMR and other altcoin trades may take awhile. Let's see what the plan for the next phase of API work will be.