butido-server

[matthiasbeyer]

This is more of a perspectival issue.

So, right now the issue is that running butido from three different workstations from three different users results in a lot of builds on the build infrastructure. The individual instances of butido don't know from eachother and thus submit jobs to the builders as they see fit.

Having a central instance "butido-server" that schedules builds would enable better handling of load, but also some neat optimizations of build ordering.

Also, splitting a "server" part from the current CLI approach would reduce complexity in the CLI implementation, as the CLI would only be a thin-client (in the long run).

There are, of course, some subjects to think about:

The first step would be to make the server as slim as possible, and let the CLI compute everything. The server would only be responsibe for scheduling the individual jobs to the builder nodes.

Later on, though, one could think of a more advanced setup, where the server computes the scripts, the DAG and the optimal execution path. This is not trivial though.

Having a server component in the setup would also introduce other issues: build artifacts must then be stored on the server and synced to the client (the maintainer) on-demand. Definitively do-able.

The database would be another instance.
Caching of some form could be introduced (memcached, redis) to ease the load on the DB (build logs, etc are rather constant and do not change that often, so in-memory caching would be easy)

And while thinking about that, it might even be possible to split the server instance into some form of micro-services. One central scheduler instance and then "agents" which handle the build-orchestration of individual jobs on the builder instances. The builder instances could run (rather small) agents themselves, so that the communication between the orchestrating-parts and the executing-parts would be in our ecosystem.

After that, the executing-parts could implement different backends, not only docker. Building on non-x86_64 (think ARM!!!) could be rather easy to add!