Using a proper workflow management system

[vuillaut]

Seeing PR #111 and @LukasNickel comments, I am thinking that if we want to make something more generic and robust, maybe we should move to a proper workflow management system.
To consider:

• https://www.commonwl.org/
• https://toil.readthedocs.io/

[morcuended]

I also thought about this possibility for lstosa some time ago. Here is another workflow management system: https://snakemake.readthedocs.io/ which also integrates the possibility of working with job schedulers like SLURM.

[vuillaut]

Just mentioning that CWL and CWL integrate with batch schedulers.
See e.g. https://toil.readthedocs.io/en/releases-3.6.x/batchSystem.html

[LukasNickel]

I have heard snakemake being mentioned a lot in the past (but for different workflows). Have you worked with it already?

[morcuended]

Just played with it a bit, but nothing serious.

[vuillaut]

FYI: starting to look into snakemake jut for comparison 🧐

[vuillaut]

Maybe this idea is a good start to merge with lstosa (or maybe I am being too ambitious here).

[morcuended]

You are ambitious :) but indeed it would be a good starting point to join efforts.

[maxnoe]

There are many alternatives here, e.g. luigi / sciluigi:
https://luigi.readthedocs.io/
https://github.com/pharmbio/sciluigi

I evaluated some of them for the my monte carlo production / processing during my phd thesis and in the end used none of them because I wanted to have a database of jobs where I can track state and settings and none of the tools could work with that.

In the end, I used peewee for the orm abstraction (probably would use sqlalchemy + alembic today) and a custom abstraction layer for submitting jobs to clusters (I implemented a slurm and a local worker cluster).

https://github.com/fact-project/mopro3

The custom database first approach has some pretty great advantages, it allows debugging the workflows much more easily since you can e.g. ask for the log files of all failed runs like this:
(Pseudo code)

for job in Stage1Job.query(Stage1Job.log_file).filter_by(state="error"):
    print(job.log_file)