Workflow examples

[gedw99]

Just wanted to say how great this looks. Thank you for all the hard work on this @ripienaar

I am going to try it out and feedback if thats ok ?

The wiki really helped me see how to use this. Is a good idea.

Love how the CLI wraps the NATS CLI to give an easy way to interact with the system.

[ripienaar]

Thanks for the kind words, happy to have feedback :)

[ripienaar]

But do read the Status section :) its super early days

[gedw99]

yep i know it's all a work in progress :) thanks for the warning though.

Am not going to use this in Prod...

Part of the reason i was so excited to see this is because a higher order Task system on top on NATS is a low risk proposition because all the really tricky stuff is already worked out at the NATS level.

[ripienaar]

Yeah indeed, though not to say its not difficult, I want to support things like composite tasks where a task is a group of tasks with dependencies. This is really hard in this model, but I have some ideas :)

[ripienaar]

(moved to discussions)

[gedw99]

Sounds like a plan so cunning you can put a tail on it and call it a weasel ...

https://www.youtube.com/watch?v=Gp5St7hORyw

[gedw99]

Dependency chains of Tasks is a really useful thing often forgotten about until it hits you later.

[ripienaar]

There's some interesting (though not exactly great) prior ideas in Machinery to look at for this, but I think they're much later I have things like scheduled tasks to sort out first.

[gedw99]

https://github.com/RichardKnop/machinery#workflows

The concepts of groups, chords and chain are the constructs your thinking about ?

• I think these base constructs are a good way to think about them.

Rehydration of state: If one part of the workflow fails, pick up the state and re run.

Idempotency: NATS partitions provide that. But we need a workflow to have a CorrelationID perhaps to group them all togther.

---

Another one to look at is: https://docs.temporal.io/workflows/
Fault-oblivious stateful Workflow code is the core abstraction of Temporal

• versioning, so that old / existing ones are not messed up
• rehydration: https://docs.temporal.io/workflows/#continue-as-new
• CorrelationID: https://docs.temporal.io/workflows/#workflow-id

The workflow routing / logic is coded, and not really part of the system core state: https://github.com/temporalio/temporal/blob/master/schema/sqlite/v3/temporal/schema.sql has just tasks, etc

Here you can see the approach of what a golang coder has to do for a workflow:
https://github.com/temporalio/samples-go/blob/main/child-workflow-continue-as-new/worker/main.go#L13

I have used this and its ok. Things i do not like:

• way too much boilerplate as seen in thexamples.
• workflows feel like an afterthought to a degree.

[ripienaar]

I recently added dependencies between tasks - they are not great tbh. However one can build a DAG and it will run it through, as a super basic building block I can imagine how you would get there from here. But lots to do before we get even anywhere near temporal (temporal is awesome).

It's for sure not the kind of thing I am likely to code myself but would consider using something like an existing workflow engine along with an API this project provide to solve the problem.

[gedw99]

I recently added dependencies between tasks - they are not great tbh. However one can build a DAG and it will run it through, as a super basic building block I can imagine how you would get there from here. But lots to do before we get even anywhere near temporal (temporal is awesome).

Yep just saw it too :) It's a good start though. It's a simple ancestor pattern ?

It's for sure not the kind of thing I am likely to code myself but would consider using something like an existing workflow engine along with an API this project provide to solve the problem.

Ah you want to "export" the workflow execution to another engine.

Here is one approach:

One "not boil the ocean" way is a KanBan style approach / Concept. Its simple but flexible...
The core concept is that each Task can have a tag that maps to a Kanban column. It is just a way of organising tasks.
Human actors can be alerted when it moves to a kanban column as a result of executing under the hood. The column can be configured with a "listener" and will send a notification.
Computer / Service Actors, are exactly the same. The listener is just the "next" process to run.

A general pattern for the state aspects is like a golang func where the out is a "result, err". The Kanban items hold the error being nil or some error. The result being a "ToString" of a human readable status.

As a workflow runs through it tasks, the tag changes, and so hence the kanban column it is in.
Each workflow task when "instaitated is given its "tag" name. This allows workflows to be reusable, and to get different tags.

This is nice because its really allowing different Actors / Services to be "visible" and to collaborate in a choreography way , and for human actors to react as i suggested above.
The workflows and their tasks are orchestration already designed at "compile time".
The kanban and the column listeners are choreography because its reactive at "runtime".

Its flexible and easy to work on as a developer too because its simple.
Because you have Orchestration and Choreography you can do cyclic workflows and deeply nesting ones. The key is the tagging concept.

Curious what you think. BPM's are complex buggers...

[ripienaar]

Interesting, you can essentially do that now by just setting dependencies accordingly - bit awkward. Yeah that swim lane style approach is pretty interesting.

[gedw99]

swim lane - forget the word for it :)

yeah saw this for task dependencies func: https://github.com/choria-io/asyncjobs/blob/main/task.go#L212

Task.Queue could be the Swimlane name, but i think it would be better to have a Task.Tag (optional).
Tag can then be used for anything at all. Like "swimlane: department x" or "swimlane: service Y "
Others might want to use the tags for any other external way of managing / reacting / observing the task flows.
Thats why its just a loose Name:Value.
Maybe use NATS KV for Tags, and so then the watch of nats KV can do the kickoff off the Choreography.
Pretty cool.

Then other Tasks listening against that Task.Tag of "swimlane: service Y " can then be Choreographed.
TaskResult ( https://github.com/choria-io/asyncjobs/blob/main/task.go#L106 ) might need a meta field that is optional. Not sure right now.

[gedw99]

Is there an example of a multi task workflow with dependencies to other tasks yet ? Could we add one to get a feel for things ?

does the cli support this yet ?

Would like to play around with the way you have envisaged it before I jump to assumptions.

[ripienaar]

You can see examples here

asyncjobs/processor_test.go

Line 376 in 6dcd63a

Describe("Dependencies", func() {

CLI supports it

asyncjobs/ajc/task_command.go

Line 56 in 6dcd63a

add.Flag("depends", "Sets IDs to depend on, comma sep or pass multiple times").StringsVar(&c.dependencies)

[gedw99]

Ok thanks . Will build larger examples from that and let you know