You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We need to think about how some special data structures, like streams, can be written to state and shared between steps in the same workflow.
Usually right now we strictly serialize state objects which can cause loss of data (regexes will be ruined for example, and functions dropped).
But internally, it's super useful to be able to share a stream between steps. Then eg the http adaptor can fetch a stream from the server, and pass it to eg dhis2 to upload directly from the stream.
This may also relate to #648, if the same solution allows functions to be internally written to state.
There are two things to watch out for:
The final state returned by the runtime must be safely serialisable. A lot of workflows don't care about the notion of "final state", but some do, and we generally need to make sure we have consistent handling
The worker will send dataclip output from each step to lightning through a websocket. This needs to be properly serializable so that it can be sent to lightning. That does raise a problem of: if the input to a step requires a non-serializable value (like a stream), how could we resume the run from that point? We might need some means of flagging to say "this step is not reproducible". Same problem in the CLI.
So maybe we have a serialize() function somewhere in util with a flag for internal and external mode? Or strict and non-strict?
The text was updated successfully, but these errors were encountered:
We need to think about how some special data structures, like streams, can be written to state and shared between steps in the same workflow.
Usually right now we strictly serialize state objects which can cause loss of data (regexes will be ruined for example, and functions dropped).
But internally, it's super useful to be able to share a stream between steps. Then eg the http adaptor can fetch a stream from the server, and pass it to eg dhis2 to upload directly from the stream.
This may also relate to #648, if the same solution allows functions to be internally written to state.
There are two things to watch out for:
So maybe we have a
serialize()
function somewhere in util with a flag for internal and external mode? Or strict and non-strict?The text was updated successfully, but these errors were encountered: