Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runtime: allow streams to be "serialized" to state internally #865

Open
josephjclark opened this issue Feb 10, 2025 · 0 comments
Open

Runtime: allow streams to be "serialized" to state internally #865

josephjclark opened this issue Feb 10, 2025 · 0 comments

Comments

@josephjclark
Copy link
Collaborator

We need to think about how some special data structures, like streams, can be written to state and shared between steps in the same workflow.

Usually right now we strictly serialize state objects which can cause loss of data (regexes will be ruined for example, and functions dropped).

But internally, it's super useful to be able to share a stream between steps. Then eg the http adaptor can fetch a stream from the server, and pass it to eg dhis2 to upload directly from the stream.

This may also relate to #648, if the same solution allows functions to be internally written to state.

There are two things to watch out for:

  • The final state returned by the runtime must be safely serialisable. A lot of workflows don't care about the notion of "final state", but some do, and we generally need to make sure we have consistent handling
  • The worker will send dataclip output from each step to lightning through a websocket. This needs to be properly serializable so that it can be sent to lightning. That does raise a problem of: if the input to a step requires a non-serializable value (like a stream), how could we resume the run from that point? We might need some means of flagging to say "this step is not reproducible". Same problem in the CLI.

So maybe we have a serialize() function somewhere in util with a flag for internal and external mode? Or strict and non-strict?

@github-project-automation github-project-automation bot moved this to New Issues in v2 Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: New Issues
Development

No branches or pull requests

1 participant