Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker: do not send input_dataclip_id if the previous output was too large #774

Merged
merged 6 commits into from
Sep 19, 2024

Conversation

josephjclark
Copy link
Collaborator

Tracking an issue right now where:

  • Step 1 does not send a dataclip id back to lightning
  • Step 2 starts and says "I'm using this dataclip id" in step:start
  • Lightning says "woah nelly, I've never heard of such a wondrous dataclip, I'm not having that"
  • Lightning aborts the run on its side
  • Meanwhile the engine is still running and trying to send messages, which are now getting blocked because the run doesn't actually exist any more

This PR fixes it by simply not sending an input_dataclip_id if that dataclip was withheld from Lightning. I believe this is safe in lightning.

I will create an integration test against the mock and run a manual test against lightning locally

@josephjclark
Copy link
Collaborator Author

Nice - this ran locally with a payload of 0 (all log lines are redacted too)

image

@josephjclark
Copy link
Collaborator Author

Skipping the integration test - it won't tell us much against the mock, so it's not very valuable.

I'm happy with the code, unit tests and manual test

Copy link
Member

@taylordowns2000 taylordowns2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it. So you track each withheld dataclip id and? Makes sense to me.

@josephjclark josephjclark merged commit f33a865 into main Sep 19, 2024
7 checks passed
@josephjclark josephjclark deleted the payload-limit-test branch September 19, 2024 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants