Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catch all exceptions in k8s submit_command() #926

Merged
merged 2 commits into from
Aug 21, 2023

Conversation

nemacysts
Copy link
Member

@nemacysts nemacysts commented Aug 21, 2023

It's entirely possible that creating a task_processing task (and/or submitting one) can result in an exception. At the moment, this results in the affected ActionRun getting stuck in the Starting state - but this is a lie and means that the normal monitoring/alerting on failed runs does not kick in.

It's entirely possible the creating a task_processing task (and/or
submitting one) can result in an exception. At the moment, this results
in the affected ActionRun getting stuck in the Starting state - but this
is a lie and means that the normal monitoring/alerting on failed runs
does not kick in.
Copy link
Member

@bobokun bobokun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, if an exception does occur will it show up in META in tronweb? If not it might be useful to add the details there so users have a better understanding of why the job failed

@nemacysts
Copy link
Member Author

lgtm, if an exception does occur will it show up in META in tronweb? If not it might be useful to add the details there so users have a better understanding of why the job failed

@bobokun not atm - I think that i could probably get it to show up there if i did something like task.log.exception(...) - but i'm a little wary of doing that in case there's something funky in creating the task that means we never even initialized the logger

@nemacysts
Copy link
Member Author

(merging in master so that tests pass)

@nemacysts nemacysts merged commit 9d1d0e8 into master Aug 21, 2023
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants