-
Notifications
You must be signed in to change notification settings - Fork 469
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DevX: Improve error reporting for benchmark jobs. #8125
Comments
…6286) related to issue: pytorch/executorch#8125 example run: https://github.com/pytorch/executorch/actions/runs/13275229180/job/37063477575 Set defult warning for missing artifact  add test for no artifacts
@yangw-dev There is another tiny improvement we can make. As shown on the screenshot, there are two entries for "iPhone 15". They are actually different, one is running on iOS 18, and the other one is iOS 17. We should considering adding the major.minor os version to avoid confusion and improve the logging readability. cc: @huydhn |
Given that there is an issue where timing out job won't print anything. I'm tempted to split this part into 2 steps:
Thoughts? |
🐛 Describe the bug
As shown in an example run, there are two issues that make it difficult to understand the expected behavior and locate the exact error:
benchmark-on-device
jobs were still marked as successful (highlighted in the red box). The expected behavior is thatbenchmark-on-device
jobs should either not be scheduled or be marked as skipped/canceled if a dependent job, such as export, fails.export-models
job, the error in "Upload artifacts to S3" step is misleading. It should be marked as skipped or canceled, as the "Run script in the container" step failed to generate the .pte file (highlighted in the yellow box).cc: @huydhn @cbilgin @digantdesai @kimishpatel
Versions
trunk
cc @huydhn @kirklandsign @shoumikhin @mergennachin @byjlw
The text was updated successfully, but these errors were encountered: