run analyzer crashes when trying to get plots for fully cached runs

[avigailmo]

We tried to run the analyzer (version v0.9.0 or on main) including the timeline plots (-P) on an omics run that all its tasks had cache hit
using this command:
python -m omics.cli.run_analyzer 3460497 -P /tmp/

it failed on keyError:
KeyError: 'stopTime'

This is the full log trace:

No tasks found with timing data, a plot cannot be created
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/Users/avigailmoldovan/dev/amazon-omics-tools/omics/cli/run_analyzer/main.py", line 644, in
timeline.plot_timeline(resources, title=title, max_duration_hrs=run_duration_hrs)
File "/Users/avigailmoldovan/dev/amazon-omics-tools/omics/cli/run_analyzer/timeline.py", line 131, in plot_timeline
max_stop_time = data["stopTime"].max()
~~~~^^^^^^^^^^^^
File "/Users/avigailmoldovan/Library/Caches/pypoetry/virtualenvs/amazon-omics-tools-SSF_CxZg-py3.12/lib/python3.12/site-packages/pandas/core/frame.py", line 4102, in getitem
indexer = self.columns.get_loc(key)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/avigailmoldovan/Library/Caches/pypoetry/virtualenvs/amazon-omics-tools-SSF_CxZg-py3.12/lib/python3.12/site-packages/pandas/core/indexes/range.py", line 417, in get_loc
raise KeyError(key)
KeyError: 'stopTime'

[markjschreiber]

Thanks for the bug report. I will update the plotting logic for tasks with no stop and start time. These are presumably cached tasks that technically didn't run. I am thinking that they probably shouldn't be shown in the timeline (as they have no time)? What do you think?

[markjschreiber]

I suppose a fully cached run might just gracefully produce no timeline plot and maybe just issue a warning?

[avigailmo]

I suppose a fully cached run might just gracefully produce no timeline plot and maybe just issue a warning?

I agree. no timeline plot required for fully cached run/ cached tasks - I'm OK with every way you choose to solve this and prevent the script from crashing.

[markjschreiber]

Pull #86 seems to fix this issue for me. Can you confirm it works for you? If so I can release it.

[avigailmo]

Pull #86 seems to fix this issue for me. Can you confirm it works for you? If so I can release it.

works great thanks!

arn,type,name,startTime,stopTime,runningSeconds,cpusRequested,gpusRequested,memoryRequestedGiB,omicsInstanceTypeReserved,omicsInstanceTypeMinimum,recommendedCpus,recommendedMemoryGiB,estimatedUSD,minimumUSD,cpuUtilizationRatio,memoryUtilizationRatio,storageUtilizationRatio,cpusReserved,cpusMaximum,cpusAverage,gpusReserved,memoryReservedGiB,memoryMaximumGiB,memoryAverageGiB,storageReservedGiB,storageMaximumGiB,storageAverageGiB
arn:aws:omics:us-east-1:380827583499:task/9461320,task,CreateReferenceCache,,,,,,,,,,,,,,,,,,,,,,,,,
arn:aws:omics:us-east-1:380827583499:task/2565127,task,MergeCramFiles,,,,,,,,,,,,,,,,,,,,,,,,,
arn:aws:omics:us-east-1:380827583499:task/5909508,task,AlignWithUA,,,,,,,,,,,,,,,,,,,,,,,,,
arn:aws:omics:us-east-1:380827583499:task/9782914,task,Demux,,,,,,,,,,,,,,,,,,,,,,,,,
arn:aws:omics:us-east-1:380827583499:task/9003163,task,Sorter,,,,,,,,,,,,,,,,,,,,,,,,,
arn:aws:omics:us-east-1:380827583499:task/7932889,task,ConvertSorterStatsToH5,,,,,,,,,,,,,,,,,,,,,,,,,
arn:aws:omics:us-east-1:380827583499:task/1691114,task,CreateReportSingleSampleQC,,,,,,,,,,,,,,,,,,,,,,,,,
arn:aws:omics:us-east-1:380827583499:run/3460497,run,master-def153-8c9f-nightly,2024-12-22T00:57:52.936Z,2024-12-22T01:36:03.455Z,2290.519000,,,,,,,,,,,,,,,,,,,,,,
No tasks found with timing data, possibly all tasks were cached, a plot cannot be created

[markjschreiber]

A new '0.10.0' release has been made for https://pypi.org/project/amazon-omics-tools/