Skip to content

Commit

Permalink
Minor update to the TODO list
Browse files Browse the repository at this point in the history
  • Loading branch information
LucaCanali committed Apr 4, 2024
1 parent 53c3392 commit 9f1acc8
Showing 1 changed file with 3 additions and 4 deletions.
7 changes: 3 additions & 4 deletions docs/TODO_and_issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,13 @@ If you plan to contribute to sparkMeasure development, please start by reviewing

**Known issues and TODO list**
* TODO: write more tests for GitHub CI actions.
* TODO: Measure and report Task/stage failures and other errors that are not handled by the code in this version, this puts the effort
on the user to validate the output.
* TODO: Measure and report Task/stage failures and other errors that are not handled by the code in this version,
this puts the effort on the user to validate the output.
* TODO: Task metrics values collected by sparkMeasure are only for successfully executed tasks. Note that
resources used by failed tasks are not collected in the current version. Can this be improved?
* TODO: for Flight recorder mode with file output: implement writing metrics to output files incrementally,
rather than using the current approach of buffering everything in memory and writing at the end.
Note flight recorder mode for Kafka and for InfluxDB does not suffer from this issue.
Note flight recorder mode for Kafka, InfluxDB and for Prometheus push gateway do not suffer from this issue.
* TODO: add code/exceptions to handle error conditions that can arise in sparkMeasure code.
* TODO: add more statistics related to job execution, for example report start/min/max.number of executors
the job had, which is useful in the case of yarn with spark dynamic allocation.
* TODO: add a Prometheus-compatible sink for the flight recorder mode

0 comments on commit 9f1acc8

Please sign in to comment.