-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DOP-18570] Implement SparkMetricsRecorder #302
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 6, 2024 17:11
77ce48b
to
e74373c
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 6, 2024 18:33
e74373c
to
bf1888c
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 6, 2024 18:43
bf1888c
to
333a880
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #302 +/- ##
===========================================
+ Coverage 94.84% 94.96% +0.12%
===========================================
Files 210 225 +15
Lines 8200 8823 +623
Branches 1413 1507 +94
===========================================
+ Hits 7777 8379 +602
- Misses 299 312 +13
- Partials 124 132 +8 ☔ View full report in Codecov by Sentry. |
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 6, 2024 18:59
333a880
to
959e20b
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 7, 2024 16:29
959e20b
to
359c24b
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 7, 2024 16:37
359c24b
to
e852cfc
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 7, 2024 16:51
e852cfc
to
5c068cd
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 7, 2024 19:34
5c068cd
to
75ab1fb
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 7, 2024 19:57
75ab1fb
to
d7865b4
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 7, 2024 20:15
d7865b4
to
b5561b2
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 7, 2024 20:16
b5561b2
to
42870a5
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 7, 2024 20:25
42870a5
to
a553290
Compare
dolfinus
changed the title
[DOP-18570] Collect Spark metrics in DBWriter and FileDFWriter
[DOP-18570] Implement SparkMetricsRecorder
Aug 8, 2024
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 08:26
2361388
to
a80f5bf
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 09:09
a80f5bf
to
b7e460c
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 09:56
b7e460c
to
2ce8ec9
Compare
7 tasks
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 10:09
2ce8ec9
to
7f8e766
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 10:57
7f8e766
to
b622e64
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 11:07
b622e64
to
5b7a67a
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 11:11
5b7a67a
to
a81fdb7
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 11:26
a81fdb7
to
16e8af4
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 11:34
16e8af4
to
3bd2410
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 11:47
3bd2410
to
58b4622
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 13:20
58b4622
to
edded80
Compare
dolfinus
force-pushed
the
feature/DOP-18570
branch
from
August 8, 2024 13:22
edded80
to
8b5c8b0
Compare
maxim-lixakov
approved these changes
Aug 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Summary
SparkListener
which captures events for object chainexecution -> job -> stage -> task
. It also records raw metrics for tasks, stages and executions od metric values.SparkMetricsCollector
which converts raw metrics to a set of models with human-readable representation of metric values.DBWriter and FileDFWriter will be modified in a separated PR: #303
Related issue number
Checklist
docs/changelog/next_release/<pull request or issue id>.<change type>.rst
file added describing change(see CONTRIBUTING.rst for details.)