Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KqpRun improved multi query mode and storage settings #13177

Merged

Conversation

GrigoriyPA
Copy link
Collaborator

@GrigoriyPA GrigoriyPA commented Jan 5, 2025

Changelog entry

Improved multi query mode and storage settings

Changelog category

  • Improvement

Additional information

  • Added trace opt id for tracing single -p query
  • Supported separate plan / ast / statistics / timeline
  • Supported static storage
  • Added verbose setting
  • Added flags for log settings
  • Added memory info into kqprun embedded UI
  • Added profile memory allocations option
  • Improved flame graph script

Copy link

github-actions bot commented Jan 5, 2025

2025-01-05 08:26:05 UTC Pre-commit check linux-x86_64-release-asan for 1195957 has started.
2025-01-05 08:26:29 UTC Artifacts will be uploaded here
2025-01-05 08:28:59 UTC ya make is running...
🟢 2025-01-05 08:31:25 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
88 88 0 0 0 0

🟢 2025-01-05 08:31:35 UTC Build successful.

@GrigoriyPA GrigoriyPA marked this pull request as draft January 5, 2025 08:26
Copy link

github-actions bot commented Jan 5, 2025

2025-01-05 08:28:03 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1195957 has started.
2025-01-05 08:28:15 UTC Artifacts will be uploaded here
2025-01-05 08:30:43 UTC ya make is running...
🟢 2025-01-05 08:37:37 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
2492 1359 0 0 1133 0

🟢 2025-01-05 08:37:55 UTC Build successful.

Copy link

github-actions bot commented Jan 5, 2025

2025-01-05 08:47:59 UTC Pre-commit check linux-x86_64-relwithdebinfo for 904b5d2 has started.
2025-01-05 08:48:11 UTC Artifacts will be uploaded here
2025-01-05 08:50:33 UTC ya make is running...
🟢 2025-01-05 08:56:25 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
2492 1359 0 0 1133 0

🟢 2025-01-05 08:56:42 UTC Build successful.

Copy link

github-actions bot commented Jan 5, 2025

2025-01-05 08:48:27 UTC Pre-commit check linux-x86_64-release-asan for 904b5d2 has started.
2025-01-05 08:48:40 UTC Artifacts will be uploaded here
2025-01-05 08:51:11 UTC ya make is running...
🟢 2025-01-05 08:53:34 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
88 88 0 0 0 0

🟢 2025-01-05 08:53:41 UTC Build successful.

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 08:46:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4e8af91 has started.
2025-01-06 08:47:01 UTC Artifacts will be uploaded here
2025-01-06 08:49:23 UTC ya make is running...
🟢 2025-01-06 08:55:17 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
2492 1359 0 0 1133 0

🟢 2025-01-06 08:55:32 UTC Build successful.

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 08:46:55 UTC Pre-commit check linux-x86_64-release-asan for 4e8af91 has started.
2025-01-06 08:47:07 UTC Artifacts will be uploaded here
2025-01-06 08:49:37 UTC ya make is running...
🟢 2025-01-06 08:52:04 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
88 88 0 0 0 0

🟢 2025-01-06 08:52:10 UTC Build successful.

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 09:32:42 UTC Pre-commit check linux-x86_64-relwithdebinfo for 81ab384 has started.
2025-01-06 09:32:53 UTC Artifacts will be uploaded here
2025-01-06 09:35:15 UTC ya make is running...
🟢 2025-01-06 09:40:38 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
2492 1359 0 0 1133 0

🟢 2025-01-06 09:40:53 UTC Build successful.

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 09:33:00 UTC Pre-commit check linux-x86_64-release-asan for 81ab384 has started.
2025-01-06 09:33:12 UTC Artifacts will be uploaded here
2025-01-06 09:35:41 UTC ya make is running...
🟢 2025-01-06 09:37:19 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
88 88 0 0 0 0

🟢 2025-01-06 09:37:26 UTC Build successful.

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 10:39:06 UTC Pre-commit check linux-x86_64-release-asan for 0827896 has started.
2025-01-06 10:39:18 UTC Artifacts will be uploaded here
2025-01-06 10:42:10 UTC ya make is running...
🟡 2025-01-06 11:41:19 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13144 13082 0 30 1 31

2025-01-06 11:42:24 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-06 11:54:10 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
106 (only retried tests) 75 0 5 1 25

2025-01-06 11:54:18 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-06 11:58:03 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
60 (only retried tests) 33 0 1 0 26

🟢 2025-01-06 11:58:10 UTC Build successful.
🟢 2025-01-06 11:58:40 UTC ydbd size 3.6 GiB changed* by +34.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: dcb70d0 merge: 0827896 diff diff %
ydbd size 3 872 739 328 Bytes 3 872 774 512 Bytes +34.4 KiB +0.001%
ydbd stripped size 1 350 441 520 Bytes 1 350 442 416 Bytes +896 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 10:39:07 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0827896 has started.
2025-01-06 10:39:19 UTC Artifacts will be uploaded here
2025-01-06 10:42:13 UTC ya make is running...
🟡 2025-01-06 11:31:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20269 18971 0 3 1169 126

2025-01-06 11:32:54 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-06 11:45:04 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
186 (only retried tests) 66 0 0 0 120

🟢 2025-01-06 11:45:11 UTC Build successful.
🟢 2025-01-06 11:45:30 UTC ydbd size 2.1 GiB changed* by +25.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: dcb70d0 merge: 0827896 diff diff %
ydbd size 2 243 627 536 Bytes 2 243 653 776 Bytes +25.6 KiB +0.001%
ydbd stripped size 472 255 312 Bytes 472 255 376 Bytes +64 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 19:18:23 UTC Pre-commit check linux-x86_64-release-asan for fa98d8f has started.
2025-01-06 19:18:35 UTC Artifacts will be uploaded here
2025-01-06 19:21:26 UTC ya make is running...
🟡 2025-01-06 20:19:12 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13147 13068 0 41 8 30

2025-01-06 20:20:20 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-06 20:36:18 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
126 (only retried tests) 88 0 7 7 24

2025-01-06 20:36:27 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-06 20:44:54 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
66 (only retried tests) 37 0 4 0 25

🟢 2025-01-06 20:45:00 UTC Build successful.
🟢 2025-01-06 20:45:33 UTC ydbd size 3.6 GiB changed* by +34.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 2876ba4 merge: fa98d8f diff diff %
ydbd size 3 873 639 440 Bytes 3 873 674 632 Bytes +34.4 KiB +0.001%
ydbd stripped size 1 350 625 264 Bytes 1 350 626 160 Bytes +896 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 19:19:16 UTC Pre-commit check linux-x86_64-relwithdebinfo for fa98d8f has started.
2025-01-06 19:19:30 UTC Artifacts will be uploaded here
2025-01-06 19:22:32 UTC ya make is running...
🟡 2025-01-06 20:10:50 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20273 18971 0 7 1170 125

2025-01-06 20:12:30 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-06 20:24:26 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
194 (only retried tests) 74 0 0 0 120

🟢 2025-01-06 20:24:36 UTC Build successful.
🟢 2025-01-06 20:24:58 UTC ydbd size 2.1 GiB changed* by +25.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 2876ba4 merge: fa98d8f diff diff %
ydbd size 2 244 193 944 Bytes 2 244 220 184 Bytes +25.6 KiB +0.001%
ydbd stripped size 472 320 624 Bytes 472 320 688 Bytes +64 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 21:33:58 UTC Pre-commit check linux-x86_64-relwithdebinfo for 62becca has started.
2025-01-06 21:34:10 UTC Artifacts will be uploaded here
2025-01-06 21:37:01 UTC ya make is running...
🟡 2025-01-06 22:31:53 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20285 18974 0 6 1169 136

2025-01-06 22:33:31 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-06 22:45:21 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
207 (only retried tests) 80 0 2 0 125

2025-01-06 22:45:31 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-01-06 22:56:50 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
185 (only retried tests) 60 0 1 0 124

🟢 2025-01-06 22:56:57 UTC Build successful.
🟢 2025-01-06 22:57:21 UTC ydbd size 2.1 GiB changed* by +25.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 2876ba4 merge: 62becca diff diff %
ydbd size 2 244 193 944 Bytes 2 244 220 184 Bytes +25.6 KiB +0.001%
ydbd stripped size 472 320 624 Bytes 472 320 688 Bytes +64 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 6, 2025

2025-01-06 21:33:59 UTC Pre-commit check linux-x86_64-release-asan for 62becca has started.
2025-01-06 21:34:10 UTC Artifacts will be uploaded here
2025-01-06 21:37:06 UTC ya make is running...
🟡 2025-01-06 22:46:01 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13159 13092 0 30 8 29

2025-01-06 22:47:07 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-06 22:59:15 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
116 (only retried tests) 83 0 5 2 26

2025-01-06 22:59:23 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-06 23:11:02 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
60 (only retried tests) 31 0 3 1 25

🟢 2025-01-06 23:11:09 UTC Build successful.
🟢 2025-01-06 23:11:38 UTC ydbd size 3.6 GiB changed* by +34.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 2876ba4 merge: 62becca diff diff %
ydbd size 3 873 639 440 Bytes 3 873 674 632 Bytes +34.4 KiB +0.001%
ydbd stripped size 1 350 625 264 Bytes 1 350 626 160 Bytes +896 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA added the rebase-and-check Rebase PR with the current base branch and check label Jan 9, 2025
@github-actions github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Jan 9, 2025
@GrigoriyPA GrigoriyPA marked this pull request as ready for review January 9, 2025 09:49
Copy link

github-actions bot commented Jan 9, 2025

2025-01-09 09:52:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1e673a4 has started.
2025-01-09 09:52:49 UTC Artifacts will be uploaded here
2025-01-09 09:55:49 UTC ya make is running...
🟡 2025-01-09 10:54:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20289 18988 0 9 1169 123

2025-01-09 10:56:01 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-09 11:08:19 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
194 (only retried tests) 70 0 2 0 122

2025-01-09 11:08:29 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-01-09 11:20:29 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
182 (only retried tests) 60 0 0 0 122

🟢 2025-01-09 11:20:37 UTC Build successful.
🟢 2025-01-09 11:21:02 UTC ydbd size 2.1 GiB changed* by +25.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 592bc99 merge: 1e673a4 diff diff %
ydbd size 2 244 699 432 Bytes 2 244 725 688 Bytes +25.6 KiB +0.001%
ydbd stripped size 472 406 672 Bytes 472 406 736 Bytes +64 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 9, 2025

2025-01-09 09:53:00 UTC Pre-commit check linux-x86_64-release-asan for 1e673a4 has started.
2025-01-09 09:53:11 UTC Artifacts will be uploaded here
2025-01-09 09:56:04 UTC ya make is running...
🟡 2025-01-09 11:01:50 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13160 13102 0 24 6 28

2025-01-09 11:02:57 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-09 11:14:31 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
96 (only retried tests) 63 0 2 7 24

2025-01-09 11:14:39 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-01-09 11:30:09 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
57 (only retried tests) 32 0 0 0 25

🟢 2025-01-09 11:30:16 UTC Build successful.
🟢 2025-01-09 11:30:44 UTC ydbd size 3.6 GiB changed* by +34.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 0859ec3 merge: 1e673a4 diff diff %
ydbd size 3 874 438 760 Bytes 3 874 473 936 Bytes +34.4 KiB +0.001%
ydbd stripped size 1 350 831 216 Bytes 1 350 832 112 Bytes +896 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

ydb/tests/tools/kqprun/kqprun.cpp Outdated Show resolved Hide resolved
ydb/tests/tools/kqprun/kqprun.cpp Outdated Show resolved Hide resolved
ydb/tests/tools/kqprun/kqprun.cpp Outdated Show resolved Hide resolved
@GrigoriyPA GrigoriyPA force-pushed the YQ-3993-KqpRun-support-static-storage branch from a16614d to 21a9320 Compare January 13, 2025 07:01
@GrigoriyPA GrigoriyPA requested a review from uzhastik January 13, 2025 07:02
Copy link

github-actions bot commented Jan 13, 2025

2025-01-13 07:05:23 UTC Pre-commit check linux-x86_64-relwithdebinfo for 135b3fd has started.
2025-01-13 07:05:34 UTC Artifacts will be uploaded here
2025-01-13 07:08:24 UTC ya make is running...
🟢 2025-01-13 08:03:40 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20303 19008 0 0 1169 126

🟢 2025-01-13 08:05:27 UTC Build successful.
🟢 2025-01-13 08:05:43 UTC ydbd size 2.1 GiB changed* by +25.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 0ae852e merge: 135b3fd diff diff %
ydbd size 2 245 751 616 Bytes 2 245 778 120 Bytes +25.9 KiB +0.001%
ydbd stripped size 472 620 080 Bytes 472 620 144 Bytes +64 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 13, 2025

2025-01-13 07:07:46 UTC Pre-commit check linux-x86_64-release-asan for 135b3fd has started.
2025-01-13 07:07:57 UTC Artifacts will be uploaded here
2025-01-13 07:10:48 UTC ya make is running...
🟡 2025-01-13 08:17:09 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13169 13115 0 19 7 28

2025-01-13 08:18:16 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-13 08:30:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
86 (only retried tests) 57 0 1 2 26

2025-01-13 08:30:32 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-01-13 08:41:37 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
51 (only retried tests) 26 0 0 1 24

🟢 2025-01-13 08:41:44 UTC Build successful.
🟡 2025-01-13 08:42:12 UTC ydbd size 3.6 GiB changed* by +128.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 039db66 merge: 135b3fd diff diff %
ydbd size 3 893 372 472 Bytes 3 893 504 512 Bytes +128.9 KiB +0.003%
ydbd stripped size 1 360 148 528 Bytes 1 360 179 248 Bytes +30.0 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA merged commit 37a80f2 into ydb-platform:main Jan 13, 2025
12 checks passed
@GrigoriyPA GrigoriyPA deleted the YQ-3993-KqpRun-support-static-storage branch January 13, 2025 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants