Add S3 unavailability test #13700

avevad · 2025-01-22T11:34:15Z

Changelog entry

...

Changelog category

Not for changelog (changelog entry is not required)

Additional information

This resolves #13545

github-actions · 2025-01-22T11:38:19Z

⚪ 2025-01-22 11:38:19 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2dee358 has started.
⚪ 2025-01-22 11:38:23 UTC Artifacts will be uploaded here
⚪ 2025-01-22 11:40:44 UTC ya make is running...
🟡 2025-01-22 11:53:43 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
52	46	0	3	2	1

⚪ 2025-01-22 11:53:55 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-22 12:04:45 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
8 (only retried tests)	2	0	3	2	1

⚪ 2025-01-22 12:04:53 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-01-22 12:20:22 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
8 (only retried tests)	3	0	3	1	1

🟢 2025-01-22 12:20:29 UTC Build successful.
🟢 2025-01-22 12:20:40 UTC ydbd size 2.1 GiB changed* by +14.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `a30b332`	merge: `2dee358`	diff	diff %
ydbd size	2 219 414 120 Bytes	2 219 429 176 Bytes	+14.7 KiB	+0.001%
ydbd stripped size	469 474 448 Bytes	469 477 392 Bytes	+2.9 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-22T11:38:36Z

⚪ 2025-01-22 11:38:35 UTC Pre-commit check linux-x86_64-release-asan for 2dee358 has started.
⚪ 2025-01-22 11:38:46 UTC Artifacts will be uploaded here
⚪ 2025-01-22 11:41:08 UTC ya make is running...
🟡 2025-01-22 11:54:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40	30	0	3	5	2

⚪ 2025-01-22 11:54:34 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-22 12:05:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11 (only retried tests)	3	0	3	2	3

⚪ 2025-01-22 12:05:43 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-22 12:20:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
9 (only retried tests)	2	0	3	2	2

🟢 2025-01-22 12:20:42 UTC Build successful.
🟢 2025-01-22 12:20:55 UTC ydbd size 3.6 GiB changed* by +31.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `a30b332`	merge: `2dee358`	diff	diff %
ydbd size	3 856 029 280 Bytes	3 856 061 160 Bytes	+31.1 KiB	+0.001%
ydbd stripped size	1 348 618 416 Bytes	1 348 627 248 Bytes	+8.6 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ydb/tests/olap/ttl_tiering/ttl_unavailable_s3.py

maximyurchuk · 2025-01-22T13:00:42Z

Мне норм, предлагаю:

окнуть у Жени
что-то сделать с asan (перенести в лардж?)
и нельзя ли как-то ускорить тест? Он идет 6 минут relwithdebinfo

maximyurchuk · 2025-01-22T13:10:19Z

ydb/tests/olap/ttl_tiering/ttl_unavailable_s3.py

+        print("!!! simulating S3 hang up -- sending SIGSTOP", file=sys.stderr)
+        os.kill(self.s3_pid, signal.SIGSTOP)
+
+        time.sleep(60)


Может можно без слипа? Типа как-то подождать, что ydb поняла что s3 отвалился в цикле с маленьким слипом

zverevgeny · 2025-01-22T13:23:36Z

ydb/tests/olap/ttl_tiering/ttl_unavailable_s3.py

+        for i in range(ROWS_CHUNKS_COUNT // 2, ROWS_CHUNKS_COUNT):
+            upsert_chunk(i)
+
+        assert self.wait_for(lambda: get_stat() != stat_old, 600), "data distribution continuation"


Тут можно легко какую-т девиацию защитать за успех. Лучше явно проверить, что какое-то осмысленное количество порций перемещено. Как в .sys так и в самих бакетах

И нужно ещё проверить перемещение в обратную сторону

github-actions · 2025-01-22T16:48:44Z

⚪ 2025-01-22 16:48:44 UTC Pre-commit check linux-x86_64-release-asan for 8348fd8 has started.
⚪ 2025-01-22 16:48:55 UTC Artifacts will be uploaded here
⚪ 2025-01-22 16:51:16 UTC ya make is running...
⚫ 2025-01-22 16:57:57 UTC Check cancelled

github-actions · 2025-01-22T16:50:24Z

⚪ 2025-01-22 16:50:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8348fd8 has started.
⚪ 2025-01-22 16:50:54 UTC Artifacts will be uploaded here
⚪ 2025-01-22 16:53:42 UTC ya make is running...
⚫ 2025-01-22 16:57:53 UTC Check cancelled

…navailable-s3

github-actions · 2025-01-22T16:59:31Z

⚪ 2025-01-22 16:59:31 UTC Pre-commit check linux-x86_64-relwithdebinfo for 98e94c7 has started.
⚪ 2025-01-22 17:02:41 UTC Artifacts will be uploaded here
⚪ 2025-01-22 17:05:03 UTC ya make is running...
🟡 2025-01-22 17:17:04 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
52	50	0	1	0	1

⚪ 2025-01-22 17:17:12 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-22 17:18:28 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
4 (only retried tests)	2	0	1	0	1

⚪ 2025-01-22 17:18:37 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-01-22 17:19:44 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
4 (only retried tests)	2	0	1	0	1

🟢 2025-01-22 17:19:50 UTC Build successful.
🟡 2025-01-22 17:20:00 UTC ydbd size 2.1 GiB changed* by +857.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `9527980`	merge: `98e94c7`	diff	diff %
ydbd size	2 219 486 640 Bytes	2 220 365 016 Bytes	+857.8 KiB	+0.040%
ydbd stripped size	469 487 440 Bytes	469 653 680 Bytes	+162.3 KiB	+0.035%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-22T17:02:04Z

⚪ 2025-01-22 17:02:03 UTC Pre-commit check linux-x86_64-release-asan for 98e94c7 has started.
⚪ 2025-01-22 17:02:18 UTC Artifacts will be uploaded here
⚪ 2025-01-22 17:04:44 UTC ya make is running...
🟡 2025-01-22 17:35:22 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40	31	0	4	3	2

⚪ 2025-01-22 17:35:31 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-22 17:46:20 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10 (only retried tests)	1	0	4	2	3

⚪ 2025-01-22 17:46:27 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-22 17:57:26 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10 (only retried tests)	2	0	4	1	3

🟢 2025-01-22 17:57:32 UTC Build successful.
🟡 2025-01-22 17:57:47 UTC ydbd size 3.6 GiB changed* by +1.9 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `9527980`	merge: `98e94c7`	diff	diff %
ydbd size	3 856 161 680 Bytes	3 858 144 024 Bytes	+1.9 MiB	+0.051%
ydbd stripped size	1 348 659 344 Bytes	1 349 521 232 Bytes	+841.7 KiB	+0.064%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-23T08:55:06Z

⚪ 2025-01-23 08:55:05 UTC Pre-commit check linux-x86_64-relwithdebinfo for 67e2616 has started.
⚪ 2025-01-23 08:55:17 UTC Artifacts will be uploaded here
⚪ 2025-01-23 08:57:38 UTC ya make is running...
🟢 2025-01-23 09:08:31 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
52	51	0	0	0	1

🟢 2025-01-23 09:08:37 UTC Build successful.
🟢 2025-01-23 09:08:47 UTC ydbd size 2.1 GiB changed* by +2.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `239f04b`	merge: `67e2616`	diff	diff %
ydbd size	2 220 425 536 Bytes	2 220 428 304 Bytes	+2.7 KiB	+0.000%
ydbd stripped size	469 665 968 Bytes	469 670 576 Bytes	+4.5 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-23T08:55:10Z

⚪ 2025-01-23 08:55:10 UTC Pre-commit check linux-x86_64-release-asan for 67e2616 has started.
⚪ 2025-01-23 08:55:22 UTC Artifacts will be uploaded here
⚪ 2025-01-23 08:57:46 UTC ya make is running...
🟡 2025-01-23 09:09:58 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40	32	0	2	3	3

⚪ 2025-01-23 09:10:07 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-23 09:28:52 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
8 (only retried tests)	1	0	2	3	2

⚪ 2025-01-23 09:29:00 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-23 09:40:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
7 (only retried tests)	1	0	2	1	3

🟢 2025-01-23 09:40:42 UTC Build successful.
🟢 2025-01-23 09:40:56 UTC ydbd size 3.6 GiB changed* by -96 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `48c14e3`	merge: `67e2616`	diff	diff %
ydbd size	3 858 261 672 Bytes	3 858 261 576 Bytes	-96 Bytes	-0.000%
ydbd stripped size	1 349 573 328 Bytes	1 349 573 264 Bytes	-64 Bytes	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Add S3 unavailability test

7ef93dc

avevad added area/cs/tiering test-case labels Jan 22, 2025

avevad requested a review from zverevgeny January 22, 2025 11:34

avevad self-assigned this Jan 22, 2025

github-actions bot added the not-for-changelog label Jan 22, 2025

Merge branch 'main' into test-unavailable-s3

6ae4e74

avevad requested a review from maximyurchuk January 22, 2025 11:38

maximyurchuk reviewed Jan 22, 2025

View reviewed changes

ydb/tests/olap/ttl_tiering/ttl_unavailable_s3.py Show resolved Hide resolved

maximyurchuk reviewed Jan 22, 2025

View reviewed changes

ydb/tests/olap/ttl_tiering/ttl_unavailable_s3.py Show resolved Hide resolved

maximyurchuk reviewed Jan 22, 2025

View reviewed changes

zverevgeny reviewed Jan 22, 2025

View reviewed changes

avevad mentioned this pull request Jan 22, 2025

Tiering doesn't work after S3 unavailability #13719

Closed

zverevgeny previously approved these changes Jan 22, 2025

View reviewed changes

Comment out failing check

b511214

avevad dismissed zverevgeny’s stale review via fce3406 January 22, 2025 16:47

Merge remote-tracking branch 'origin/test-unavailable-s3' into test-u…

c310842

…navailable-s3

avevad force-pushed the test-unavailable-s3 branch from fce3406 to c310842 Compare January 22, 2025 16:57

zverevgeny previously approved these changes Jan 22, 2025

View reviewed changes

avevad enabled auto-merge (squash) January 22, 2025 17:02

Fix codestyle

a6ccf14

avevad dismissed zverevgeny’s stale review via a6ccf14 January 23, 2025 08:51

maximyurchuk approved these changes Jan 23, 2025

View reviewed changes

avevad merged commit 12fbcb7 into ydb-platform:main Jan 23, 2025
12 checks passed

This was referenced Jan 27, 2025

YQ-4052 fixed url escaping for s3 insert #13891

Merged

YQ-4043 fixed hanging in kqp write actor #14017

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add S3 unavailability test #13700

Add S3 unavailability test #13700

avevad commented Jan 22, 2025

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

maximyurchuk commented Jan 22, 2025

maximyurchuk Jan 22, 2025

zverevgeny Jan 22, 2025

zverevgeny Jan 22, 2025

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

Add S3 unavailability test #13700

Add S3 unavailability test #13700

Conversation

avevad commented Jan 22, 2025

Changelog entry

Changelog category

Additional information

github-actions bot commented Jan 22, 2025 • edited Loading

github-actions bot commented Jan 22, 2025 • edited Loading

maximyurchuk commented Jan 22, 2025

maximyurchuk Jan 22, 2025

Choose a reason for hiding this comment

zverevgeny Jan 22, 2025

Choose a reason for hiding this comment

zverevgeny Jan 22, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 22, 2025 • edited Loading

github-actions bot commented Jan 22, 2025 • edited Loading

github-actions bot commented Jan 22, 2025 • edited Loading

github-actions bot commented Jan 22, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading