maintain access to tiers removed from TTL #13733

swalrus1 · 2025-01-22T15:47:11Z

Changelog entry

Maintain access to populated tiers removed from TTL on CS

Changelog category

Improvement
Bugfix

Additional information

...

github-actions · 2025-01-22T15:49:31Z

⚪ 2025-01-22 15:49:30 UTC Pre-commit check linux-x86_64-release-asan for a48564f has started.
⚪ 2025-01-22 15:49:48 UTC Artifacts will be uploaded here
⚪ 2025-01-22 15:52:55 UTC ya make is running...
🟡 2025-01-22 17:02:44 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11679	11572	0	34	43	30

⚪ 2025-01-22 17:03:50 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-22 17:16:09 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
141 (only retried tests)	106	0	4	5	26

⚪ 2025-01-22 17:16:17 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-22 17:35:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
56 (only retried tests)	27	0	2	1	26

🟢 2025-01-22 17:35:43 UTC Build successful.
🟢 2025-01-22 17:36:10 UTC ydbd size 3.6 GiB changed* by +84.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `0dc9b3c`	merge: `a48564f`	diff	diff %
ydbd size	3 856 111 672 Bytes	3 856 198 336 Bytes	+84.6 KiB	+0.002%
ydbd stripped size	1 348 642 992 Bytes	1 348 672 656 Bytes	+29.0 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-22T15:51:03Z

⚪ 2025-01-22 15:51:02 UTC Pre-commit check linux-x86_64-relwithdebinfo for a48564f has started.
⚪ 2025-01-22 15:51:17 UTC Artifacts will be uploaded here
⚪ 2025-01-22 15:54:23 UTC ya make is running...
🟡 2025-01-22 16:57:23 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
25946	23415	0	6	2398	127

⚪ 2025-01-22 16:59:49 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-22 17:11:44 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
195 (only retried tests)	77	0	0	0	118

🟢 2025-01-22 17:11:51 UTC Build successful.
🟢 2025-01-22 17:12:17 UTC ydbd size 2.1 GiB changed* by +13.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `9527980`	merge: `a48564f`	diff	diff %
ydbd size	2 219 486 640 Bytes	2 219 500 064 Bytes	+13.1 KiB	+0.001%
ydbd stripped size	469 487 440 Bytes	469 490 896 Bytes	+3.4 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-23T11:19:36Z

⚪ 2025-01-23 11:19:35 UTC Pre-commit check linux-x86_64-release-asan for a6a2a3e has started.
⚪ 2025-01-23 11:19:47 UTC Artifacts will be uploaded here
⚪ 2025-01-23 11:22:43 UTC ya make is running...
⚫ 2025-01-23 11:26:54 UTC Check cancelled

github-actions · 2025-01-23T11:19:38Z

⚪ 2025-01-23 11:19:38 UTC Pre-commit check linux-x86_64-relwithdebinfo for a6a2a3e has started.
⚪ 2025-01-23 11:19:50 UTC Artifacts will be uploaded here
⚪ 2025-01-23 11:22:44 UTC ya make is running...
⚫ 2025-01-23 11:26:53 UTC Check cancelled

github-actions · 2025-01-23T11:28:18Z

⚪ 2025-01-23 11:28:18 UTC Pre-commit check linux-x86_64-release-asan for d7891ef has started.
⚪ 2025-01-23 11:28:29 UTC Artifacts will be uploaded here
⚪ 2025-01-23 11:31:25 UTC ya make is running...
⚫ 2025-01-23 11:59:47 UTC Check cancelled

github-actions · 2025-01-23T11:30:09Z

⚪ 2025-01-23 11:30:08 UTC Pre-commit check linux-x86_64-relwithdebinfo for d7891ef has started.
⚪ 2025-01-23 11:30:21 UTC Artifacts will be uploaded here
⚪ 2025-01-23 11:33:25 UTC ya make is running...
⚫ 2025-01-23 11:59:49 UTC Check cancelled

github-actions · 2025-01-23T12:02:21Z

⚪ 2025-01-23 12:02:20 UTC Pre-commit check linux-x86_64-release-asan for 59d1beb has started.
⚪ 2025-01-23 12:04:19 UTC Artifacts will be uploaded here
⚪ 2025-01-23 12:07:54 UTC ya make is running...
🟡 2025-01-23 13:14:48 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11284	11235	0	18	4	27

⚪ 2025-01-23 13:15:48 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-23 13:28:20 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
77 (only retried tests)	48	0	2	2	25

🟡 2025-01-23 13:29:19 UTC ydbd size 3.6 GiB changed* by +553.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `bd79c91`	merge: `59d1beb`	diff	diff %
ydbd size	3 858 261 672 Bytes	3 858 828 432 Bytes	+553.5 KiB	+0.015%
ydbd stripped size	1 349 573 328 Bytes	1 349 652 752 Bytes	+77.6 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-23T12:03:58Z

⚪ 2025-01-23 12:03:58 UTC Pre-commit check linux-x86_64-relwithdebinfo for 59d1beb has started.
⚪ 2025-01-23 12:04:10 UTC Artifacts will be uploaded here
⚪ 2025-01-23 12:07:12 UTC ya make is running...
🟡 2025-01-23 12:54:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
18734	17355	0	11	1240	128

⚪ 2025-01-23 12:55:47 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-23 13:14:58 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
257 (only retried tests)	136	0	0	0	121

🟢 2025-01-23 13:15:07 UTC Build successful.
🟡 2025-01-23 13:15:29 UTC ydbd size 2.1 GiB changed* by +363.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `bd79c91`	merge: `59d1beb`	diff	diff %
ydbd size	2 220 428 304 Bytes	2 220 800 856 Bytes	+363.8 KiB	+0.017%
ydbd stripped size	469 670 576 Bytes	469 704 304 Bytes	+32.9 KiB	+0.007%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-23T16:21:01Z

⚪ 2025-01-23 16:21:00 UTC Pre-commit check linux-x86_64-relwithdebinfo for 503ee63 has started.
⚪ 2025-01-23 16:21:05 UTC Artifacts will be uploaded here
⚪ 2025-01-23 16:24:13 UTC ya make is running...
🔴 2025-01-23 16:35:13 UTC Build failed, see the logs. Also see fail summary

github-actions · 2025-01-23T16:22:19Z

⚪ 2025-01-23 16:22:19 UTC Pre-commit check linux-x86_64-release-asan for 503ee63 has started.
⚪ 2025-01-23 16:23:22 UTC Artifacts will be uploaded here
⚪ 2025-01-23 16:26:22 UTC ya make is running...
🔴 2025-01-23 16:34:38 UTC Build failed, see the logs. Also see fail summary

github-actions · 2025-01-23T16:43:53Z

⚪ 2025-01-23 16:43:52 UTC Pre-commit check linux-x86_64-release-asan for 5161f29 has started.
⚪ 2025-01-23 16:44:06 UTC Artifacts will be uploaded here
⚪ 2025-01-23 16:47:10 UTC ya make is running...
⚫ 2025-01-23 16:50:09 UTC Check cancelled

github-actions · 2025-01-23T16:44:29Z

⚪ 2025-01-23 16:44:29 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5161f29 has started.
⚪ 2025-01-23 16:44:41 UTC Artifacts will be uploaded here
⚪ 2025-01-23 16:47:42 UTC ya make is running...
⚫ 2025-01-23 16:50:02 UTC Check cancelled

github-actions · 2025-01-23T16:52:15Z

⚪ 2025-01-23 16:52:14 UTC Pre-commit check linux-x86_64-release-asan for b7e7d58 has started.
⚪ 2025-01-23 16:52:19 UTC Artifacts will be uploaded here
⚪ 2025-01-23 16:55:13 UTC ya make is running...
🟡 2025-01-23 17:52:05 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11284	11209	0	31	16	28

⚪ 2025-01-23 17:53:41 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-23 18:06:21 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
113 (only retried tests)	86	0	0	2	25

🟢 2025-01-23 18:06:32 UTC Build successful.
🟡 2025-01-23 18:06:57 UTC ydbd size 3.6 GiB changed* by +509.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `38d0188`	merge: `b7e7d58`	diff	diff %
ydbd size	3 858 369 408 Bytes	3 858 890 920 Bytes	+509.3 KiB	+0.014%
ydbd stripped size	1 349 602 288 Bytes	1 349 671 184 Bytes	+67.3 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-23T16:52:37Z

⚪ 2025-01-23 16:52:36 UTC Pre-commit check linux-x86_64-relwithdebinfo for b7e7d58 has started.
⚪ 2025-01-23 16:52:51 UTC Artifacts will be uploaded here
⚪ 2025-01-23 16:55:49 UTC ya make is running...
🟢 2025-01-23 18:12:36 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
18736	17426	0	0	1181	129

🟢 2025-01-23 18:14:39 UTC Build successful.
🟡 2025-01-23 18:14:56 UTC ydbd size 2.1 GiB changed* by +338.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `38d0188`	merge: `b7e7d58`	diff	diff %
ydbd size	2 220 477 584 Bytes	2 220 824 016 Bytes	+338.3 KiB	+0.016%
ydbd stripped size	469 681 648 Bytes	469 707 696 Bytes	+25.4 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ivanmorozov333 · 2025-01-23T16:55:13Z

ydb/core/tx/tiering/manager.h

        }

-        TTierRefGuard(const NTiers::TExternalStorageId& tierId, TTiersManager& owner);
-        ~TTierRefGuard();
+        void UpsertConfig(NTiers::TTierConfig config) {


ivanmorozov333 · 2025-01-23T16:55:50Z

ydb/core/tx/tiering/manager.h

    YDB_READONLY_DEF(std::shared_ptr<NMetadata::NSecret::TSnapshot>, Secrets);

 private:
    void OnConfigsUpdated(bool notifyShard = true);
-    void RegisterTier(const NTiers::TExternalStorageId& name);
-    void UnregisterTier(const NTiers::TExternalStorageId& name);
+    void RegisterTierManager(const NTiers::TExternalStorageId& name, std::optional<NTiers::TTierConfig> config);


ivanmorozov333 · 2025-01-23T16:55:58Z

ydb/core/tx/tiering/manager.h


    void UpdateSecretsSnapshot(std::shared_ptr<NMetadata::NSecret::TSnapshot> secrets);
-    void UpdateTierConfig(const NTiers::TTierConfig& config, const NTiers::TExternalStorageId& tierId, const bool notifyShard = true);
-    bool AreConfigsComplete() const;
+    void UpdateTierConfig(std::optional<NTiers::TTierConfig> config, const NTiers::TExternalStorageId& tierId, const bool notifyShard = true);


swalrus1 requested a review from a team as a code owner January 22, 2025 15:47

github-actions bot added the improvement label Jan 22, 2025

swalrus1 force-pushed the allow-missing-eds branch from 490a027 to 4e64227 Compare January 23, 2025 11:15

swalrus1 force-pushed the allow-missing-eds branch from a7120b4 to 63faf17 Compare January 23, 2025 15:58

swalrus1 added 3 commits January 23, 2025 19:18

allow missing tiers in schema

5f7e5f7

fix data race

9fe3e88

fix invalid eds config case

66b075a

swalrus1 force-pushed the allow-missing-eds branch from 32255f1 to 66b075a Compare January 23, 2025 16:18

swalrus1 changed the title ~~maintain access to populated tiers removed from TTL~~ maintain access to tiers removed from TTL Jan 23, 2025

github-actions bot added improvement and removed improvement labels Jan 23, 2025

nit

2b0bc69

fix text compilation

4e42956

ivanmorozov333 reviewed Jan 23, 2025

View reviewed changes

ivanmorozov333 approved these changes Jan 23, 2025

View reviewed changes

swalrus1 merged commit a3d9ffb into ydb-platform:main Jan 23, 2025
12 checks passed

swalrus1 deleted the allow-missing-eds branch January 23, 2025 18:37

swalrus1 mentioned this pull request Jan 26, 2025

tiering fixes #13839

Merged

swalrus1 added a commit to swalrus1/ydb that referenced this pull request Jan 26, 2025

maintain access to tiers removed from TTL (ydb-platform#13733)

556de27

swalrus1 linked an issue Jan 26, 2025 that may be closed by this pull request

VERIFY failed on eviction to tier with not ready storage manager #13469

Closed

zverevgeny linked an issue Jan 26, 2025 that may be closed by this pull request

CS fails when Tier deleted #13449

Closed

This was referenced Jan 27, 2025

YQ-4052 fixed url escaping for s3 insert #13891

Merged

YQ-4043 fixed hanging in kqp write actor #14017

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

maintain access to tiers removed from TTL #13733

maintain access to tiers removed from TTL #13733

swalrus1 commented Jan 22, 2025

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

ivanmorozov333 Jan 23, 2025

ivanmorozov333 Jan 23, 2025

ivanmorozov333 Jan 23, 2025

maintain access to tiers removed from TTL #13733

maintain access to tiers removed from TTL #13733

Conversation

swalrus1 commented Jan 22, 2025

Changelog entry

Changelog category

Additional information

github-actions bot commented Jan 22, 2025 • edited Loading

github-actions bot commented Jan 22, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

github-actions bot commented Jan 23, 2025 • edited Loading

ivanmorozov333 Jan 23, 2025

Choose a reason for hiding this comment

ivanmorozov333 Jan 23, 2025

Choose a reason for hiding this comment

ivanmorozov333 Jan 23, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading