[Disk Manager] Create snapshot migration task to another database #2820

jkuradobery · 2025-01-09T12:59:03Z

YDB storage does not support backing up tables with external blobs support. So, the process of switching between databases (e.g. cluster migration) can't be performed right now. To avoid this issue, and provide seamless migration we've chosen semi-manual solution:

migrate snapshots data manually one by one
Shut down controlplane for a short period of time for all the pending tasks to finish.
Migrate remaining snapshots
Clean up not required snapshots.
backup & restore all the tables which do not contain external blobs except chunk map
Here is implementation of a task which migrates a single snapshot's data to the database.

PersistenceConfig contains no information on external blobs.

cloud/disk_manager/internal/pkg/dataplane/config/config.proto

cloud/disk_manager/internal/pkg/dataplane/migrate_snapshot.go

cloud/disk_manager/pkg/app/run.go

cloud/disk_manager/pkg/admin/snapshots.go

cloud/disk_manager/internal/pkg/dataplane/migrate_snapshot.go

BarkovBG · 2025-01-20T14:13:51Z

cloud/disk_manager/internal/pkg/dataplane/migrate_snapshot.go

+	storageSize := dataChunkCount * chunkSize
+
+	t.state.SnapshotSize = size
+	t.state.SnapshotStorageSize = storageSize


может проверить что storageSize совпадают в src и dst?

cloud/disk_manager/internal/pkg/dataplane/migrate_snapshot.go

cloud/disk_manager/pkg/admin/snapshots.go

cloud/disk_manager/pkg/app/dataplane.go

BarkovBG

а тесты будут?)

jkuradobery added 8 commits January 9, 2025 09:44

Create snapshot migration task

c224201

Fix ya.make for proto

d47bd38

Register migration task, add config options

974d260

Add task migration scheduling

764e16f

Fix build, add new database initialization

2637e24

Fix incorrect S3 config

13b4b80

Make MigrationDestinationStorageConfig a SnapshotConfig

d229890

PersistenceConfig contains no information on external blobs.

Fix panic in task scheduling

7e4c9af