Code

Backup working#11881

Code

Draft

flowguru wants to merge 101 commits intoapple:release-7.3apple/foundationdb:release-7.3from flowguru:backup-workingflowguru/foundationdb:backup-workingCopy head branch name to clipboard

+3,550-383

Commits

Commits on Nov 11, 2024

• New backup consolidated commit

  

  Hao

  committed
  7db9155

  Copy full SHA for 7db9155

Commits on Nov 14, 2024

• add test

  

  Hao

  committed
  c45efc1

  Copy full SHA for c45efc1

• Rename restore methods of the same name to 3 different names

  Show description for c944a69
  

  Hao

  committed
  c944a69

  Copy full SHA for c944a69

• now it compiles

  

  Hao

  committed
  621e461

  Copy full SHA for 621e461

Commits on Nov 15, 2024

• start to debug

  

  Hao

  committed
  48654f2

  Copy full SHA for 48654f2

• reset offset

  

  Hao

  committed
  4e29fe6

  Copy full SHA for 4e29fe6

Commits on Nov 16, 2024

• correct step

  

  Hao

  committed
  08a61fc

  Copy full SHA for 08a61fc

• fix more

  

  Hao

  committed
  38caff2

  Copy full SHA for 38caff2

Commits on Nov 17, 2024

• now i have CycleBadRead

  

  Hao

  committed
  3063d16

  Copy full SHA for 3063d16

Commits on Nov 18, 2024

• Now mutationRef see incompatible versions

  

  Hao

  committed
  0881c2f

  Copy full SHA for 0881c2f

• now mutations can be print by before transform each mutationTransform

  Show description for 854fc48
  

  Hao

  committed
  854fc48

  Copy full SHA for 854fc48

Commits on Nov 19, 2024

• Now fix 0xff bug

  

  Hao

  committed
  908296b

  Copy full SHA for 908296b

• now realize that finish is exactly once, commit to alog range need to happen in a execute method depiste it is idempotent

  

  Hao

  committed
  d32028e

  Copy full SHA for d32028e

Commits on Nov 20, 2024

• Add RestoreLogDataPartitionedTaskFunc

  Show description for b7b0fe4
  

  Hao

  committed
  b7b0fe4

  Copy full SHA for b7b0fe4

• comments most printing

  

  Hao

  committed
  9ae62c7

  Copy full SHA for 9ae62c7

Commits on Nov 21, 2024

• Fix incompatible version

  

  Hao

  committed
  34a46b8

  Copy full SHA for 34a46b8

• Fix bug of writing totalBytes as int64 but read it as int32

  Show description for 65340fa
  

  Hao

  committed
  65340fa

  Copy full SHA for 65340fa

Commits on Dec 4, 2024

• try to see why test fails

  flowguru

  committed
  8ac75fc

  Copy full SHA for 8ac75fc

• found that cycle was never run after restore

  Show description for 360bbf0
  flowguru

  committed
  360bbf0

  Copy full SHA for 360bbf0

• Changed to shared_ptr

  flowguru

  committed
  66ef25b

  Copy full SHA for 66ef25b

Commits on Dec 5, 2024

• now find there are additional 8 bytes sometimes, causing test to fail

  flowguru

  committed
  de6f6d5

  Copy full SHA for de6f6d5

• Working now

  flowguru

  committed
  c737657

  Copy full SHA for c737657

• add skip loading in cycle test

  flowguru

  committed
  4d32968

  Copy full SHA for 4d32968

• Comment print stderr

  flowguru

  committed
  25f7b04

  Copy full SHA for 25f7b04

• Change default to not as partitioned log

  flowguru

  committed
  cf3a0c5

  Copy full SHA for cf3a0c5

• Fix bug of merge sort

  flowguru

  committed
  fd493f4

  Copy full SHA for fd493f4

• Add more logs

  flowguru

  committed
  09c2796

  Copy full SHA for 09c2796

Commits on Dec 6, 2024

• debug missing verison issue

  flowguru

  committed
  ceefeb5

  Copy full SHA for ceefeb5

• Fix txnCount bug

  flowguru

  committed
  56ec102

  Copy full SHA for 56ec102

• remove trace to run joshua

  flowguru

  committed
  f896653

  Copy full SHA for f896653

Commits on Dec 7, 2024

• Fix initial small diff between begin and end bug

  flowguru

  committed
  6eff3f9

  Copy full SHA for 6eff3f9

• the first param2 has prefix of size, its 32 bytes not 64

  flowguru

  committed
  71bfbd4

  Copy full SHA for 71bfbd4

Commits on Dec 10, 2024

• disable cycle test traces

  flowguru

  committed
  8579fd8

  Copy full SHA for 8579fd8

• when there are multiple partitions for a version, only the first partition size is added to total size

  Show description for 6625513
  flowguru

  committed
  6625513

  Copy full SHA for 6625513

• Added FlowGuruNewVersion, suspect same version is not returned once, but being splited into multiple calls

  Show description for 75d5750
  flowguru

  committed
  75d5750

  Copy full SHA for 75d5750

• Add FlowguruFoundNewMinVersion, i want to see if the 311782629 comes from the same k or different k

  Show description for 94ea890
  flowguru

  committed
  94ea890

  Copy full SHA for 94ea890

• Added FlowguruIteration, to see why 0, 2 and 3 have 311782629 first and 1 has it later

  flowguru

  committed
  5f5b962

  Copy full SHA for 5f5b962

• Add FlowguruAddFile, to confirm tag=1 is processed by another worker

  Show description for 1eab59a
  flowguru

  committed
  1eab59a

  Copy full SHA for 1eab59a

• Try to fix the bug that each tag might have different begin,end in each file

  flowguru

  committed
  c18fcb8

  Copy full SHA for c18fcb8

• Fix existing compiling bug, astonishing

  flowguru

  committed
  30d6b83

  Copy full SHA for 30d6b83

Commits on Dec 11, 2024

• now start to debug before=3000, after=0 issue

  Show description for 6d41428
  flowguru

  committed
  6d41428

  Copy full SHA for 6d41428

• Suspect we only need a range file and it is not being restored

  flowguru

  committed
  11da0ca

  Copy full SHA for 11da0ca

• Fix keyAfter bug due to tuple

  flowguru

  committed
  5c9de22

  Copy full SHA for 5c9de22

• comment traces to test again

  Show description for 18d292e
  flowguru

  committed
  18d292e

  Copy full SHA for 18d292e

• debug cycle got longer and node count changed issue

  flowguru

  committed
  f29a437

  Copy full SHA for f29a437

• print FlowGuruRestoreMutation and FlowGuruBeforeSendRequest

  flowguru

  committed
  942ddb4

  Copy full SHA for 942ddb4

• FlowGuruRestoreMutation happens first when reading from file

  Show description for e09f4be
  flowguru

  committed
  e09f4be

  Copy full SHA for e09f4be

• added FlowGuruGetApplyChanges, which should indicate the range to restore from log

  flowguru

  committed
  7494d66

  Copy full SHA for 7494d66

• Add log in sendCommitTransactionRequest

  flowguru

  committed
  0888b25

  Copy full SHA for 0888b25

• Now confirms all the mutations commits succesfully by proxy, suspect range file is restored in the wrong taskfunc

  Show description for e85acef
  flowguru

  committed
  e85acef

  Copy full SHA for e85acef

• Fix bug: one range file might be restored multiple times and override the correct value from log files

  Show description for 8716e30
  flowguru

  committed
  8716e30

  Copy full SHA for 8716e30

• remove logs

  flowguru

  committed
  788478f

  Copy full SHA for 788478f

• according to FlowGuruPrintBySubVersion, 33 and 34 form a cycle

  Show description for a245a1d
  flowguru

  committed
  a245a1d

  Copy full SHA for a245a1d

• Add back FlowGuruAddM2, compare FlowGuruAddM2 and FlowGuruPrintBySubVersion

  flowguru

  committed
  7cf324e

  Copy full SHA for 7cf324e

• Add FlowguruAddFile and FlowGuruStartRestoreLogDataPartitionedTaskFunc

  flowguru

  committed
  c398701

  Copy full SHA for c398701

• Added FlowguruCheckEachVersion and FlowGuruCheckMinVersion

  Show description for 1a1cea3
  flowguru

  committed
  1a1cea3

  Copy full SHA for 1a1cea3

Commits on Dec 12, 2024

• Add readNextBlock

  flowguru

  committed
  ec1929e

  Copy full SHA for ec1929e

• Add Offset in FlowGuruRestoreMutation

  flowguru

  committed
  9407088

  Copy full SHA for 9407088

• Now find FlowguruAddFile and FlowGuruAllVersionConsumed only got printed once, suspect only the first taskfunc got to read the file

  flowguru

  committed
  0bcb799

  Copy full SHA for 0bcb799

• find the bug, that log files comes before range file, and so only the first taskfunc has the log file

  flowguru

  committed
  217abfa

  Copy full SHA for 217abfa

Commits on Dec 13, 2024

• try fix the bug

  flowguru

  committed
  31d6ac5

  Copy full SHA for 31d6ac5

• Reduce traces to print otherwise segfault

  flowguru

  committed
  f7dd96c

  Copy full SHA for f7dd96c

• Failing again, debugging by adding FlowGuruBeforeSendRequest

  flowguru

  committed
  c02d64d

  Copy full SHA for c02d64d

• Fix bug that when beginVersion = restoreVersion, we should run this task instead of quit, because restoreVersion will be restored in this task

  flowguru

  committed
  7021b4a

  Copy full SHA for 7021b4a

• enable FlowGuruCommitSucceed and FlowGuruCommitError to see if commit suceeds or not

  flowguru

  committed
  2176bcb

  Copy full SHA for 2176bcb

• Fix firstVersion bug

  flowguru

  committed
  1356190

  Copy full SHA for 1356190

• find who is missing via FlowGuruNodeCountChanged and see their mutations

  flowguru

  committed
  b5c7aa6

  Copy full SHA for b5c7aa6

• now there are multiple same subverison for the same commit version according to FlowGuruPrintBySubVersion

  Show description for c0a3265
  flowguru

  committed
  c0a3265

  Copy full SHA for c0a3265

• We can see multipe subversion for the same version from FlowGuruAddEachSubVersion, now trying to see if backup worker also sees same subversions too by adding FlowGuruAddMutation

  flowguru

  committed
  93b3bfe

  Copy full SHA for 93b3bfe

• FlowGuruAddMutation does not have many same subversion for the same commit version

  flowguru

  committed
  99441cb

  Copy full SHA for 99441cb

• Found the bug, with the restore range of [15,25], for start log files

  Show description for c623385
  flowguru

  committed
  c623385

  Copy full SHA for c623385

• Remove trace and print

  flowguru

  committed
  2fb9f57

  Copy full SHA for 2fb9f57

• now cycle got longer

  flowguru

  committed
  845d5f3

  Copy full SHA for 845d5f3

• enable FlowGuruPrintBySubVersion and FlowGuruBeforeSendRequest again

  flowguru

  committed
  17bfc1d

  Copy full SHA for 17bfc1d

• only print for 2,8 and 9

  flowguru

  committed
  9b0f769

  Copy full SHA for 9b0f769

• Add PartitionedLogIteratorSimple

  flowguru

  committed
  729db90

  Copy full SHA for 729db90

Commits on Dec 14, 2024

• Fix bug, not clear the previous verion's vector

  flowguru

  committed
  3b183ec

  Copy full SHA for 3b183ec

• Use two buffers again

  flowguru

  committed
  e61199d

  Copy full SHA for e61199d

• move test to slow

  flowguru

  committed
  7f0316a

  Copy full SHA for 7f0316a

• Fix overflow issue

  flowguru

  committed
  d1e7852

  Copy full SHA for d1e7852

• Add more logs, find there will be overlap between log files of a single tag

  Show description for bc0e65a
  flowguru

  committed
  bc0e65a

  Copy full SHA for bc0e65a

• Add virtual destructor to property free memory

  flowguru

  committed
  7396ca6

  Copy full SHA for 7396ca6

• remove all traces

  flowguru

  committed
  a4a026a

  Copy full SHA for a4a026a

Commits on Dec 16, 2024

• add endVersonForEachFile

  flowguru

  committed
  9ed79a0

  Copy full SHA for 9ed79a0

• Added FlowGuruCheckMinVersion and FlowGuruGenerateNewVersion

  flowguru

  committed
  3637fff

  Copy full SHA for 3637fff

• Add FlowGuruReadNextFile

  flowguru

  committed
  ef11802

  Copy full SHA for ef11802

• now if we read file0 and reach the end, then fileIndex will be set to 1 but we are still reading file0

  flowguru

  committed
  d10accd

  Copy full SHA for d10accd

• it seems to work now

  flowguru

  committed
  83d8046

  Copy full SHA for 83d8046

Commits on Dec 17, 2024

• use simple iterator

  flowguru

  committed
  36b5aed

  Copy full SHA for 36b5aed

• use shared_ptr for simple iterator

  flowguru

  committed
  07da57f

  Copy full SHA for 07da57f

• It seems to work

  flowguru

  committed
  f200712

  Copy full SHA for f200712

• remove traces to test

  flowguru

  committed
  9508b3a

  Copy full SHA for 9508b3a

• add back FlowGuruPrintBySubVersion and FlowGuruBeforeSendRequest to debug node changed

  Show description for eea8fca
  flowguru

  committed
  eea8fca

  Copy full SHA for eea8fca

• now FlowGuruCheckMinVersion has MinVersion="163155036" Vec="163155036, 46167206, 163155036, "

  flowguru

  committed
  1dfccf1

  Copy full SHA for 1dfccf1

• add FlowGuruSimpleFindOverlapAndSkipAfter

  flowguru

  committed
  f34f7f1

  Copy full SHA for f34f7f1

• Add FlowguruAddFile and FlowGuruLogFile to check fileSize

  flowguru

  committed
  0afbfbd

  Copy full SHA for 0afbfbd

• Fix bug not setting fileOffset

  flowguru

  committed
  016a7aa

  Copy full SHA for 016a7aa

Commits on Jan 3, 2025

• Change each restore window to 10s

  flowguru

  committed
  de13f43

  Copy full SHA for de13f43

• two buffers impl

  flowguru

  committed
  6e4d520

  Copy full SHA for 6e4d520

Commits on Jan 4, 2025

• Seems fixed

  flowguru

  committed
  18c5e82

  Copy full SHA for 18c5e82

Commits on Jan 7, 2025

• comment traces

  flowguru

  committed
  9f29020

  Copy full SHA for 9f29020