You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, partitions can only be scanned forward from a known position (0 without a persisted index), because the data block size is only stored in the document header/prefix.
This makes things simpler in the partition, but strictly requires a consistent index in order to allow random range access.
If the document size would also be appended to the document, the partition would effectively become a double linked list and allow backward scanning from a known position (e.g. file end).
This would allow a couple of optimizations, most notably in order to detect a failed write the index would become irrelevant and hence allow recovery on a partition level (see also #31), because for that we need to scan the partition backwards and find the last finished commit and truncate up to that.
Possibly worth to note that a failed write will likely have an invalid "document size" at the end though, so likely it's not as trivial for this specific use-case.
The text was updated successfully, but these errors were encountered:
Currently, partitions can only be scanned forward from a known position (0 without a persisted index), because the data block size is only stored in the document header/prefix.
This makes things simpler in the partition, but strictly requires a consistent index in order to allow random range access.
If the document size would also be appended to the document, the partition would effectively become a double linked list and allow backward scanning from a known position (e.g. file end).
This would allow a couple of optimizations, most notably in order to detect a failed write the index would become irrelevant and hence allow recovery on a partition level (see also #31), because for that we need to scan the partition backwards and find the last finished commit and truncate up to that.
Possibly worth to note that a failed write will likely have an invalid "document size" at the end though, so likely it's not as trivial for this specific use-case.
The text was updated successfully, but these errors were encountered: