Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow scanning partition backwards #125

Closed
albe opened this issue Jul 25, 2020 · 1 comment
Closed

Allow scanning partition backwards #125

albe opened this issue Jul 25, 2020 · 1 comment
Labels

Comments

@albe
Copy link
Owner

albe commented Jul 25, 2020

Currently, partitions can only be scanned forward from a known position (0 without a persisted index), because the data block size is only stored in the document header/prefix.
This makes things simpler in the partition, but strictly requires a consistent index in order to allow random range access.
If the document size would also be appended to the document, the partition would effectively become a double linked list and allow backward scanning from a known position (e.g. file end).

This would allow a couple of optimizations, most notably in order to detect a failed write the index would become irrelevant and hence allow recovery on a partition level (see also #31), because for that we need to scan the partition backwards and find the last finished commit and truncate up to that.
Possibly worth to note that a failed write will likely have an invalid "document size" at the end though, so likely it's not as trivial for this specific use-case.

@albe
Copy link
Owner Author

albe commented Jul 3, 2021

Resolved by #149

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant