add cutmark concept #199
Open
add cutmark concept #199
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Let's copy in a new version from systemd
…chunks This is a low hanging optimization fruit: when we encode a stream, then pass the chunk compression/storing to a pool of worker threads. This doesn't speed up encoding as much as I hoped, but still: The firefox 63 sources (2.2G) are encoded in 1m16,902s instead of 1m24,281s on my 4 cpu system. i.e. roughly a 10% speed-up.
Let's optimize chunking a bit: let's optionally accept a list of "cutmarks", which are special bit sequences (up to 64bit) that indicate particularly good chunking cut points. This can be used to optimize chunking in data streams we know a few semantics about. The intention is that the object markers of .caidx files are set to be cutmarks, so that we rather cut between objects than at entirely arbitrary positions. This commit only adds logic to find these cutpoints, based on a list of defined cutpoints, the code calling into the chunker does not make use of it yet.
This allows us to update the internal state of the chunker with some data, without necessarily determining where to break.
This way a caidx file is enough to configure the seeder appropriately.
That way we can derive all necessary parameters from the caidx automatically to tune the seeder so that the chunking matches again.
The test checks if the seed is fully reliable and sufficient as a chunk store: it packs up a tree, then removes the chunk store, but retaining hte original tree to us as seed in place ot the chunk store.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please review after #196, since this PR incorporates it.