Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance during repeated binfuse::sharded_filter::add_shard() #3

Open
oschonrock opened this issue Dec 10, 2024 · 0 comments
Open

Comments

@oschonrock
Copy link
Owner

oschonrock commented Dec 10, 2024

the unmapping and re-mmapping take ~400ms for 256 shards. Could be mitigated with a "predictive growth plus trim at the end" strategy.

This may seem like a lot, however for real data, unlike in a test, where building the whole filter (for hibp for example) takes 2-3minutes for 1billion entries, the 400ms is not significant.

The primary use case for sharded_filter is for such large datasets, so this would probably be a microptimisation.

@oschonrock oschonrock changed the title performance during repeated binfuse::sharded_filter::add() performance during repeated binfuse::sharded_filter::add_shard() Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant