Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid constructing rust strings when performing varbin/varbinview dict compression #1954

Closed
robert3005 opened this issue Jan 15, 2025 · 1 comment

Comments

@robert3005
Copy link
Member

robert3005 commented Jan 15, 2025

Dict compression does a very naive iteration of the array which constructs rust strings from the array data. Since we only need to hash the bytes we can avoid constructing those strings and we can avoid wrapping them in options.

https://github.com/spiraldb/vortex/blob/develop/encodings/dict/src/compress.rs#L88-L107

related to #1453

@robert3005 robert3005 changed the title Dict compression of VarBinView can be optimized Avoid constructing rust strings when performing varbin/varbinview dict compression Jan 15, 2025
@robert3005
Copy link
Member Author

We do not construct rust strings, we always operate on byte slices

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant