CasObject v2 #16
CasObject v2 #16
Conversation
rajatarya
commented
Sep 19, 2024
rajatarya
commented
- Changes header to footer.
- Supports chunk compression and chunk headers
- Updated unit-tests
* change range to u32 * remove conversions
- Moved info_length outside of CasObjectInfo and in CasObject - Updated default length of Info struct
cas_object now has header moved to footer, handles ranges, and can support chunk compression. All unit-tests are passing, and a couple new unit-tests are written to support the changes to CasObjectInfo structs.
| CompressionScheme::None => compressed_buf, | ||
| CompressionScheme::LZ4 => { | ||
| let mut uncompressed_buf = Vec::new(); | ||
| let mut dec = FrameDecoder::new(Cursor::new(compressed_buf)); |
There was a problem hiding this comment.
We could also just pass the reader in here?
There was a problem hiding this comment.
What if the reader is longer than the rest of the chunk? Maybe best to just do a .take()
There was a problem hiding this comment.
I think the concern @assafvayner is raising is that we only want to read/decode the length of this chunk, so is there a way to only copy a specific number of bytes from reader -> writer without introducing an in-memory buffer?
There was a problem hiding this comment.
yes this is the second piece, the lz4_flex library doesn't give you a good way to know how much was written. That being said there is a really easy change to make the library make the info available, there was a PR for it, no progress since. We could fork it and do that.
