Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(erofs): initial commit for erofs support #626

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

rchincha
Copy link
Contributor

Fixes opencontainers/image-spec#1190

What type of PR is this?

Which issue does this PR fix:

What does this PR do / Why do we need it:

If an issue # is not available please add repro steps and logs showing the issue:

Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?

Does this PR introduce any user-facing change?:


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@hsiangkao
Copy link

hsiangkao commented Jun 14, 2024

Hi @rchincha, I'm currently working on incremental builds and I plan to release erofs-utils v1.8 this month (with multi-threaded mkfs, incremental builds, Intel QPL, etc.), but anyway, I'd suggest use cgo for initial EROFS support (with formal liberofs APIs exported) for now (or even binary integration as the very first step) since this go implemention seems somewhat incomplete (and maybe even broken.. I don't have enough time to look into that since other prioritied stuffs are on hands..)

@rchincha
Copy link
Contributor Author

@hsiangkao I suspect that cgo path may be what ends up happening. Currently, just prototyping quickly to understand the interfaces and scope of changes/work.

@rchincha rchincha force-pushed the erofs branch 2 times, most recently from f6a8a55 to c700aaa Compare August 29, 2024 21:09
@hsiangkao
Copy link

hsiangkao commented Sep 18, 2024

One thing I might need to mention here is that erofs supports external blobs or chunks since Linux 5.16 compared to Squashfs because Dragonfly Nydus once asked this feature to deduplicate data in chunks among different container images.

I'm not sure if it's worthwhile to highlight this, you could just make a tiny metadata with external blobs (which can be used for multiple images) for reference.
Currently only Nydus has userspace tools to generate chunk blobs, if that is interested in other use cases, I could seek more time to implement this in mkfs.erofs too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

performance: what can image-spec do to improve handling of large images?
3 participants