Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom chunker #183

Closed
alanshaw opened this issue Nov 29, 2021 · 4 comments
Closed

Custom chunker #183

alanshaw opened this issue Nov 29, 2021 · 4 comments
Labels
kind/discussion Topical discussion; usually not changes to codebase need/analysis Needs further analysis before proceeding

Comments

@alanshaw
Copy link
Member

go-ipfs accepts chunkers that confirm to a "Splitter" interface: https://github.com/ipfs/go-ipfs-chunker

It would be really nice to pass a chunker to this library to enable.

refs storacha/ipfs-car#99

@alanshaw alanshaw added the need/triage Needs initial labeling and prioritization label Nov 29, 2021
@mikeal
Copy link

mikeal commented Dec 16, 2021

there is support for a custom chunker although i think the interface may be a bit different https://github.com/ipfs/js-ipfs-unixfs/tree/master/packages/ipfs-unixfs-importer#overriding-internals

@Gozala
Copy link
Contributor

Gozala commented Jan 4, 2022

I am somewhat skeptical that providing importer a chunker is going to be good API given that importer takes several files and each may need to be chunked differently.

That is to suggest that it may be a good idea to reconsider overall importer API in the context of #186 so that chunkers could be provided along with files as opposed to having a one chunker or chunker dispatcher.

@lidel lidel added kind/discussion Topical discussion; usually not changes to codebase need/analysis Needs further analysis before proceeding and removed need/triage Needs initial labeling and prioritization labels Apr 8, 2022
@achingbrain
Copy link
Member

The importer takes a chunker option which is a function of type Chunker.

Please re-open if this doesn't solve the problem.

@achingbrain
Copy link
Member

achingbrain commented Jul 27, 2023

For per-file chunkers, the FileCandidate type could be extended to allow passing a chunker/layout, etc for just that directory entry, but it's not come up before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/discussion Topical discussion; usually not changes to codebase need/analysis Needs further analysis before proceeding
Projects
None yet
Development

No branches or pull requests

5 participants