Web3.storage driver #11
Conversation
drivers/drivers.go
Outdated
| @@ -43,6 +43,7 @@ type OSDriver interface { | |||
| NewSession(path string) OSSession | |||
| Description() string | |||
| UriSchemes() []string | |||
| Publish() string | |||
There was a problem hiding this comment.
Could be called Flush as well for a more common concept in I/O
There was a problem hiding this comment.
I think Publish() is more verbose, since it's explicit that it will return a publicly accessible URL.
There was a problem hiding this comment.
I think I agree with Rafal - Flush() has connotations of cleanup, rather than something that returns something useful
leszko
changed the title
drivers/w3s.go
Outdated
|
|
||
| func (session *W3sSession) ReadData(ctx context.Context, name string) (*FileInfoReader, error) { | ||
| // not supported | ||
| return nil, nil |
There was a problem hiding this comment.
Should we return an error from these unsupported methods to make failures more obvious?
There was a problem hiding this comment.
Yeah, good idea. I'll add an error for not implemented functions.
drivers/w3s.go
Outdated
| if err != nil { | ||
| return nil, err | ||
| } | ||
| child, err = rc.addFileToDag(ctx, child, tail, filename, fileCid) |
There was a problem hiding this comment.
This whole DAG logic seems quite complex! does web3.storage not provide any lib for that? kudos on figuring all of this out, if not
There was a problem hiding this comment.
They provide a lib, but unfortunately only JS. And it's not good enough to be worth running this part as a separate NodeJS binary.
Co-authored-by: Yondon Fu <[email protected]>
… it in the OS URL)
Add web3.storage driver.
fix livepeer/catalyst#375
How does it work?
SaveData()does the following:pubIdPublish()uploads the whole dag directory structure into web3.storage and therefore makes all the HLS manifest files usable.Additional comments
ipfs-car --pack- it archives any file into a CAR file, I think that technically we could do the same using Golang andgo-carlibrary, but I think it may be more work and it may be more error-prone, since big files are actually sharded in the IPFS structuresw3 can store add- it uploads a CAR into web3.storagew3 can upload add- it binds together a number of CARs and publishes their root with the public URLkeyandproofvariables are not used right now; in this PR the localw3credentials are used.Publish()step,the each subdirectory is uploaded as a separate CAR; technically, we should not need it, we should just create one CAR with the whole subdirectories and upload it. However, currently theUPDATE:go-carlibrary fails for these scenarios, so that's the workaround. Here's the go-car PR which fixes the issue: Allow using WalkOption in WriteCar function ipld/go-car#357go-carmerged my PR, so we there is no longer need to create each subdirectory separately. Updated the PR.