You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we handle both file conversions and file transfers through convert. This ends up blowing up the graph a bit because we have to manage all pairs of format+location.
I wonder if there is some way to orthogonalize these two ideas and have a separate convert graph for formats and some other system for relocation / file transfers.
I don't have a clear idea of how to do this yet but it seems worth thinking about.
The text was updated successfully, but these errors were encountered:
We have file formats
CSV, JSON, HDF5, ...
and locations
SSH, HFDS, S3, ...
Currently we handle both file conversions and file transfers through
convert
. This ends up blowing up the graph a bit because we have to manage all pairs of format+location.I wonder if there is some way to orthogonalize these two ideas and have a separate convert graph for formats and some other system for relocation / file transfers.
I don't have a clear idea of how to do this yet but it seems worth thinking about.
The text was updated successfully, but these errors were encountered: