feat: shard support prefix #23
Conversation
|
@dgilland could you review this pull request on your spare time? |
| def shard(digest, depth, width): | ||
| # This creates a list of `depth` number of tokens with width | ||
| # `width` from the first part of the id plus the remainder. | ||
| def shard(digest, depth, width, prefix="") -> List[str]: |
There was a problem hiding this comment.
This method is an internal utility that's not meant for direct usage outside the library. It's meant to be passed a hash's HEX digest which then generates a file path. So a prefix argument doesn't make sense within that context.
I don't see the new prefix argument being used anywhere within the library so curious what the desire is here? Are you using shard() directly or have some other usage in mind?
There was a problem hiding this comment.
Yes, I want to use shard only in my python code.
Here was the scenario. I have a export task run in backend. It will generate a file name export_0x<uuid>.csv.
In order to prevent create huge amounts of file in one single directory. I use shard to create the file path.
import uuid
from hashfs.utils import shard
target_destination = Path(shard(
f"export_0x{uuid.uuid4().hex}.csv",
prefix="export_0x",
depth=2, width=2))Now the directory already contains thousands of file, I want to distribute them into sub directorys.
for f in list(Path("").iterdir()):
f.rename(shard(f.name, prefix="export_0x"))There was a problem hiding this comment.
In that case, I'd suggest copy/pasting the function into your own code instead of relying on this library for it.
No description provided.