Deposition path gets wiped during archive cleanup and we don't tell the user this will happen

[krivard]

When using --fsspec, the --deposition-path option receives the directory where the user expects files to be downloaded.

The orchestrator deletes the contents of this directory during cleanup, irrespective of whether those files were created by archiver or pre-date the run:

https://github.com/catalyst-cooperative/pudl-archiver/blob/43eacab34afb356ec430a698617e883a8eea568a/src/pudl_archiver/orchestrator.py#L29C1-L33C54

This could result in data loss if the user specifies a directory that is used for multiple purposes. We do not inform them of this risk in the README.

Potential resolutions:

1. Warn the user that the --deposition-path directory gets wiped during cleanup, so they know what to expect. Candidate locations:
   • README: Usage
   • README: Local testing
2. Create and destroy our own directory within --deposition-path so it's guaranteed to be used only by us

[jdangerx]

Thanks for writing this up! I would personally vote for "don't do something scary" over "warn the user that the code does something scary, then do the thing anyway."

Creating a temp directory seems like a good enough workaround. If y'all want to take this on Margay, go for it - otherwise I'm happy to pull it into Inframundo-land for next sprint.

[jdangerx]

@mariahjrogers this might be a good issue to jump into, instead of #193 - let me know if this piques your interest!