Temp dir for cloud storages

[sunilke]

Hello , I had question like why do we require temporary directory for cloud storages as anyways all of them supports in built atomic uploads and also we have not enabled atomic upload with resume mode for cloud storage.

In cloud storages docs, for eg this doc for s3(https://github.com/drakkan/sftpgo/blob/main/docs/s3.md) says : local home directory is still required to store temporary files , why can't we upload files/objects directly to s3 as anyways we are dependent on atomicity of cloud providers(here s3)?

Please help me with this questions and exact use case of temporary/home directory requirement for cloud provider storages. Also correct me if my understandings are wrong. Thanks

[drakkan]

Hi,

we want to support multipart downloads/uploads and so we have to provide a random access file like object to the aws sdk (other cloud providers have a similar requirement).

To provide a random access reader/writer we use an unliked file that is stored in the local home directory. An unliked file will be deleted when SFTPGo closes it or if the SFTPGo process terminates, so it is guaranteed that it will never remain on the disk once the transfer is completed.

An alternative approach is to use an in memory pipe like this but in this case you have to set the concurrency to 1 as you can see here and so you will have slower transfers.

I also evaluated to develop an in-memory file like object but suppose this real case (it was reported in the past):

• the aws sdk uploads 5 parts in parallel
• the first part fails while the other for 4 are ok
• the sdk will upload other parts so basically all the file was uploaded except the first part
• after a minute or so the sdk detects the failure and try to read again the first part

with an unliked file this is not an issue, we still have the unliked file on the local disk. With a memory file like object we have to find a reliable way to understand if a part was successfully uploaded (with all the cloud sdks) and free the memory once this happens.

The only downside of the unliked file approach is that if you need to upload a 10GB file you need 10GB of local disk space.

I don't exclude that in the future I will work on a memory file like object for cloud based providers (or someone will send a patch), but for now you need a local home dir.

My explanation is a bit technical, I hope it is clear enough.

[sunilke]

One follow up doubt:
Here you said that to upload a 10GB file we need 10GB of local disk space so that means when user requests to upload file of 10gb , that file upload will begin on temp directory and from there the aws sdk will start reading(and writing on the bucket) it.
So total bandwidth used will be 20 gb?

[drakkan]

One follow up doubt: Here you said that to upload a 10GB file we need 10GB of local disk space so that means when user requests to upload file of 10gb , that file upload will begin on temp directory and from there the aws sdk will start reading(and writing on the bucket) it. So total bandwidth used will be 20 gb?

A user uploads 10GB to the SFTPGo server via SFTP/WebDAV/FTP etc, then SFTPGo uploads 10GB to S3, using an unlinked file or keeping the 10GB in memory doesn't change anything here.

[sunilke]

understood , thanks.

[sunilke]

Thanks for your prompt response and on the point explanation. Will come back in future if I have any more doubts around this.