How to load files from a cloud storage directly into a indexing pipeline

[celha]

Hi,

I'm currently wondering what the best practice is for loading files into a pipeline that aren't stored locally. For example, I have several files in a s3 bucket and want to load them directly from the cloud. I want to prevent downloading them first from s3 and then loading them into the pipeline.
Any ideas? Did I miss something?

Thank you in advance!

[vblagoje]

@celha imho you might be overthinking this. It should be simple to download files directly from s3 and then index them. I don't think we provide some out of the box component for this, but it shouldn't be hard.
We are currently working on some integration that will make these kinds of data imports easy in Haystack 2.x
Hopefully, you made some progress already.

Best,
Vladimir