Haystack with ElasticSearch cold tier

[TheRabidWolverine]

Hi, I can see that there are tutorials on how Haystack can be integrated with ElasticSearch. However, ES has various tiers of data available - warm, cold, frozen, etc. Does Haystack work with the cold data (almost entirely on disk)? Does it work at all, and if so, will performance take a hit? And will that Haystack cluster need to be hosted on a node with memory big enough to fit the embeddings from the cold tier in it as well, or just the warm (in-memory) part of the overall ES stack with the cold part fetched from disk during runtime query? Either way, any proof of concept, or tutorial pointing towards the same will be helpful.

[julian-risch]

@TheRabidWolverine With Haystack, you'll typically work with text data and not with time series data such as logs and metrics. For that reason, the data you have indexed typically doesn't really age in the sense that the data doesn't become less relevant over time. The ElasticSearch tier for that kind of text data is the content tier.
Text data in Haystack usually doesn't become less and less frequently queried over time and thus doesn't age. So you wouldn't move the data from warm to cold to frozen but always keep it in content. If you still moved your text data manually to these tiers you can expect lower search speed, yes. Haystack shouldn't be affected by the tier you are using in any other way.
Not an expert of tiers in ElasticSearch here so you might get better answers from ElasticSearch support channels. 🙂

[TheRabidWolverine]

Thanks a lot! So imagine millions of documents are being fed to Haystack, it shoud be able to answer based on that content. So how do we host Haystack? What are its memory requirements, say to host the entire Wikipedia corpus? Will that same hosted Haystack instance also be usable to generate "related questions" and their answers module as well based on a given question? Will thr hosted Haystack instance store all content in memory, or can it be part memory and part access to a document corpus, either raw, or index, like ES?