How to use experimental InMemoryChatMessageStore in a real server environment with multiple users

[wirtsi]

So I am still trying to wrap my head around how to use the InMemoryChatMessageStore in a real world scenario. What I have is

• FastAPI server initiates websocket request with client
• Inside the async FastAPI coroutine I call my haystack pipeline
• Inside the coroutine, I do the new Pipeline() and attach the components

By doing this, every user gets his "own" pipeline including the message store.
Unfortunately building the pipeline takes around 20s to build for every request, so you don't want to do that.

My assumption would be that you rather instantiate the pipeline once and then run that instance inside the coroutines. But this would require that the InMemoryChatMessageStore is able to distinguish between different coroutines/users that are using it.

I was thinking of passing a lambda to the Retriever / Writers that pull the correct store ... is that a feasible approach or am I just missing something with the component?

[vblagoje]

@wirtsi you need to do your own threading for each user and share the pipeline between threads. We are working on async support that will enable another, perhaps more suitable, model of handling concurrent requests on a pipeline. Async support should be launched soon - this quarter.