A streaming proxy for LLM answers

[sciabarracom]

Why you need it?

LLm applications require a way to stream the results as they are generated

How it could be?

We need a stereaming proxy. It should work this way.

Since actions while produce input cannot communicate back with the frontend, they should write the temporary content in redis.

The proxy is mounted on /stream of each user with an ingress, and when it receive a request /stream/<key> it will stream the content as it arrives in the key <user>:<key>

Other related information

We should explore the best way to execute the request.

We should be able to execute a request liket how can I become rich and forget.
Normally openwhisk returns an activation id, and the result should be written in redis using this id for the streaming . So an asyncrhonous request should be more or less:

1. webaction invoking an asyncrhonous request and returning the activation id
2. action talking to an LLM and writing the result to redis in the key corresponding to the activtion id

[francescotimperi]

An idea could be to use Redis/Valkey Streams. (an example here )

[sciabarracom]

please check #82 as this spec is outdated