An SSE streamer for LLM , v2

[sciabarracom]

Spec for the streamer.

The streamer is available as stream.<apihost> over https.

Optionally we may have also <user>.<apihost>/stream that is acutually proxying to stream.<apihost>/<user>

An user will stream an action <package>/<action> with:

const url = "/stream/<package>/<action>"
// or "https://stream.<apihost>/<user>/<package>/<action>"
const evtSource = new EventSource(url, {
  withCredentials: true,
  // other parameters here
});
evtSource.onmessage = (event) => {
  const data = event.data
  // do something with the event data
};

The steamer once received the request, assuming it is a json post,

• parses the json
• opens a socket for listening in a random port
• invoke the action <user>/<package>/<action> asynchronously passing to the data additionally in the args the extra parameters STREAM_HOST and STREAM_PORT
• wait in the socket the user writes the content to be streamed make a buffer
• writes the content in the strem when it arrives
• closes the socket once done

The user will use the stremer as follows:

def main(args):
    streamer = (args.get("STREAM_HOST"), args.get("STREAM_PORT"))
    if not socket[0] or not socket[1]:
        return {"body": "pleae invoke with the stremer"}
    // invoke a call to a streaming api server like OpenAI
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect(streamer)
        while <there are data>:
                data = <read data>
                s.sendall(data)
    return {"body": "done"}

[d4rkstar]

this task was completed