WebRTC and ML

[ndrean]

This article uses an Elixir implementation of the Web Real-Time-Communication API to perform (pseudo) real-time object detection via a ML model.

Due to the latency of ML processes, this should be run only every nth frame.

A word about WebRTC

The WebRTC allows P2P communication and - as explained in the mdn link - is build on top of:

• the RTCPeerConnection interface, to start a connection between peers (SIGNALING),
• the MediaStream interface, a stream of audio or video content, and/or
• the RTCDataChannel interface, bidirectional peer-to-peer data exchange.

As a side note, websockets uses TCP whilst webRTC uses UDP (faster but less reliable).

The blog and YT video

The ML model is a Hugging Face model.
They use WebRTC to capture a video stream to be sent to the server.
The connection between the browser and the Phoenix server is made via a - dynamically supervised - channel attached to the socket.
It uses Bumblebee to create an Nx.Serving from the model. It will receive a frame as an input from the channel and push back the result.

Youtube

[ndrean]

Take a look at the Xav README.

I opened an issue as I cannot use it on my Mac.