Playback and synchronization of MediaStreamTracks created by VideoTrack Generator

[aboba]

VideoTrackGenerator (the successor to MSTG) converts a stream of VideoFrames to a MediaStreamTrack.

VideoFrames may have been "transformed" by a process that generates jitter. For example, they may have been processed by a Machine Learning algorithm that takes a varying amount of time, or they may have been produced by a decoder that received encoded chunks over a network with varying delays.

Currently, the specification does not talk about what an application can expect of VideoTrackGenerator in terms of playout delay and rendering smoothness. Does VideoTrackGenerator assume that there is a jitter buffer upstream that generates VideoFrames with constant delays? Or does VideoTrackGenerator adjust the playout delay in response to the jitter it experiences in the VideoFrames it receives?

[tonyherre]

Should the VideoFrame's timestamp relate to this as well? The webcodecs spec defines it as "The presentation timestamp", with a note about potentially using it to guide rate control in the encoder.
It would be great if we could specify its behaviour in the context of mediacapture transforms.

[alvestrand]

I think the timestamps in the stream coming from VTG are the timestamps coming into the VTG - dejittering should happen just before playback (or other process that needs a regular cadence).

Needs discussion.

[dontcallmedom-bot]

This issue was discussed in Joint Media/WebRTC WG meeting at TPAC – 26 September 2024 (Playback and sync of tracks created by VideoTrackGenerator)

[dontcallmedom-bot]

This issue was discussed in WebRTC November 19 2024 meeting – 19 November 2024 (Issue #86: Playback and sync of tracks created by VideoTrackGenerator)