video-processor pipeline WebGL2 throws TypeError: can't access property "VERTEX_SHADER", gl is null

[thlee1122]

I found a bug with WebGL2.
We recently upgraded the twilio-video-processor to the latest version, 2.0.0.
With the above-updated version, the WebGL2 pipeline video background blur functionality throws a TypeError: can't access property "VERTEX_SHADER", gl is null.

Below is the screenshot of the error:

Software versions:

• Browser(s): Chrome (Version 114.0.5735.106)
• Operating System: MacOS Ventura 13.4
• twilio-video-processors.js: 2.0.0
• twilio-video: 2.27.0
• Third-party libraries (e.g., Angular, React, etc.): React

[alex93293]

I managed to find the root cause for this.

The default pipeline for the video processors is set to WebGL2, but the default processor options in twilio-video is set to { inputFrameBufferType: 'offscreencanvas', outputFrameBufferContextType: '2d' }, which are options for Canvas2D.

There are 2 ways to fix this:

1. Pass WebGL2 options to track.addProcessor

track.addProcessor(processor, { inputFrameBufferType: "video", outputFrameBufferContextType: "webgl2" });

2. Use Canvas2D pipeline*

processor = new VirtualBackgroundProcessor({
  pipeline: Pipeline.Canvas2D,
  ...options
});

I only managed to find this out by reading through the example.

*Canvas2D might not have full functionality with Safari - the Gaussian Blur didn't work for Safari 16.5.1 MacOS (haven't tested others)

[thlee1122]

@alex93293 Thank you for this! This looks like a work around. Is the Twilio video processor team going to release an actual fix for this issue?

[lgenzelis]

Also worked for me! Does anyone knows which one should we be using? (webgl2 or canvas2d) I mean, since the defaults are different for @twilio/video-processors and twilio-video.

As a sidenote, I'm sticking with canvas2d for now, since webgl2 completely ignores the fitType option (and having a ImageFit.Contain is must for me).

[sajitkhadka]

I had the same issue when I upgraded to v2.0.0.
The following seemed to work for me but it was painfully slow. so I ended up using Canvas2D pipeline option instead.
track.addProcessor(processor, { inputFrameBufferType: "video", outputFrameBufferContextType: "webgl2" });

Edit: It seems for higher resolution it is slow. width: 1280, height: 720,
if I change width:640, height:480. it seems to be okay.

[luisrivas]

Thanks for reporting this issue. We just released the v3.x of the video processors, where the Canvas2D and WebGL2 pipelines have been replaced by a hybrid pipeline. Please update to the most recent version. If you continue to experience any problems after upgrading, please let us know, and we will be happy help you further.