Video with preprocessor VirtualBackground

[kiliancabrera]

• I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• I verified that the Quickstart application works in my environment.
• I am not sharing any Personally Identifiable Information (PII)
  or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Code to reproduce the issue:

import { useEffect, useCallback } from 'react'
import {
  VirtualBackgroundProcessor,
  ImageFit,
  isSupported,
  Pipeline,
} from '@twilio/video-processors'

import background from './videocall-call-view_3.png'

const rawImagePaths = [background]

let imageElements = new Map()

const getImage = (index) => {
  return new Promise((resolve, reject) => {
    if (imageElements.has(index)) {
      return resolve(imageElements.get(index))
    }
    const img = new Image()
    img.onload = () => {
      imageElements.set(index, img)
      resolve(img)
    }
    img.onerror = reject
    img.src = rawImagePaths[index]
  })
}

const virtualBackgroundAssets = '/assets/virtualbackground'
let virtualBackgroundProcessor

export default function useBackgroundSettings(videoTrack, room, isAdmin) {
  const removeProcessor = useCallback(() => {
    if (videoTrack && videoTrack.processor) {
      videoTrack.removeProcessor(videoTrack.processor)
    }
  }, [videoTrack])

  const addProcessor = useCallback(
    (processor) => {
      if (!videoTrack || videoTrack.processor === processor) {
        return
      }
      removeProcessor()
      videoTrack.addProcessor(processor, {
        inputFrameBufferType: 'video',
        outputFrameBufferContextType: 'webgl2',
      })
    },
    [videoTrack, removeProcessor]
  )

  useEffect(() => {
    if (!isSupported || !isAdmin) {
      return
    }

    const handleProcessorChange = async () => {
      if (!virtualBackgroundProcessor) {
        virtualBackgroundProcessor = new VirtualBackgroundProcessor({
          assetsPath: virtualBackgroundAssets,
          backgroundImage: await getImage(0),
          fitType: ImageFit.Cover,
          pipeline: Pipeline.WebGL2,
        })
        await virtualBackgroundProcessor.loadModel()
      }
      if (!room?.localParticipant) {
        return
      }

      addProcessor(virtualBackgroundProcessor)
    }
    handleProcessorChange()
  }, [videoTrack, room, addProcessor, removeProcessor])
}

As of today I have detected that the video with the background processor suffers a considerable drop in performance and it is impossible to make a video call, this with the same version and everything exactly the same did not happen a few days ago, can someone confirm that it is also happening?

Software versions:

• Browser(s): 113.0.5672.63
• Operating System: MacOs and Windows 11
• twilio-video.js: 2.27.0
• twilio-video-preprocessor: 2.0.0
• Third-party libraries (e.g., Angular, React, etc.):

[manjeshbhargav]

Hi @kiliancabrera ,

Sorry for the delayed response. Can you share your application so I can try and reproduce it locally?

[Lisztos]

This one seems highly related to this one: #2007