Cross-origin restriction circumvention?

[WofWca]

We can't directly analyze audio from video elements whose source is cross-origin, although they can be played back just fine. And this applies to many websites (Zoom call recordings, Invidious, Google Drive videos, many others).

Currently what we're doing is suggest opening the source directly in a new tab:

But would be cool if it just worked somehow. Preferably without violating what's cross-origin restriction is made for -

leaking information to the script where the cross-origin media element is played - you could say that information about which parts are silent and which are not could be considered that? But e.g. accessing `el.duration` can be considered this too? So maybe the cross-origin restriction in itself doesn't make too much sense in the first place? Or maybe it must also restrict the loading and playback of the media at all?)

Somewhat of an option is to contact the website owners and ask them to apply the required cross-origin headers to the source (I think it's Access-Control-Allow-Origin?) but you can't contact them all.

More info:

• The responsible Web Audio API part of the spec: https://webaudio.github.io/web-audio-api/#MediaElementAudioSourceOptions-security
• An issue related to this, with some suggested solutions: A way to a) detect if MediaElementAudioSourceNode is CORS-restricted & b) revert createMediaElementSource WebAudio/web-audio-api#2453
• HTMLMediaElement's spec, search for "cross": https://html.spec.whatwg.org/multipage/media.html#security-and-privacy-considerations
• Same issue in a similar extension: MediaElementAudioSource outputs zeroes due to CORS access restrictions vantezzen/skip-silence#71

Advice or thoughts are appreciated (as always).

[LoneCoder21]

wait how did u detect the likelihood of media being unsupported?

[WofWca]

@LoneCoder21

It's this check

jumpcutter/src/entry-points/content/helpers/isSourceCrossOrigin.ts

Line 37 in 81b4e50

return elCurrentSrcUrl.origin !== document.location.origin;

See the comment:

jumpcutter/src/entry-points/content/AllMediaElementsController.ts

Lines 96 to 115 in c63821f

	function getAppropriateControllerType(
	settings: Pick<Settings, typeof controllerTypeDependsOnSettings[number]>,
	elementSourceIsCrossOrigin: boolean,
	): ControllerKind {
	// Analyzing audio data of a CORS-restricted media element is impossible because its
	// `MediaElementAudioSourceNode` outputs silence (see
	// https://webaudio.github.io/web-audio-api/#MediaElementAudioSourceOptions-security,
	// https://github.com/WofWca/jumpcutter/issues/47,
	// https://html.spec.whatwg.org/multipage/media.html#security-and-privacy-considerations),
	// so it's not that we only are unable to analyze it - the user also becomes unable to hear its sound.
	// The following is to avoid that.
	//
	// Actually, the fact that a source is cross-origin doesn't guarantee that `MediaElementAudioSourceNode`
	// will output silence. For example, if the media data is served with `Access-Control-Allow-Origin`
	// header set to `document.location.origin`. But currently it's not easy to detect that. See
	// https://github.com/WebAudio/web-audio-api/issues/2453.
	// It's better to not attach to an element than to risk muting it as it's more confusing to the user.
	return elementSourceIsCrossOrigin && settings.dontAttachToCrossOriginMedia
	? ControllerKind.ALWAYS_SOUNDED
	: settings.experimentalControllerType

So it's not like we detect an actual likelihood expressed as a number between 0 and 100%, but rather "it's cross-origin, so it's probably not gonna work".

[LoneCoder21]

Oh i see. I had this issue in my extension too so I'll just use this option 😞