Skip to content

Latest commit

 

History

History
476 lines (306 loc) · 14.8 KB

index.md

File metadata and controls

476 lines (306 loc) · 14.8 KB
marp theme paginate backgroundColor footer
true
default
true
![width:50px](gfx/w3c.png) Second Screen WG/CG virtual meeting

Virtual Display Use Case Exploration

width:100px

A Collection of Interesting Ideas

Second Screen WG/CG virtual meeting

8 June 2022

Anssi Kostiainen

bg width:400px right:35%

<style> table, th, tr, td { border: 0px !important; border-collapse: collapse !important; border:none !important; outline:none !important; } </style>
<iframe width="533" height="300" src="https://www.youtube-nocookie.com/embed/rmg5tZ4iSx8?start=34" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Deskreen demo

 <iframe width="533" height="300" src="https://www.youtube-nocookie.com/embed/IbN9f8uNT2M?start=182" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Sidecar demo

Virtual Display?

"a display instance without a physical connection to the graphics card"

"a software representation of a physical display"

UX Considerations 🤩

1️⃣ Display Arrangement

bg width:600px bg width:600px

2️⃣ Display Source Picker

width:600px

(Chrome display source picker UI, https://www.webrtc-experiment.com/getDisplayMedia/)

3️⃣ Virtual Display Creation

width:800px

(Deskreen Virtual Display Picker UI, https://deskreen.com/)

2️⃣ getDisplayMedia UI vs 3️⃣ Deskreen Virtual Display UI

bg width:600px bg width:600px

4️⃣ Window Placement

Implicit consent example: drag and drop

bg width:800px

OSes have a variety of implicit consenting mechanisms

  • Keyboard shortcuts in Windows: "Windows logo key + Shift + Left arrow or Right arrow to move an app or window in the desktop from one monitor to another."

  • Drag and drop evolved in macOS: "Drag the app window you want to move to the edge of the screen [or] enter Mission Control, then drag the window up to the space you want to use."

Could such implicit permission grants work for web app window placement?

The traditional Web API approach: user's consent via browser prompt

bg

bg

OS-level: controls built into window manager

bg

bg

Browser prompt vs OS controls

bg width:600px bg width:600px

Goal: Enable UX innovation and protect the user's privacy

How? Set a strong privacy baseline & leave room for Quality of Implementation differentiation?

Virtual Display support for the web

Ideation 🧪

Goals

  • "Let me create a new virtual display surface & place windows on it, get its relative positioning and a representation I can pass on to other Web APIs."
  • MediaStream as the "common currency" for getDisplayMedia & WebRTC interop
  • Reuse existing API constructs and browser and OS UI/UX

Constraints

  • Agnostic with regard to the display technology
  • Meet security and privacy expectations

Assumptions

  • Web users expect multi-screen use cases to work similarly with Virtual Displays
  • Web APIs shouldn't(?) fully abstract away virtual/physical display boundary

Requirements (to make this a 1st class web capability) 👋

  • A new virtual display surface type that maps to an OS-level concept
  • Allow {extending|mirroring} of {monitor|window|browser} display surface to a virtual display surface
  • User-initiated and programmatic window placement
  • Use platform-native UX concepts (e.g. drag/move windows across, share controls)

Related work

  • Platform-level: open-source projects (Deskreen), mainstream product features (Sidecar) and more...

Related, but different use cases

Deskreen Experiments 🧪

Use Case 1️⃣: Browser of the tablet as an ad-hoc Virtual Display

"Sometimes I want to use the display of my tablet along with my primary device's display and let web apps make use of both. I want to be able to control what web content to present on my tablet."

Download video as .mp4

Use Case 2️⃣: TV as an ad-hoc Virtual Display

"Sometimes when lounging in the living room I want to extend my workspace to the TV. I want to stay in control of what web content to share on the TV to a larger audience."

Download video as .mp4

1️⃣ Prototype Implementation Details

  1. Deskreen mirrors a virtual display (created with BetterDummy) to a browser window on a tablet
  • WebRTC MediaStream
  • Same LAN

Deskreen Architecture

height:500px

See also: session initiation sequence diagram

2️⃣ Prototype Implementation Details

  1. Deskreen mirrors a virtual display (created with BetterDummy) to an offscreen browser window
  2. Presentation API mirrors that offscreen browser window to a TV connected to a Chromecast (1-UA mode triggered via browser UI)

Design Noodling 🍜

1️⃣ Display Arrangement

⚙️ OS & Platform APIs - built-in feature, some API gaps

🌍 Web APIs

2️⃣ Display Source Picker

🌍 Web APIs

getDisplayMedia extension ideas (reusable for picking multiple virtual displays?):

3️⃣ Virtual Display Creation

⚙️ OS & Platform APIs - "advanced users only"

🚫 Web APIs

API Ideation 💡

Reusable concepts

getDisplayMedia extensions

IDL sketch

partial interface MediaDevices {
  Promise<MediaStream> getDisplayMedia(optional DisplayMediaStreamConstraints constraints = {});
};

enum DisplayCaptureSurfaceType {
  "monitor",
  "window",
  "browser",
  "virtual" // new
};

Considered Alternatives

New "getDisplaySurface" API

  • Redundant if getDisplayMedia extensions satisfy reqs

Presentation API extension

Remote Playback API extensions

  • Remote Playback API via HTMLMediaElement.srcObject = ms too much indirection

Possible path from an Exploration onwards

  • 1️⃣ width:40px   Explainer 🔗

    • Use cases, reqs, API sketch, considered alternatives     📍We are here!

  • 2️⃣ width:40px   Explainer review

    • Web community, browser & OS vendors, hardware ecosystem

  • 3️⃣ width:40px   Draft web spec

    • W3C Second Screen CG in collaboration with WebRTC WG?

  • 4️⃣  🧪 Prototype in code
  • 5️⃣  📢 Prototype feedback

    • Web developer feedback can change the shape of the API significantly

  • 6️⃣ width:40px   Standards track

    • Subject to support, possible WG adoption

</html>