Oculus-touch-controls & hand-tracking-controls combo: The controller does not appear in combination. (1.5 and master works)

[smeybi]

Description:
Just as information, in case someone wants to use this with A-Frame 1.6, it works in A-Frame 1.5!
master does not work, was a mistake, I have now edited it.

When using oculus-touch-controls and hand-tracking-controls together, the controller does not appear. It only shows up when the hands and controller are switched during the session, at which point the controller appears and functions. If the controllers are put down during the session and the hands are used for hand tracking, then when the controllers are picked up again, the controller appears and works. It just doesn’t show up at the start with the controllers.

• A-Frame Version: 1.6
• Reproducible Code Snippet or URL:

<a-entity oculus-touch-controls="hand: left" hand-tracking-controls="hand: left"></a-entity>

[dmarcos]

If it works on master then it will be fixed in the next version. Have you tried the very latest master build?

https://cdn.jsdelivr.net/gh/aframevr/aframe@ce9fd2321c97daf01bf9f098ad8b625c1603bc63/dist/aframe-master.min.js

Thanks

[smeybi]

Yes, it works in the master, i have tried!
I wanted to report it here so that someone who has the problem can see that it is a bug and has already been fixed, just not yet in version 1.6.
So he can use 1.5 or master until the new version is released.

[dmarcos]

Cool. Thanks for taking the time to report

[mrxz]

That's peculiar. I can reproduce the issue with the emulator, but I see it happening on master as well. Just tested it on a Quest 3 and I do see the reported issue even with master. Here's a live reproduction: https://glitch.com/edit/#!/jagged-three-cinema?path=index.html%3A3%3A138

Doing a git bisect leads to the following (dbdb9da):

dbdb9dacf2c6b62a1fdbd15e073efaed32cbd145 is the first bad commit
commit dbdb9dacf2c6b62a1fdbd15e073efaed32cbd145
Author: Diego Marcos Segura <[email protected]>
Date:   Fri Dec 1 10:24:50 2023 -0800

    Hide hand tracking controls entity when there's no hand tracking available

 src/components/hand-tracking-controls.js | 4 ++++
 1 file changed, 4 insertions(+)

Which explains the observed behaviour. The hand-tracking-controls hides the entire entity initially, and only makes it visible once hands are detected.

[dmarcos]

Some info from Discord:

Multimodal is supported for regular controllers and hands. 
To non-primary inputs will move from inputsources to trackedsources. 
So if you put the controllers down, 
hands will move from tracked to input and controllers from input to tracked

[dmarcos]

A-Frame is now only looking into xrSession.inputSource and not xrSession.trackedSource. We need to adjust

[dmarcos]

Spec as reference https://pr-preview.s3.amazonaws.com/cabanier/webxr/pull/1361.html#dom-xrsession-trackedsources

[dmarcos]

With trackedSource now containing the "secondary" input I would not expect multimodal input to work in any A-Frame version. I'm assuming Meta Browser. So not sure how 1.5 or master might be working

[mrxz]

Probably best to create a separate issue for trackedSources support, as this issue isn't about having hand-tracking and controller tracking simultaneous, but about a regression in supporting both in an experience. Annoyingly both can be referred to as "multimodal", but this issue is about "an app supporting multiple modalities" and the trackedSources would allow "using two modalities simultaneously" (e.g. one hand and one controller).

A-Frame is now only looking into xrSession.inputSource and not xrSession.trackedSource. We need to adjust

I'm not sure A-Frame should adjust. Reading the spec, the user agent determines which input sources are "active" and which are (only) "tracked". In theory the active input sources could already be multi-modal, but IIRC the Meta Browser only ever briefly exposed it as such and reverted this behaviour.

Tracked sources are intended to provide pose data. We should consider what the best way to surface them is, without overcomplicating things for the user. Simply treating them as if they were "active" input sources is going to cause issue. For example there could be 4 sources (hands + controllers) and the custom pinch detection logic of A-Frame would emit pinch events from the tracked hands even if the controllers are deemed the only "active" input sources by the browser.