Hand tracking choppy on Vision Pro

[dmarcos]

It feels jittery / low framerate. Any other headset is fine (e.g Quest). Can test on https://aframe.io/aframe/examples/showcase/hand-tracking/ and compare Quest3 and Vision Pro. It's difficult to capture on video but Quest 3 hand tracking much smoother

[dmarcos]

Also hand model is distorted. Might be something particular to Vision Pro joint tracking we're not accounting for

[dmarcos]

Thoughts about distortion. I vaguely remember saying @rcabanier mentioning that Quest doesn't return the joints data that exactly match user's hands for privacy reasons. Instead a generic hand size is returned. Wonder if the discrepancies I'm seeing are related to the different approaches between Meta Browser and Safari. It's subtle and hard to capture in video.

[dmarcos]

Did a second test and Quest 3 hand tracking definitely feels less jittery than Vision Pro with the example linked above: https://aframe.io/aframe/examples/showcase/hand-tracking/

This might not be a bug in Safari or A-Frame just differences between hand tracking systems.
I captured a video in each headset to compare. Ignore the color differences (separate issue) Videos were taking in same room, in same lighting conditions just a few mins apart.

Vision Pro

handTrackingAVPSmall.mp4

Quest 3

handTrackingQuest3.mp4

[dmarcos]

Read somewhere that hand tracking only runs at 30Hz on Vision Pro. Might be part of the issue.

[Clodo76]

I also made a comparison video: https://www.youtube.com/watch?v=bcdmz-gW7HM
(real and virtual synced manually, so don't judge it by lag).

Vision Pro is more accurate when i join hands.

Anyway, the issue on this topic is a very known problem of VisionPro: it update hand-tracking only at 30hz.

[dmarcos]

looks like hand tracking will get 60Hz updates in Vision OS 2. Hopefully this improves

[mkeblx]

Hand tracking on visionOS 2 should be updated at display refresh rate / 90Hz.

visionOS 2 video:
https://github.com/aframevr/aframe/assets/293795/d8f9e202-c06f-42e1-9672-5ad91d9c44bf

[dmarcos]

@mkeblx thanks. do you notice an improvement?

[dmarcos]

@mkeblx Can we considered this fixed in Vision OS 2?