OpenXR: Add support for hand tracking source extension

[BastiaanOlij]

Originally the hand tracking extension in OpenXR left a few behaviors undefined. As a result only Valves runtime provided hand tracking data both when the user was using a controller (where the runtime infers tracking data from inputs and sensors) or when full optical hand tracking was available.
Other runtimes would only provide hand tracking data if actual hand tracking was used and no data when the user used a controller.
This meant that it was up to the developer to determine if hand tracking data was available and implement a fallback.
As OpenXR does not provide enough information to fully infer finger positions from controller input this was limited.

The need for the XR runtime to provide inferred hand tracking data when controllers were used has resulted in more runtimes willing to provide this feature however there needed to be a way for the runtime to communicate capabilities, and for the game to inform the runtime what it can handle.

The Hand Tracking Data Source Extension was introduced for this purpose. Enabling this communicates to the XR runtime that we want to use the new functionality and we specify which data sources we wish to receive hand tracking data for.

When polling hand tracking data we can also retrieve the source in use.

If hand tracking is enabled (see OpenXR project settings) and this extension is available, we will enable the extension. We will always request both available data sources.
The new method OpenXRInterface::get_hand_tracking_source allows us to query the source of the hand tracking data.

[BastiaanOlij]

This extension is now supported on Quest 3 (and probably 2 and Pro as well), was able to test this successfully.

Both inferred and optical hand tracking is working with the Quest 3 and it switches seamlessly between them.

Tested with a slightly modified version of:
godotengine/godot-demo-projects#973

[dsnopek]

Skimming the code, this looks good to me!

Tested with a slightly modified version of:
godotengine/godot-demo-projects#973

Could you share your modified version of that demo? I'd be happy to test this :-)

[BastiaanOlij]

Could you share your modified version of that demo? I'd be happy to test this :-)

Oh, it's just removing the hand interaction profile that I added, and the call that checks for this extension.

[BastiaanOlij]

@dsnopek project was to big for Github so I uploaded the modified project to dropbox. Quest correctly switches between inferred and full hand tracking but it's not showing the correct data source, so I'm having a look at why atm.

You do need to add the vendors plugin manually (and install the compiled templates)

https://www.dropbox.com/scl/fi/cgqwevo82ggpu4p7b65h4/handtracking-source-demo.zip?rlkey=dvnjxvt8mix2lftu811t9cath&dl=0

[BastiaanOlij]

@dsnopek there is an error in the start_vr.gd of that project where it quits Godot on session end. That was an old experiment of mine where I drew the wrong conclusion :)

On further testing I am, as before, getting hand tracking info but I'm not getting useful information from get_hand_tracking_source. Still trying to figure out if I made a mistake or if Meta (I'm testing on Quest 3) doesn't fully support this.

[dsnopek]

@BastiaanOlij Thanks!

I'm seeing the same as what you're seeing with the demo project, on both Quest 3 and Quest Pro. I poked around a little bit, and the headset does seem to be reporting that it supports this extension, and so it should be initialized when calling xrCreateHandTrackerEXT(). However, it seems we always get false on data_source_state.isActive, which seems strange, since we are getting valid location data for joints despite the spec saying this:

So, either we've got a subtle mistake somewhere, or Meta is not strictly following the spec?

[BastiaanOlij]

@dsnopek I've asked some questions but haven't got any feedback other than "it should work", by now you have better contacts to poke then me :P

For me the important thing is that hand tracking works even when you're using a controller so that we finally have parity between SteamVR and Oculus (SteamVR doesn't need this extension, it works like this by default).

Getting the data source name is bonus, I don't think that part is supported by anyone but Meta atm.

[dsnopek]

@BastiaanOlij I figured it out!

We were passing the XrHandTrackingDataSourceStateEXT struct into the next pointer on the wrong struct. It's supposed to go on XrHandJointLocationsEXT, not XRHandJointsLocateInfoEXT.

Apply this patch, and it starts working beautifully :-)

[BastiaanOlij]

@dsnopek thanks! I need to look into OpenXRs validation layers so it will spit out mistakes like this.

Works like a charm now, I think this is ready to be merged!

[dsnopek]

The latest version works great! The code looks good too :-)

[akien-mga]

Thanks!