Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

OpenXR

The SenseGlove Unreal Engine Plugin has provided OpenXR-compatible hand tracking by implementing XR_EXT_hand_tracking since v2.1.0.

Typically a user does not need to know anything about OpenXR to use the plugin, so this section of the handbook is for advanced users who are looking for a way to directly consume the OpenXR data coming directly from either a SenseGlove device or if enabled in the plugin settings from hand-tracking.

Since the SenseGlove Unreal Engine Plugin registers itself as an OpenXRHandTracking device it becomes a hand-tracking provider for Unreal Engine, thus the OpenXR data from SenseGlove could always be retrieved from the Unreal Engine's IXTrackingSystem with two caveats:

  • The first caveat is, if another OpenXR-compatible hand-tracking plugin, e.g. Epic's own OpenXRHandTracking, is enabled simultaneously it's not guaranteed that the FXRHandTrackingState struct retrieved from the IXTrackingSystem::GetHandTrackingState() method is coming from SenseGlove, as these methods return the first hand-tracking plugin they could find. Thus, SenseGlove provides its own implementation of GetHandTrackingState() which guarantees the retrieved FXRHandTrackingState is coming from the SenseGlove Unreal Engine Plugin; and this is the preferred way to that.
  • The second caveat is, Unreal IXTrackingSystem::GetHandTrackingState() method is blind to SenseGlove's wrist-tracking settings and offsets, therefore the returned FXRHandTrackingState retrieved using this engine function won't take into account the wrist-tracker offsets, and the hands end up at the wrong orientation or position. In contrast,

Important

In order to retrieve the latest FXRHandTrackingState available, The SenseGlove Unreal Engine Plugin provides an alternative implementation for IXTrackingSystem::GetHandTrackingState(), which guarantees the OpenXR hand-tracking data is coming from a SenseGlove device and also takes into account the SenseGlove's wrist-tracker offsets automatically.

This is the recommended approach over Unreal Engine's own IXTrackingSystem::GetHandTrackingState() or you have to ensure the data received is coming from a SenseGlove device yourself, and also take into account the SenseGlove's wrist-tracker settings and calculate the wrist offsets either using one of the SGHapticGlove::GetWristLocation() variants, or manually.

In short IXTrackingSystem::GetHandTrackingState() does not respect the offsets from Project Settings > SenseGlove > Tracking > Wrist-Tracking Settings

Caution

In order to retrieve the latest FXRHandTrackingState available, The SenseGlove Unreal Engine Plugin provides an alternative implementation for IXTrackingSystem::GetHandTrackingState(), which guarantees the OpenXR hand-tracking data is coming from a SenseGlove device and also takes into account the SenseGlove's wrist-tracker offsets automatically.

This is the recommended approach over Unreal Engine's own IXTrackingSystem::GetHandTrackingState() or you have to ensure the data received is coming from a SenseGlove device yourself, and also take into account the SenseGlove's wrist-tracker settings and calculate the wrist offsets either using one of the GetWristLocation() variants, e.g., SGHandLayer::GetWristLocation(), SGHapticGlove::GetWristLocation(), etc, or manually.

In short IXTrackingSystem::GetHandTrackingState() does not respect the offsets from Project Settings > SenseGlove > Tracking Settings > Wrist Tracking Settings.

Tip

Starting with version v2.8.0, the SenseGlove Unreal Engine Plugin provides a highly convenient, high-level abstraction that eliminates all of the manual steps described above. By using SGHandTrackerComponent you can simply add the component to your Pawn class (or any actor that requires hand-tracking data), configure its properties, and retrieve the tracking data with a single function call when needed.

The SenseGlove UE Plugin automatically handles all required settings and offset calculations for your positional tracking hardware, whether you are using pure hand tracking or a SenseGlove device. It also provides an optional debug hand out of the box, allowing you to instantly visualize hand-tracking data without writing a single line of code.

Furthermore, version v2.8.0 ships with a companion component, SGHapticsComponent, which makes it easy to add haptic feedback to your own custom or third-party hand manipulation systems.

Tip

Sometimes, using IXTrackingSystem::GetHandTrackingState() is unavoidable, e.g., when using a third-party hand manipulation system such as VR Expansion Plugin (VRE), which internally relies on IXTrackingSystem::GetHandTrackingState() to retrieve the hand-tracking data. In that case SenseGlove provides methods such as SGHandLayer::GetWristLocation(), SGHapticGlove::GetWristLocation(), etc, which you can use to reliably calculate the wrist offsets:

// Get the OpenXR hand-tracking data for the right hand
FXRHandTrackingState HandTrackingState;
IXTrackingSystem::GetHandTrackingState(
    GetWorld(),
    EXRSpaceType::UnrealWorldSpace,
    EControllerHand::Right,
    HandTrackingState);

// Return if the struct data is invalid!
if (!bGotHandTrackingState || !HandTrackingState.bValid)
{
    return;
}

// Return if the device is not being tracked!
if (HandTrackingState.TrackingStatus == ETrackingStatus::NotTracked)
{
    return;
}

// Ensure that HandTrackingState.HandKeyLocations has the location data
// for 26 joints!
if (!ensureAlwaysMsgf(HandTrackingState.HandKeyLocations.Num()
                      == EHandKeypointCount,
                      TEXT("Invalid HandKeyLocations count!")))
{
    return;
}

// Ensure that HandTrackingState.HandKeyRotations has the rotation data
// for 26 joints!
if (!ensureAlwaysMsgf(HandTrackingState.HandKeyRotations.Num()
                      == EHandKeypointCount,
                      TEXT("Invalid HandKeyRotations count!")))
{
    return;
}

{
    // FQuat variant of FSGHandLayer::GetWristLocation()
    // Get the OpenXR hand-tracking data with the correct wrist offsets for
    // Meta Quest 3 Controllers
    FVector WristLocation;
    FQuat WristRotation;
    FSGHandLayer::GetWristLocation(
        true, // true for a right-handed glove and false for a left-handed one
        HandTrackingState.HandKeyLocations[0], // 0 is the Wrist joint location
        HandTrackingState.HandKeyRotations[0], // 0 is the wrist joint rotation
        ESGPositionalTrackingHardware::Quest3Controller,
        WristLocation, WristRotation);

    // WristLocation and WristRotation variables now contain the correct
    // OpenXR hand-tracking data with the wrist offsets applied correctly
}

{
    // FRotator variant of FSGHandLayer::GetWristLocation()
    // Get the OpenXR hand-tracking data with the correct wrist offsets for
    // Meta Quest 3 Controllers
    FVector WristLocation;
    FRotator WristRotation;
    FSGHandLayer::GetWristLocation(
        true, // true for a right-handed glove and false for a left-handed one
        HandTrackingState.HandKeyLocations[0], // 0 is the Wrist joint location
        HandTrackingState.HandKeyRotations[0].Rotator, // 0 is the wrist joint rotation
        ESGPositionalTrackingHardware::Quest3Controller,
        WristLocation, WristRotation);

    // WristLocation and WristRotation variables now contain the correct
    // OpenXR hand-tracking data with the wrist offsets applied correctly
}

In the next sections we'll see: