Consuming FXRHandTrackingState

Taking a closer look at the FXRHandTrackingState declaration inside the Unreal Engine's HeadMountedDisplay module at [Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h), figuring out the data structure might not seem very straightforward:

USTRUCT(BlueprintType)
struct FXRHandTrackingState
{
    GENERATED_USTRUCT_BODY();

    // The state is valid if poses have ever been provided.
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    bool bValid = false;
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FName DeviceName;
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FGuid ApplicationInstanceID;

    UPROPERTY(BlueprintReadOnly, Category = "XR")
    EXRSpaceType XRSpaceType = EXRSpaceType::UnrealWorldSpace;

    UPROPERTY(BlueprintReadOnly, Category = "XR")
    EControllerHand Hand = EControllerHand::Left;

    UPROPERTY(BlueprintReadOnly, Category = "XR")
    ETrackingStatus TrackingStatus = ETrackingStatus::NotTracked;

    // The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    TArray<FVector> HandKeyLocations;

    // The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    TArray<FQuat> HandKeyRotations;

    // The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    TArray<float> HandKeyRadii;
};

Which on the Blueprint side it looks like this:

But, fear not, we've got you covered!

FXRHandTrackingState in Unreal Engine

FXRHandTrackingState is a structure in Unreal Engine designed to hold detailed information about the state of a hand-tracking device at a given moment. This structure is essential for handling hand-tracking inputs in virtual reality (VR) applications, providing the necessary data to accurately track and represent the user's hand movements and actions within the virtual environment.

Structure Members of FXRHandTrackingState

• bValid

  • Description: A boolean flag indicating whether the data is valid or not.
  • Usage: This is used to check if the motion controller data is correctly initialized and can be used for further processing.

• DeviceName

  • Type: FName
  • Description: The name of the device.
  • Usage: Identifies which device the data is coming from, useful when multiple devices are in use.

• ApplicationInstanceID

  • Type: FString
  • Description: A unique identifier for the application instance.
  • Usage: Helps in differentiating data from different instances of an application, ensuring the correct instance processes the data.

• XRSpaceType

  • Type: EXRSpaceType
  • Description: Enum specifying the type of XR space being used (e.g., unreal world or tracking space).
  • Usage: Specifies the coordinate system the XR Device is tracking itself in.

• Hand

  • Type: EControllerHand
  • Description: Enum indicating which hand is being tracked (left or right).
  • Usage: Helps identify whether the hand-tracking data pertains to the left or right hand, essential for hand-specific actions or interactions.

• TrackingStatus

  • Type: EXRTrackingStatus
  • Description: Enum indicating the tracking status of the hand-tracking device.
  • Usage: Shows whether the hand-tracking device is being tracked accurately, with possible statuses like Tracked, NotTracked, etc.

• HandKeyLocations

  • Type: TArray<FVector>
  • Description: An array of vectors representing key locations of the hand.
  • Usage: Provides detailed locations of key points on the hand, useful for precise hand-tracking and interaction.

• HandKeyRotations

  • Type: TArray<FQuat>
  • Description: An array of quaternions representing key rotations of the hand.
  • Usage: Complements the hand key locations with rotational data, ensuring accurate representation of hand movements.

• HandKeyRadii

  • Type: TArray<float>
  • Description: An array of floats representing the radii of key points of the hand.
  • Usage: Gives the size of the hand key points, aiding in collision detection and interaction fidelity.

Organization of FXRHandTrackingState

The structure is organized to encapsulate all relevant data needed for hand-tracking in a coherent and accessible manner. Boolean flag bValid provides quick checks on the state of the controller data. Identifiers DeviceName and ApplicationInstanceID ensure the correct association of data. Arrays HandKeyLocations, HandKeyRotations, and HandKeyRadii allow detailed hand-tracking, which is critical for immersive VR experiences. Lastly, the tracking status TrackingStatus informs the system of the reliability of the data being processed and whether the hands are actively being tracked or they are inactive at the moment.

Processing the Data for Drawing and Animating a Virtual Hand

In order to draw and animate a virtual hand in real-time whether the data is coming from hand-tracking or a SenseGlove device, we could consume the data from the HandKeyLocations and HandKeyRotations fields of the FXRHandTrackingState struct.

Both HandKeyLocations and HandKeyRotations contain 26 elements as defined by OpenXR's XR_HAND_JOINT_COUNT_EXT and XrHandJointLocationsEXT, etc.

Unreal Engine also provides an enum called EHandKeypoint naming the 26 joints, and the equivalent of XR_HAND_JOINT_COUNT_EXT as EHandKeypointCount inside [Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h) as follows:

/**
 * Transforms that are tracked on the hand.
 * Matches the enums from WMR to make it a direct mapping
 */
UENUM(BlueprintType)
enum class EHandKeypoint : uint8
{
    Palm,
    Wrist,
    ThumbMetacarpal,
    ThumbProximal,
    ThumbDistal,
    ThumbTip,
    IndexMetacarpal,
    IndexProximal,
    IndexIntermediate,
    IndexDistal,
    IndexTip,
    MiddleMetacarpal,
    MiddleProximal,
    MiddleIntermediate,
    MiddleDistal,
    MiddleTip,
    RingMetacarpal,
    RingProximal,
    RingIntermediate,
    RingDistal,
    RingTip,
    LittleMetacarpal,
    LittleProximal,
    LittleIntermediate,
    LittleDistal,
    LittleTip
};

const int32 EHandKeypointCount = static_cast<int32>(EHandKeypoint::LittleTip) + 1;

So, getting the any joint's location or rotation is as easy as casting the enum value and passing it as the array index.

    FXRHandTrackingState HandTrackingState;
    const bool bGotHandTrackingState = FSGXRTracker::GetHandTrackingState(
        GetWorld(), EXRSpaceType::UnrealWorldSpace, EControllerHand::Left, HandTrackingState);

    // Return if the struct data is invalid!
    if (!bGotHandTrackingState || !HandTrackingState.bValid)
    {
        return;
    }

    // Return if the device is not being tracked!
    if (HandTrackingState.TrackingStatus == ETrackingStatus::NotTracked)
    {
        return;
    }

    // Ensure that HandTrackingState.HandKeyLocations has the location data
    // for 26 joints!
    if (!ensureAlwaysMsgf(HandTrackingState.HandKeyLocations.Num()
                          == EHandKeypointCount,
                          TEXT("Invalid HandKeyLocations count!")))
    {
        return;
    }

    // Ensure that HandTrackingState.HandKeyRotations has the rotation data
    // for 26 joints!
    if (!ensureAlwaysMsgf(HandTrackingState.HandKeyRotations.Num()
                          == EHandKeypointCount,
                          TEXT("Invalid HandKeyRotations count!")))
    {
        return;
    }

    static constexpr int32 PalmIndex = static_cast<int32>(EHandKeypoint::Palm);

    const FVector& PalmLocation{
        HandTrackingState.HandKeyLocations[PalmIndex]
    };
    const FRotator& PalmRotation{
        HandTrackingState.HandKeyRotations[PalmIndex].Rotator()
    };
    

The equivalent Blueprint code for the above looks something like this:

OK, now that we've got a glimpse of how the virtual hand's joint data could be processed we are going to draw and animate a virtual hand in both Blueprint and C++ in the upcoming sections.