Consuming FXRMotionControllerData

Taking a closer look at the FXRMotionControllerData declaration inside the Unreal Engine's HeadMountedDisplay module at [Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h), figuring out the data structure might not seem very straightforward:

USTRUCT(BlueprintType)
struct FXRMotionControllerData
{
    GENERATED_USTRUCT_BODY();

    UPROPERTY(BlueprintReadOnly, Category = "XR")
    bool bValid = false;
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FName DeviceName;
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FGuid ApplicationInstanceID;
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    EXRVisualType DeviceVisualType = EXRVisualType::Controller;

    UPROPERTY(BlueprintReadOnly, Category = "XR")
    EControllerHand HandIndex = EControllerHand::Left;

    UPROPERTY(BlueprintReadOnly, Category = "XR")
    ETrackingStatus TrackingStatus = ETrackingStatus::NotTracked;

    // Vector representing an object being held in the player's hand
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FVector GripPosition = FVector(0.0f);
    // Quaternion representing an object being held in the player's hand
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FQuat GripRotation = FQuat(EForceInit::ForceInitToZero);

    // For handheld controllers, gives a vector for pointing at objects
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FVector AimPosition = FVector(0.0f);
    // For handheld controllers, gives a quaternion for pointing at objects
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FQuat AimRotation = FQuat(EForceInit::ForceInitToZero);

    // For handheld controllers, gives a vector for representing the hand
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FVector PalmPosition = FVector(0.0f);
    // For handheld controllers, gives a quaternion for representing the hand
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    FQuat PalmRotation = FQuat(EForceInit::ForceInitToZero);

    // The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    TArray<FVector> HandKeyPositions;
    // The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    TArray<FQuat> HandKeyRotations;
    // The indices of this array are the values of EHandKeypoint (Palm, Wrist, ThumbMetacarpal, etc).
    UPROPERTY(BlueprintReadOnly, Category = "XR")
    TArray<float> HandKeyRadii;

    UPROPERTY(BlueprintReadOnly, Category = "XR")
    bool bIsGrasped = false;
};

Which on the Blueprint side it looks like this:

FXRMotionControllerData Blueprint representation

But, fear not, we've got you covered!

FXRMotionControllerData in Unreal Engine

FXRMotionControllerData is a structure in Unreal Engine designed to hold detailed information about the state of a motion controller device at a given moment. This structure is essential for handling motion controller inputs in virtual reality (VR) applications, providing the necessary data to accurately track and represent the user's hand movements and actions within the virtual environment.

Structure Members of FXRMotionControllerData

  • bValid

    • Description: A boolean flag indicating whether the data is valid or not.
    • Usage: This is used to check if the motion controller data is correctly initialized and can be used for further processing.

  • DeviceName

    • Type: FName
    • Description: The name of the device.
    • Usage: Identifies which motion controller device the data is coming from, useful when multiple devices are in use.

  • ApplicationInstanceID

    • Type: FString
    • Description: A unique identifier for the application instance.
    • Usage: Helps in differentiating data from different instances of an application, ensuring the correct instance processes the data.

  • DeviceVisualType

    • Type: EXRVisualType
    • Description: Enum specifying the visual type of the device (e.g., controller, hand).
    • Usage: Used to differentiate between various motion controller devices or hand-tracking representations for rendering and interaction purposes.

  • HandIndex

    • Type: EControllerHand
    • Description: Enum indicating which hand is being tracked (left or right).
    • Usage: Helps identify whether the motion data pertains to the left or right hand, essential for hand-specific actions or interactions.

  • TrackingStatus

    • Type: EXRTrackingStatus
    • Description: Enum indicating the tracking status of the motion controller.
    • Usage: Shows whether the controller is being tracked accurately, with possible statuses like Tracked, NotTracked, etc.

  • GripPosition

    • Type: FVector
    • Description: The position of the grip in world coordinates.
    • Usage: Provides the 3D coordinates of the controller's grip, essential for positioning the virtual representation of the controller.

  • GripRotation

    • Type: FQuat
    • Description: The rotation of the grip in world coordinates.
    • Usage: Provides the orientation of the controller's grip, allowing for accurate rotation and alignment in the virtual space.

  • AimPosition

    • Type: FVector
    • Description: The position of the aim point in world coordinates.
    • Usage: Specifies where the controller is aiming, useful for aiming or pointing actions.

  • AimRotation

    • Type: FQuat
    • Description: The rotation of the aim point in world coordinates.
    • Usage: Determines the orientation of the aim direction, important for actions like shooting or selecting objects in VR.

  • PalmPosition

    • Type: FVector
    • Description: The position of the palm in world coordinates.
    • Usage: Provides the 3D location of the palm, important for determining hand gestures or interactions in VR.

  • PalmRotation

    • Type: FQuat
    • Description: The rotation of the palm in world coordinates.
    • Usage: Defines the orientation of the palm, crucial for hand-based interaction accuracy and realism in VR experiences.

  • HandKeyPositions

    • Type: TArray<FVector>
    • Description: An array of vectors representing key positions of the hand.
    • Usage: Provides detailed positions of key points on the hand, useful for precise hand tracking and interaction.

  • HandKeyRotations

    • Type: TArray<FQuat>
    • Description: An array of quaternions representing key rotations of the hand.
    • Usage: Complements the hand key positions with rotational data, ensuring accurate representation of hand movements.

  • HandKeyRadii

    • Type: TArray<float>
    • Description: An array of floats representing the radii of key points of the hand.
    • Usage: Gives the size of the hand key points, aiding in collision detection and interaction fidelity.

  • bIsGrasped

    • Type: bool
    • Description: A boolean indicating whether the controller is currently grasping an object.
    • Usage: Determines if the user is holding something, affecting interactions and animations.

Organization of FXRMotionControllerData

The structure is organized to encapsulate all relevant data needed for hand and motion controller tracking in a coherent and accessible manner. Boolean flags bValid and bIsGrasped provide quick checks on the state of the controller data. Identifiers DeviceName and ApplicationInstanceID ensure the correct association of data. Positional and rotational data GripPosition, GripRotation, AimPosition, and AimRotation offer precise tracking of the controller's movement. Arrays HandKeyPositions, HandKeyRotations, and HandKeyRadii allow detailed hand tracking, which is critical for immersive VR experiences. Lastly, the tracking status TrackingStatus informs the system of the reliability of the data being processed and whether the motion controller is actively being tracked or it's inactive at the moment.

Processing the Data for Drawing and Animating a Virtual Hand

In order to draw and animate a virtual hand in real-time whether the data is coming from hand-tracking or a SenseGlove device, we could consume the data from the HandKeyPositions and HandKeyRotations fields of the FXRMotionControllerData struct.

Both HandKeyPositions and HandKeyRotations contain 26 elements as defined by OpenXR's XR_HAND_JOINT_COUNT_EXT and XrHandJointLocationsEXT, etc.

Unreal Engine also provides an enum called EHandKeypoint naming the 26 joints, and the equivalent of XR_HAND_JOINT_COUNT_EXT as EHandKeypointCount inside [Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h](https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Source/Runtime/HeadMountedDisplay/Public/HeadMountedDisplayTypes.h) as follows:

/**
 * Transforms that are tracked on the hand.
 * Matches the enums from WMR to make it a direct mapping
 */
UENUM(BlueprintType)
enum class EHandKeypoint : uint8
{
    Palm,
    Wrist,
    ThumbMetacarpal,
    ThumbProximal,
    ThumbDistal,
    ThumbTip,
    IndexMetacarpal,
    IndexProximal,
    IndexIntermediate,
    IndexDistal,
    IndexTip,
    MiddleMetacarpal,
    MiddleProximal,
    MiddleIntermediate,
    MiddleDistal,
    MiddleTip,
    RingMetacarpal,
    RingProximal,
    RingIntermediate,
    RingDistal,
    RingTip,
    LittleMetacarpal,
    LittleProximal,
    LittleIntermediate,
    LittleDistal,
    LittleTip
};

const int32 EHandKeypointCount = static_cast<int32>(EHandKeypoint::LittleTip) + 1;

So, getting the any joint's position or rotation is as easy as casting the enum value and passing it as the array index.

    FXRMotionControllerData MotionControllerData;
    const bool bGotMotionControllerData = FSGXRTracker::GetMotionControllerData(
        GetWorld(), EControllerHand::Left, MotionControllerData);

    // Return if the struct data is invalid!
    if (!bGotMotionControllerData || !MotionControllerData.bValid)
    {
        return;
    }

    // Return if the device is not being tracked!
    if (MotionControllerData.TrackingStatus == ETrackingStatus::NotTracked)
    {
        return;
    }

    // Ensure that MotionControllerData.DeviceVisualType is a hand!
    if (!ensureAlwaysMsgf(MotionControllerData.DeviceVisualType
                          == EXRVisualType::Hand,
                          TEXT("Invalid DeviceVisualType type!")))
    {
    }

    // Ensure that MotionControllerData.HandKeyPositions has the position data
    // for 26 joints!
    if (!ensureAlwaysMsgf(MotionControllerData.HandKeyPositions.Num()
                          == EHandKeypointCount,
                          TEXT("Invalid HandKeyPositions count!")))
    {
        return;
    }

    // Ensure that MotionControllerData.HandKeyRotations has the rotation data
    // for 26 joints!
    if (!ensureAlwaysMsgf(MotionControllerData.HandKeyRotations.Num()
                          == EHandKeypointCount,
                          TEXT("Invalid HandKeyRotations count!")))
    {
        return;
    }

    static constexpr int32 PalmIndex = static_cast<int32>(EHandKeypoint::Palm);

    const FVector& PalmPosition{
        MotionControllerData.HandKeyPositions[PalmIndex]
    };
    const FRotator& PalmRotation{
        MotionControllerData.HandKeyRotations[PalmIndex].Rotator()
    };

The equivalent Blueprint code for the above looks something like this:

Get a joint position and rotation from FXRMotionControllerData in Blueprint

OK, now that we've got a glimpse of how the virtual hand's joint data could be processed we are going to draw and animate a virtual hand in both Blueprint and C++ in the upcoming sections.