Consuming FXRHandTrackingState in C++

Before continuing this section, please ensure you've first studied the Consuming FXRHandTrackingState section.

Drawing and Animating Virtual Hands

1. Create a new Virtual Reality project based the Unreal VR Template.

2. Make sure the SenseGlove UnrealEngine plugin is installed and enabled inside your new project.

3. You could use either hand-tracking or a SenseGlove device as the input data, or both of the inside the same project. Whether you would like to use hand-tracking or a SenseGlove device, please make sure the required steps are taken for each of those first.

4. From the Tools menu choose New C++ class....

5. Choose the Unreal Engine's APawn class as the parent class for the new C++ pawn class.

6. Name the new pawn class DebugPawn.

7. Since we have created a new C++ class, this converts the current Blueprint VRTemplateMap project to a C++ one. That's why the Unreal Editor will give us a few prompts regarding opening the project in the default IDE and rebuilding the code. It might be simpler to just close the editor, then rebuild the source code inside your favorite IDE, and then start the editor with the converted project again.

8. Find and open the VRPawn Blueprint Class located at /Content/VRTemplate/Blueprints/VRPawn inside the Blueprint Editor and from the File menu choose the Reparent Blueprint class.

9. In the new Reparent blueprint window choose DebugPawn as the new parent.

10. By looking at the Parent Class label located under the Blueprint Editor window control buttons verify that the ADebugPawn class has been set as the new parent.

11. Locate the project's main Build file, in our case VirtualHandCpp/Source/VirtualHandCpp/VirtualHandCpp.Build.cs and add the InputDevice, OpenXRHMD, SenseGloveBuildHacks, SenseGloveDebug, SenseGloveSettings, and SenseGloveTracking modules as either a private or public dependency.

// Fill out your copyright notice in the Description page of Project Settings.

using UnrealBuildTool;

public class VirtualHandCpp : ModuleRules
{
    public VirtualHandCpp(ReadOnlyTargetRules Target) : base(Target)
    {
        PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
    
        PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore" });

        PrivateDependencyModuleNames.AddRange(new string[]
        {
            "InputDevice",
            "OpenXRHMD",
            "SenseGloveBuildHacks",
            "SenseGloveDebug",
            "SenseGloveSettings",
            "SenseGloveTracking"
        });

        // Uncomment if you are using Slate UI
        // PrivateDependencyModuleNames.AddRange(new string[] { "Slate", "SlateCore" });
        
        // Uncomment if you are using online features
        // PrivateDependencyModuleNames.Add("OnlineSubsystem");

        // To include OnlineSubsystemSteam, add it to the plugins section in your uproject file with the Enabled attribute set to true
    }
}

12. Locate the C++ header and source file for the ADebugPawn inside the project in your C++ IDE. In our case they are located at VirtualHandCpp/Source/VirtualHandCpp/DebugPawn.h and VirtualHandCpp/Source/VirtualHandCpp/DebugPawn.cpp.

13. Modify the DebugPawn.h header file to look like this:

// Fill out your copyright notice in the Description page of Project Settings.

#pragma once

#include "CoreMinimal.h"
#include "GameFramework/Pawn.h"

#include "SGSettings/SGDebugGizmoSettings.h"

#include "DebugPawn.generated.h"

UCLASS()
class VIRTUALHANDCPP_API ADebugPawn : public APawn
{
    GENERATED_BODY()

private:
    // The virtual hand drawing settings.
    UPROPERTY(EditDefaultsOnly, Category="DebugPawn",
        meta=(AllowPrivateAccess="false"))
    FSGDebugGizmoSettings HandDrawingSettings;

public:
    // Sets default values for this pawn's properties
    ADebugPawn();

protected:
    // Called when the game starts or when spawned
    virtual void BeginPlay() override;

public:
    // Called every frame
    virtual void Tick(float DeltaTime) override;

    // Called to bind functionality to input
    virtual void SetupPlayerInputComponent(class UInputComponent* PlayerInputComponent) override;

private:
    // The method responsible for drawing a virtual hand.
    void DrawHand(EControllerHand Hand) const;
};

14. Modify the DebugPawn.cpp implementation file to look like this:

// Fill out your copyright notice in the Description page of Project Settings.


#include "DebugPawn.h"

#include "SGDebug/SGDebugGizmo.h"
#include "SGTracking/SGXRTracker.h"

// Sets default values
ADebugPawn::ADebugPawn()
{
    // Set this pawn to call Tick() every frame.  You can turn this off to improve performance if you don't need it.
    PrimaryActorTick.bCanEverTick = true;

    // Set the default virtual hand drawing settings.
    HandDrawingSettings = FSGDebugGizmoSettings{
        1.0f,
        FColor{255, 0, 0, 255},
        FColor{0, 255, 0, 255},
        FColor{0, 0, 255, 255},
        false,
        1.1f,
        0,
        0.2f,
    };
}

// Called when the game starts or when spawned
void ADebugPawn::BeginPlay()
{
    Super::BeginPlay();
}

// Called every frame
void ADebugPawn::Tick(float DeltaTime)
{
    Super::Tick(DeltaTime);

    // Attempt at drawing the left/right virtual hands every frame.
    DrawHand(EControllerHand::Left);
    DrawHand(EControllerHand::Right);
}

// Called to bind functionality to input
void ADebugPawn::SetupPlayerInputComponent(UInputComponent* PlayerInputComponent)
{
    Super::SetupPlayerInputComponent(PlayerInputComponent);
}

void ADebugPawn::DrawHand(const EControllerHand Hand) const
{
    // Get the world and cache it, if it's null we return early.
    UWorld* World{GetWorld()};
    if (!IsValid(World))
    {
        return;
    }

    FXRHandTrackingState HandTrackingState;
    const bool bGotHandTrackingState = FSGXRTracker::GetHandTrackingState(
        World, EXRSpaceType::UnrealWorldSpace, Hand, HandTrackingState);

    // Return if the struct data is invalid!
    if (!bGotHandTrackingState || !HandTrackingState.bValid)
    {
        return;
    }

    // Return if the device is not being tracked!
    if (HandTrackingState.TrackingStatus == ETrackingStatus::NotTracked)
    {
        return;
    }

    // Ensure that HandTrackingState.HandKeyLocations has the location data
    // for 26 joints!
    if (!ensureAlwaysMsgf(HandTrackingState.HandKeyLocations.Num()
                          == EHandKeypointCount,
                          TEXT("Invalid HandKeyLocations count!")))
    {
        return;
    }

    // Ensure that HandTrackingState.HandKeyRotations has the rotation data
    // for 26 joints!
    if (!ensureAlwaysMsgf(HandTrackingState.HandKeyRotations.Num()
                          == EHandKeypointCount,
                          TEXT("Invalid HandKeyRotations count!")))
    {
        return;
    }

    // Iterate over the hand joint locations and rotations!
    for (int32 JointIndex = 0; JointIndex < EHandKeypointCount; ++JointIndex)
    {
        const FVector& JointLocation{
            HandTrackingState.HandKeyLocations[JointIndex]
        };
        const FQuat& JointRotation{
            HandTrackingState.HandKeyRotations[JointIndex]
        };

        // Draw a single joint's gizmo!
        // Please note that we could alternatively:
        // Use FSGDebugCube::Draw() to draw a cube.
        // Or use the FSGDebugVirtualHand::Draw() method and pass the
        // HandTrackingState directly to draw the virtual hand
        // all at once without iterating the joints. But, that's not
        // goal of this tutorial.
        FSGDebugGizmo::Draw(World, JointLocation, JointRotation, HandDrawingSettings);
    }
}

15. Now, rebuild the source code and go back to the VRTemplateMap, then use the VR Preview button to run the game. If everything's done correctly, you should be able to see the virtual hands inside your VR simulation.