Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Roll Your Own Customized Hand Manipulation and Interaction System

The default hand interaction system shipped with the SenseGlove Unreal Engine Plugin consists of various components, including SGPawn, SGPlayerController, SGVirtualHandComponent, SGGrabComponent, SGTouchComponent, and others. This system is very easy to get started with and is thoroughly documented throughout this handbook.

However, this simplicity comes at a cost: limited functionality. At SenseGlove, we prioritize usability and practicality. That said, developing a comprehensive hand interaction system that suits every possible use case is not an easy task. For example, projects such as the VR Expansion Plugin (VRE) — an Epic MegaGrants recipient — have been in development for over a decade, and development is still ongoing. Naturally, such depth also comes with a trade-off: a steep learning curve and reduced beginner-friendliness.

To bridge this gap and serve different groups of users, we provide the SGPawn system as a simple, intentionally limited, and beginner-friendly default solution that allows anyone to get up and running quickly.

At the same time, to support intermediate and advanced users, we have aimed for full OpenXR compatibility, opening the door to a wide range of advanced possibilities. Once enabled and loaded in Unreal Engine, the SenseGlove Unreal Engine Plugin registers itself as an OpenXRHandTracking provider. This makes it a fully compatible, drop-in replacement for Epic’s OpenXRHandTracking plugin.

As a result, it can integrate seamlessly with any third-party system or plugin that consumes OpenXR hand-tracking data. Because the SenseGlove plugin is fully OpenXR-compliant, it provides hand-tracking data in the expected OpenXR format and becomes the active provider within Unreal. If your existing interaction system (for example, the VRE plugin) already relies on OpenXR hand-tracking, SenseGlove can function as a direct tracking source instead of a physical hand.

Furthermore, the SenseGlove OpenXR backend allows you to develop and build your own hand interaction system from scratch. This system can operate either via standard OpenXR hand-tracking or with a SenseGlove device interchangeably.

Comparison of Available Approaches

The following table provides an overview and comparison of different hand interaction approaches available within the SenseGlove Unreal Engine Plugin ecosystem when it comes to hand-interaction systems:

Built-in?Works out of the box?Beginner-friendly?Learning CurveFeaturefulCustomizable?Supports Custom Gestures?
SGPawn✅ Yes✅ Yes✅ Most beginer-friendly✅ Very easy⚠️ Very basic⚠️ Very limited❌ Not yet, maybe added in the future
SenseGlove OpenXR✅ Yes❌ Requires Blueprint or C++ coding✅ Requires a few hours of watching tutotrials✅ Moderate❌ You need to develop your own features and functionalities from scratch✅ Your imagination, creativity, and skill level are your limits✅ You have to implement your own recognition logic; example pinch detection algorithm is taught in the tutorial series
VR Expansion Plugin❌ No⚠️ Partially – requires setup❌ Best suited for intermediate or advanced users⚠️ Steep✅ Diverse features and functionalities✅ Highly customizable✅ Via custom logic
Other OpenXR-compatible Plugins❌ No❓ Check their documentation❓ Check their documentation❓ Check their documentation❓ Check their documentation❓ Check their documentation❓ Check their documentation

Going Beyond SGPawn

In the following sections, we will cover:

We have also covered more in-depth and advanced topics in other parts of this handbook, available in the following sections:

  • OpenXR: an introduction to OpenXR fundamentals in Unreal Engine.
    • Consuming FXRHandTrackingState: explains the data layout of Unreal Engine’s FXRHandTrackingState.
      • Blueprint: demonstrates how to use FXRHandTrackingState data and render a debug hand in Blueprint.
      • C++: demonstrates how to use FXRHandTrackingState data and render a debug hand in C++.
    • Third-Party Integrations: provides a sample Unreal Engine 5.4 project demonstrating how to integrate SenseGlove with the VR Expansion (VRE) Plugin.
    • Third-Party Tutorials:— a tutorial series that guides you from beginner to advanced level in using FXRHandTrackingState to build your own hand interaction system by animating virtual hand meshes.