What is hand tracking?
In recent years, Hand-Tracking developed into an integral function of virtual reality (VR) devices. It allows users to use their hands as an input method instead of using conventional VR controllers. This feature opens up a variety of possibilities and challenges for developers, particularly concerning usability. In this article, we will take a closer look at the concept of hand tracking and examine the supporting VR headsets, along with their advantages and disadvantages.
For businesses: Handtracking makes applications possible without controllers. Examples can be found under Hand tracking in VR applications.
How does hand tracking work
Hand tracking allows VR device users to perform tasks and actions in the virtual environment through simple hand gestures, such as pinching, grabbing, pushing, or pulling. Precise tracking of hands in the virtual world is crucial for a realistic VR experience. A fully articulated tracking system and hardware are required to determine the position of the user's hands in virtual space relative to the real world and to capture the movements of fingers and joints as quickly and accurately as possible. This enables natural interactions and a more immersive experience.
Status: October 2026. Hand tracking is available on Meta Quest 3/3S and supported PICO-4 models. PICO also mentions the PICO-4 Ultra series in its developer documentation. The HoloLens 2 is no longer manufactured and can be classified as a legacy platform. Hand tracking is practical for short, direct interactions; controllers often remain the more reliable choice for quick input and prolonged use. Meta device comparison · PICO developer documentation · Microsoft information about HoloLens 2
VR headsets with hand tracking features
Magic Leap 2
Magic Leap 2 features an impressive 70° diagonal field of view, a dynamic dimming function, and outstanding performance with 16 GB of RAM.
HoloLens 2 (Legacy platform)
The HoloLens 2 was Microsoft's second generation of a mixed reality headset. Microsoft stopped production in December 2024; it is primarily relevant for existing installations, but not a current purchase. The device was introduced in 2019 and offers more computing power, better sensors, a longer battery life, and a larger field of view compared to the previous model. The HoloLens 2 features eye tracking functionality that allows the user's eye movements to be tracked. Instead of dedicated controllers, the HoloLens 2 is operated using gestures and voice commands. New and more intuitive hand gestures enable easy content control.
Meta Quest 2
The Meta Quest 2 is a very popular standalone VR headset. The Meta Quest 2 also supports controller-free hand tracking. Using machine learning, the headset recognizes the position of the fingers, thus enabling the control of objects and actions in the virtual environment.
Pico 4
The Pico 4, similar to the Meta Quest, features optional hand tracking control in addition to VR controller control.
Ultraleap
Leap Motion is a small device that can be connected to a computer via USB or mounted on the front of VR headsets, allowing users to manipulate digital objects with hand gestures and movements. With Ultraleap and the included SDK, VR headsets without built-in hand tracking can be upgraded to hand tracking.
Varjo XR-3 Mixed Reality Headset
Varjo XR-3's premium headset already has Ultraleap built-in to enable the best hand tracking.
Meta Quest 3
The Quest 3 features an improved hand tracking system. The system uses depth sensors to track the position and movement of hands in 3D. Hand tracking on the Quest 3 is more accurate and reliable than that of previous models. It can now also recognize more complex hand gestures, such as grasping and releasing objects.
Interaction with Hand Tracking
There are three ways users can interact with the virtual environment: direct manipulation, hand rays, and gesture recognition.
Direct ManipulationUsers can grab and interact with objects in the VR environment with their hands. This method allows for intuitive control, where objects react realistically. Users can pick up objects, press buttons, and activate UI components as if interacting with a touchscreen.
Sandblasting (laser)This method allows users to generate a beam from their hand to access distant objects. Through gestures, users can grab, rotate, or push objects in the distance. Users might even feel haptic feedback at the beginning or during interaction with the object.
Gesture recognitionIn this method, the VR device analyzes the user's hand and finger gestures and triggers corresponding reactions. One example is a system that allows a user to take a screenshot by bringing the tips of their thumbs to their index fingers to create a square shape, mimicking looking through a camera's viewfinder.
It is advisable to include gestures in the design if the inherent meaning of a gesture aligns with the desired outcome.
Advantages and disadvantages of hand tracking
Advantages
Intuitive Interaction: Hand-tracking allows users to use natural and intuitive hand gestures to control objects in the virtual environment.
Increased Immersion: By being able to see one's own hands in the VR environment in real-time, a higher degree of immersion is achieved.
No additional controllers required: Hand-tracking eliminates the need for extra controllers, improving comfort and ease of use.
Disadvantages
Limited Functionality: The interaction capabilities of hand tracking are limited compared to traditional controllers. More complex actions and games may still require the use of controllers.
Limited Tracking Range: The cameras' tracking range for hand tracking is smaller than the range in which controllers are detected. Users must therefore be more careful about how they move.
Frequently Asked Questions (FAQ)
In hand tracking, the headset's cameras capture hands and finger joints in real time and map them in the virtual scene. The hand itself becomes the input device—controllers are no longer necessary.
Including Meta Quest 2, Quest 3 and Quest Pro, Pico 4 Enterprise, Microsoft HoloLens 2, Magic Leap 2, and Apple Vision Pro. Older headsets without built-in hand tracking can be retrofitted with an Ultraleap module.
For grabbing, pointing, pressing, and simple gestures, it is reliable enough. For fast, very precise, or force-based tasks, controllers remain superior—also because they provide haptic feedback, which is completely lacking in hand tracking.
Three patterns have emerged: direct manipulation, where objects within reach are touched; hand rays for distant objects and menus; and gesture recognition, such as a pinch or thumbs up, for commands.
The hands must remain in the cameras' field of view, otherwise tracking will be interrupted. Poor lighting conditions, gloves, and fingers obstructing one another interfere with detection. In addition, there is a lack of haptic feedback, which makes it harder for users to gauge whether they have actually grasped an object.
For applications with many short sessions and changing users: trade fair appearances, museums, showrooms, product presentations, and short introductory training sessions. In these cases, immediate usability without instruction clearly outweighs the lower precision.
