Augmented reality & technology:
How does tracking work in AR?

What does tracking in AR mean?

Tracking is the process by which AR systems record the position and orientation of the user and their surroundings in real time. The aim of tracking is to anchor digital content stably and precisely in the physical world.

Important tracking functions:

• Determination of the Position of devices (e.g. smartphones, AR glasses) in the room.
• Identification and analysis of Environmental features such as walls, floors or furniture.
• Pursuit of the Movement of the user to dynamically adapt digital content.

Without reliable tracking, AR applications would be inaccurate, unstable or unusable.

Important tracking technologies in AR

1. SLAM (Simultaneous Localization and Mapping)
   SLAM is one of the most advanced tracking technologies. It enables AR systems to create a map of the environment in real time and simultaneously determine their own position within it.

   • Advantages: Ideal for complex environments and markerless applications.
   • Example: AR glasses such as Microsoft HoloLens use SLAM to precisely place virtual objects.

2. Marker-based tracking
   This uses visual markers such as QR codes or special patterns that are recognized by the camera. The markers serve as an anchor point for digital content.

   • Advantages: Simple and reliable, especially for indoor applications.
   • Example: AR advertising on posters or in magazines that is activated via QR codes.

3. Markerless tracking
   Markerless tracking uses algorithms to recognize natural features of the environment (e.g. edges, surfaces). This allows content to be placed without markers.

   • Advantages: Flexibility and application in any environment.
   • Example: IKEA Place app that projects furniture into the room in real time.

4. GPS tracking
   GPS tracking is used for location-based AR experiences. The user's position is determined using satellite signals in order to base digital content on geographical data.

   • Advantages: Ideal for outdoor applications and large areas.
   • Example: Pokémon GO, which uses GPS for the placement of PokéStops and arenas.

5. Camera-based technologies
   Cameras serve as the main sensors for most AR applications. They capture visual data that is analyzed by AI and algorithms to understand the environment.

   • Advantages: Combination with other technologies such as SLAM or marker tracking possible.
   • Example: Snapchat filters that use facial recognition with camera data.

An important limitation is the limitation of hardware power. Typically, AR tracking requires a lot of computing power to accurately determine the position of virtual objects in the real world. If the device's hardware is not powerful enough, this can lead to delayed or inaccurate tracking, which can negatively impact the AR experience. Especially on low performing mobile devices, it becomes difficult to drive AR applications. Depending on how many polygons the virtual objects shown bring, less or more performance is left for tracking. This means that the more polygonal the objects, the more difficult it is to track them cleanly in space.

Another problem can arise due to the limitations of the framework used. AR tracking technologies are usually implemented in AR frameworks such as ARKit from Apple or ARCore from Google. If the framework is not robust enough or does not provide sufficient features, this can limit tracking and negatively impact the user experience. Not all frameworks perform equally well.

Another challenge is that tracking in AR is a continuous experience that takes place in real time. It requires a high level of accuracy because the virtual objects must move in real time with the real world. A small error in tracking can result in the virtual object no longer appearing in the right place, which can significantly affect the user experience.