Anyone who uses Gaussian splatting to 3D scan real-world spaces, vehicles, or architecture will quickly realize that not every surface behaves the same way. Matte walls, seats, floors, or furniture can often be captured surprisingly well. Things get more difficult when reflections come into play. This is precisely where a topic begins that initially seems almost a bit mysterious: the apparent “world behind the mirror.”
What at first glance appears to be a strange glitch is actually a direct result of how these scans are created. The scanner cannot clearly distinguish between actual geometry and reflections. And this is precisely what causes the typical problems that repeatedly arise with mirrors, windows, shiny floors, and other reflective surfaces.
When we work with LiDAR-based devices like the PortalCam or the K1, we don’t just capture a photo of a scene. Instead, we create a 3D model using depth information and image data. This works very well with clear, physical geometry. However, things get complicated when dealing with reflective surfaces.
The scanner cannot uniquely detect reflection. It treats the reflected information as if geometry were actually present, thereby creating a spatial mirror world.
Why the scanner behaves this way
Reflective and transparent surfaces pose a challenge for LiDAR because the signal does not return as clearly as it does from matte surfaces. This can result in missing measurements, distorted depth information, or structures that appear spatially incorrect. The fact that the “mirror image” was already visible in the LiDAR preview in our case suggests that the effect begins at the point cloud level.
This mirrored world is not merely an artifact. It is precisely the mechanism that ensures a reflection appears credible from the right perspective.
The problem isn't that Gaussian Splatting fundamentally fails with mirrors. The problem is that the logic that makes the reflection appear credible can break down elsewhere.
As long as one moves from a perspective in which this mirrored world makes sense, the result can be surprisingly convincing. The mirror or window then seems plausible because the system has indeed created a kind of reflected counter-space.
It becomes difficult when there is actually no physical space for this mirror world, or when one views the scene from a completely different perspective. Then it becomes apparent that what looked like a correct reflection on one side has, in reality, been pushed outward spatially.
In my video, I demonstrate this effect using a bus that I scanned from the inside. When viewed from the inside, the reflections in the windows initially look realistic. But as soon as you move to the outside, the illusion is shattered. The reflected elements of the interior appear to float in front of the glass—as if the inside of the bus were being pushed outward through the windows.
This is not a minor beauty problem, but a structural one. From this perspective, the scene is basically unusable.
If one accepts that reflective surfaces in Gaussian splatting present a category of challenge in itself, you quickly arrive at two practical approaches.
Why the artifacts can't simply be cut away: Precisely this geometry is often the basis for the reflection appearing credible from the inside. If you remove it from the outside, you simultaneously destroy the effect on the other side.
The first approach is not to expect a single scan to perfectly solve both the interior and exterior simultaneously. Instead, one works with multiple states and blends between them – one version for the interior, one for the exterior.
This can be an elegant solution, especially in interactive applications. The user doesn't notice that the system is switching between two states in the background; instead, they simply experience a seamless scene.
The second approach is to artificially add or replace problematic reflections later. Instead of hoping that windows or mirrors are perfectly reconstructed in the raw scan, the reflections are deliberately recreated in the editor using Reflection Maps, Fake Mirrors, or similar auxiliary constructs.
Just in PlayCanvas can something like this be implemented well. Also tools like StorySplat offer no-code approaches for this.
The „world behind the mirror“ is not some mystical exception, but a logical consequence of how Gaussian splatting deals with reflective information. The scanner cannot cleanly distinguish whether it is seeing real geometry or a reflection.
Once you understand this principle, you can make much more informed decisions: when a single scan is sufficient, when you should work with multiple states—and when artificial reflections are the better solution.
Are you interested in developing a virtual reality or 360° application? You may still have questions about budget and implementation. Feel free to contact me.
I am looking forward to you
Clarence Dadson CEO Design4real
