AI has Got a 3D scene from the Reflection of the Eyes

The authors of the original article performed the image from the reflection of a person’s eyes from a picture with use of AI.

AI has Got a 3D scene from the Reflection of the Eyes. Nerf

The authors of the algorithm detailed the challenges they encountered during its development on their page, and have pledged to share the source code in the near future.

The reflective nature of the human eye is an underappreciated source of information about what the world around us looks like. By imaging the eyes of a moving person, we can collect multiple views of a scene outside the camera’s direct line-of-sight through the reflections in the eyes. In this paper, we reconstruct a 3D scene beyond the camera’s line-of-sight using portrait images containing eye reflections.
This task is challenging due to 1) the difficulty of accurately estimating eye pose and 2) the entangled texture of the eye iris and the scene reflections. Our method jointly refines the cornea poses, the radiance field depicting the scene, and the observer’s eye iris texture. We further propose a simple regularization prior on the iris texture pattern to improve reconstruction quality. Through various experiments on synthetic and real-world captures featuring people with varied eye colors, we demonstrate the feasibility of our approach to recover 3D scenes using eye reflections.

In healthy adults, the geometry of the cornea tends to be similar. This characteristic allows us to determine a person’s eye position by measuring the size of the pixels in their cornea from an image. We utilize this knowledge to train the radiance field on reflections from the eyes. This is done by projecting rays from the camera and reflecting them off the estimated eye geometry. To prevent the iris texture from appearing in the reconstruction, we use texture decomposition. This involves training a 2D texture map that learns the iris texture while simultaneously removing it from the final result with AI.

To mitigate the problem of noise when approximating the eye pose from an image, we utilize eye pose optimization, which we demonstrate to be crucial for optimal performance.

The authors also gave several examples of work using frames from music videos of Lady Gaga and Miley Cyrus.

Alex Patel

An accomplished editor with a specialization in AI articles. Born and raised in NYC, USA. He graduated the Columbia University, and worked in tech companies of USA and Canada.

Add a comment