Capturing precise 3D details with a single camera has long been a challenge. Traditional methods often require complex dual-camera setups or specialized lighting conditions that are impractical for real-world applications. However, a groundbreaking approach developed at Nanjing University is set to redefine 3D imaging.

In our latest research, published in Optica, we introduce a cutting-edge snapshot polarization stereo imaging system (SPSIM), as shown in Fig. 1. This innovative system integrates metasurface optics with artificial intelligence to extract highly detailed 3D shape information in real time.

Unlike conventional methods that rely on multiple polarizers or sequential exposures, SPSIM utilizes a specially engineered metasurface lens to capture full-Stokes polarization data in a single shot. With an extinction ratio of 25 dB—comparable to commercial polarizers—and an unprecedented central wavelength efficiency of 65%, our system outperforms standard polarization cameras.

Large-scale polarization metasurfaces for SPSIM

To further enhance the performance of the SPSIM system, we developed a large-scale polarization metasurface with dimensions of 1.65 × 1.65 mm², as shown in Fig. 2. Experimental results demonstrated that the metasurface successfully separated six distinct polarization states of incident light, precisely guiding each polarization component to its target position.

Even under narrowband conditions, the metasurface maintained its performance. Simulation results closely aligned with experimental data, confirming the efficacy of the metasurface in enhancing the SPSIM system’s capabilities.

Neural networks: The key to high-precision 3D reconstruction

The key to SPSIM’s success lies in its neural network-driven processing pipeline. By incorporating circular polarization into the imaging process, we significantly enhance surface normal accuracy, achieving depth precision within 0.15 mm. This level of detail is crucial for applications that demand extreme accuracy, such as biomedical imaging, industrial inspection, and autonomous systems.

SPSIM’s 3D reconstruction process begins with preprocessing to obtain unique values for the zenith angle (ϑ) and azimuth angle (𝜓), as shown in Fig. 3. Depth information is initially retrieved using the measured full-Stokes parameters (FSP), angle of polarization (AOP), and degree of polarization (DOP).

To address the azimuth angle (𝜓) ambiguity, we employed a shape-from-shading (SFS) approach as a physical prior. A neural network was then introduced, leveraging FSP and prior information to train an enhanced U-Net model, ensuring highly accurate surface normal recovery.

Real-world testing: Achieving remarkable 3D reconstructions

To assess the performance of the network, we conducted both qualitative and quantitative analyses of the surface normal maps reconstructed from a test set (Fig. 4a). Objects such as bottles and cups were selected for testing, revealing that under natural lighting conditions, traditional cameras and the human eye struggled to discern fine details on resin materials.

In contrast, the SFP method demonstrated high sensitivity to such details. When compared to traditional methods, the neural network-driven SFP significantly reduced reconstruction errors on smooth surfaces, showing a marked improvement in accuracy.

Furthermore, our tests confirmed the crucial role of circular polarization (CP) in shape recovery. The information provided by CP enabled the system to precisely capture subtle contour variations. Omitting CP led to significant errors in the normal map, with fewer details being captured, as shown in Fig. 4b and 4c.

This highlights the importance of incorporating CP in polarization stereo imaging to achieve a highly detailed and accurate reconstruction of object textures. Experiments also captured objects from multiple angles, and by merging point cloud data, a complete 3D texture of the object was successfully reconstructed (Fig. 5).

A new frontier in 3D imaging

Overall, our research represents a significant advancement in high-resolution 3D imaging. SPSIM’s compact and efficient design enables seamless integration into portable devices, making real-time 3D surface reconstruction feasible even in extreme environments. This breakthrough opens up new possibilities for applications in augmented reality, robotic vision, and next-generation imaging technologies.

This story is part of Science X Dialog, where researchers can report findings from their published research articles. Visit this page for information about Science X Dialog and how to participate.

Boyan Fu is currently a Ph.D. candidate at the School of Physics, Nanjing University, under the supervision of Prof. Shining Zhu and Prof. Shuming Wang. Her research focuses on metalens-based light-field manipulation and multidimensional imaging, with applications in ultracompact display devices and plenoptic imaging systems.