Abstract:

3D reconstruction has been at the core of many computer vision applications, including autonomous driving, visual inspection in manufacturing, and augmented and virtual reality (AR/VR). Despite the tremendous progress made over the years, there remain challenging open-research problems. This thesis addresses three such problems in 3D reconstruction. First, we address the problem of defocus (and hence, depth) estimation and defocus deblurring from a single capture, which is fundamentally ill-posed using a conventional camera. Next, we address the problem of non-line-of-sight (NLOS) scene reconstruction, a challenging scenario where the scene of interest is not directly visible to the camera. Finally, we address the problem of reconstructing specular or glossy objects, which violate the commonly assumed Lambertian reflectance in 3D shape reconstruction algorithms, such as photometric stereo and multi-view geometry.

We tackle these problems by leveraging differential imaging, an imaging mechanism that takes multiple measurements with differential changes in illumination, camera, or the scene. Because of the infinitesimal changes in measurements, differential imaging systems can be made compact and portable. They are commercially available, such as light field cameras that capture images of a scene with differentially varying viewpoints. Besides the small form-factor, differential imaging also helps in solving challenging computer vision problems by locally linearizing originally nonlinear phenomena so that inverse problems become easier to solve.

Specifically, in this thesis, for single-shot depth from defocus, we replace the conventional RGB sensor in a lens-based imaging system with an off-the-shelf dual-pixel (DP) sensor, which emulates a single-shot differential stereo system with a tiny baseline. We exploit the defocus blur introduced by the lens and perform defocus map estimation and defocus deblurring from a single DP image. Next, in NLOS imaging, we apply differential imaging by densely scanning a visible surface using a time-of-flight system and extract a purely geometric feature, called Fermat paths, from radiometric measurements of photons bouncing between the visible surface and the NLOS scene. We then apply well-established tools in differential geometry to reconstruct the surface of NLOS objects using the collection of Fermat paths. Finally, we reconstruct specular or glossy objects following shape from specularity approaches, and propose a reconstruction algorithm that is also based on the theory of Fermat paths.

We hope this thesis will contribute to the ultimate goal of 3D reconstruction in the wild. The techniques developed in this thesis, especially the theory of Fermat paths, will also apply to other domains, including acoustic and ultrasound imaging, lensless imaging, and seismic imaging.

More Information

Thesis Committee Members:
Srinivasa G. Narasimhan, Co-chair
Ioannis Gkioulekas, Co-chair
Keenan Crane
Gordon Wetzstein, Stanford University