Scaling up Camera Calibration and Amodal 3D Object Reconstruction for Smart Cities

Abstract

Smart cities integrate thousands of outdoor cameras to enhance urban infrastructure, but their automated analysis potential remains untapped due to various challenges. Firstly, the lack of accurate camera calibration information, such as its intrinsics parameters and external orientation, restricts the measurement of real-world distances from the captured video. To address this issue, we propose a scalable framework leveraging publicly available street-level imagery and map data to automatically reconstruct a metric 3D model of the surrounding scene, allowing for accurate calibration of in-the-wild traffic cameras around the world.

Secondly, the presence of occlusions poses significant challenges in object understanding. For example, objects in the scene may be partially occluded by other static or dynamic objects, truncated by the camera's field of view, or be self-occluded, i.e., only one side of an object is visible from a specific view. We present a holistic approach to handle such occlusions for amodal 3D shape reconstruction. The approach starts by learning occlusion categories with human supervision. Then, these learned categories are exploited in a novel framework that uses a mixed representation (keypoints, segmentations and shape basis) for objects to automatically generate a large physically realistic dataset of occlusions using freely available time-lapse imagery from traffic cameras. This dataset provides strong 2D and 3D self-supervision to a network that jointly learns amodal 2D keypoints and segmentations, which are then optimized to reconstruct 3D shapes under constraints provided by occlusion categories. Our system demonstrates significant improvements in amodal 3D reconstruction of heavily occluded objects captured at any time of the day from traffic, hand-held, and in-vehicle cameras, thus enhancing the potential of smart cities to utilize outdoor cameras for effective urban planning.