Abstract:
A core challenge in vision is building representations that capture 3D scenes over time for both perception and generation. This thesis studies consistency across views, time, and modalities by moving from dense grid-based representations toward entity-centric scene representations that can be maintained across frames and used for interactive generation.

The first part of the thesis develops consistent 3D perception systems, including methods for temporal multi-camera 3D detection, depth completion, semi-supervised detection, and streaming semantic occupancy estimation. These works progressively move from dense spatial representations toward persistent, query-based representations that model both foreground objects and background structure over time.

The second part of the thesis presents LatentWorld, a generative model for entity-centric 4D scene generation. LatentWorld represents a scene as a sparse set of grounded 3D latents, assigning persistent latents to foreground actors while using multiple latents for background regions. Generation is factorized into layout, geometry, and motion, enabling temporally coherent semantic-occupancy rollouts with stable actor identity, explicit ego and actor motion, and direct scene-level control.

Together, these works show how consistent, entity-centric representations can serve as a common foundation for understanding dynamic 3D scenes and generating plausible, controllable 4D worlds.

Thesis Committee Members:
Kris Kitani (Chair)
Deva Ramanan
Shubham Tulsiani
Wei-Chiu Ma (Cornell University)