Tuesday, September 1
11:30 am to 1:00 pm
3305 Newell-Simon Hall
11:30 am to 1:00 pm
3305 Newell-Simon Hall
Scalable Vision-Language Models through Unified 2D and 3D Representations
Abstract:
Vision-language models have become remarkably capable on images and short videos, yet they still struggle with two abilities central to embodied intelligence: spatial understanding and long-range temporal reasoning. A major reason is representational: today’s models process videos as long sequences of 2D patches, so computation grows with observation length even when the underlying scene changes little. This thesis argues that organizing perception around persistent 3D structure rather than individual frames allows model complexity to scale with scene content rather than observation length, leading to efficient inference on long videos while providing a stronger foundation for spatial reasoning.
We introduce a unified representation based on 3D feature clouds that handles both 2D and 3D inputs within a single architecture. Because the same model trains on abundant 2D image-text data alongside available 3D data, it acquires better spatial understanding without sacrificing 2D performance. We demonstrate this on 3D instance segmentation (ODIN), extend it to broader vision-language tasks (UniVLG), and scale it to billion-parameter VLMs (Qwen-3D), showing consistent improvements in spatial understanding and inference efficiency. We then move beyond static scenes to dynamic videos. We develop 3D scene representations (TrackEverything) that disentangle static and dynamic content, deduplicate the scene across time, and track all points in 3D throughout long videos. These representations convert videos into concise spatiotemporal structures that grow with scene complexity rather than video duration, enabling new capabilities such as being able to track all points across all frames in long videos (1000+ frams).
Finally, we outline proposed work on two fronts: leveraging these dynamic 3D representations for spatial and motion reasoning in vision-language models, and scaling dynamic 3D tracking to real-world video data using heterogeneous supervision beyond synthetic data.
Together, this work makes the case for moving beyond 2D patch representations toward 3D-native models for better and more efficient video understanding.
Thesis Committee:
Katerina Fragkiadaki (Chair)
Deva Ramanan
Shubham Tulsiani
Leonidas Guibas, Stanford University