Date: May 21st, 2026
Time: 3:30 – 5:00 pm
Room: NSH Room 4305
Zoom:  https://cmu.zoom.us/j/98318417145

Title: Scaling Long-Tailed Driving Perception and Planning with In-the-Wild Videos

Abstract: Closed-loop driving, where methods produce actions a simulator reacts to, remains largely tied to driving logs from instrumented fleets. Thus, reliably driving in rare-but-critical scenarios is still elusive. Meanwhile, foundation models are increasingly common in autonomous driving, vision-language and video world models are tackling open-loop tasks like scene description and video generation. Although the internet has revolutionized language and image generation, planning in autonomous driving has not seen its ImageNet moment yet. Therefore, an opportunity exists to leverage internet-scale data and tackle long-tailed autonomous driving.

We focus on work zones as a representative long-tail scenario as they are a major source of disengagements for commercial systems today. Work zones uniquely combine rare objects (e.g., construction vehicles, arrow boards), unusual layouts (e.g., temporary closures, crossing yellow lines), and unpredictable behaviors (e.g., flaggers, sudden merges). These safety-critical scenarios are considered difficult to simulate at scale.

First, we mine long-tailed driving videos from a massive corpus. We find that foundation models fail at work zone perception and fine-tuning on our data combined with simple priors makes them effective. Second, we introduce a resource-efficient, geometry-based prior that improves scene perception and long-tail object detection. Third, we focus on long-tailed closed-loop planning and develop an anytime language-action planner capable of real-time trajectory generation and contextual textual reasoning. Furthermore, by developing a novel rules-based planner that effectively handles current benchmark scenarios, we show that existing closed-loop driving benchmarks are insufficient for evaluating long-tailed behaviors.

Finally, this thesis proposes a framework that, by carefully composing geometry-aware methods, street-view imagery, and foundation models, lifts monocular videos into metric, geo-referenced 4D driving logs compatible with existing simulators. Using this framework, we create a new long-tail planning benchmark and propose to uncover insights and study the geographic scaling behavior of state-of-the-art planning methods.

Ultimately, to advance autonomous driving beyond fleets, we argue scaling of training and evaluation is achievable by harnessing internet-scale data while grounding foundation models with geometric and physical priors.

Srinivasa Narasimhan, Chair
Deva Ramanan
Maxim Likhachev
Christoph Mertz
Manmohan Chandraker, UC San Diego