The first half of the thesis discusses RefAV, a benchmark for retrieving text-specified scenarios of interest from uncurated driving logs. AVs collect terabytes of observational data during normal fleet testing, with a large majority of it boring. Traditional scenario mining techniques are error-prone and prohibitively time-consuming, often relying on hand-crafted structured queries. We revisit spatio-temporal scenario mining through the lens of recent vision-language models (VLMs) to detect whether a described scenario occurs in a driving log and, if so, precisely localize it in both time and space. We introduce a large-scale dataset of 10,000 diverse natural language queries that describe complex multi-agent interactions relevant to motion planning. We evaluate several referential multi-object trackers and present an empirical analysis of our baselines. Notably, we find that naively repurposing existing VLMs yields poor performance, suggesting that scenario mining presents unique challenges. We discuss our recently held competition and share insights from the community.

The second half of the thesis explores training and evaluating vision-language-action (VLA) models for off-road driving. The text-centric VLM pretraining gives policies the potential to generalize to scenarios not observed during fleet testing. We recast driving as a text completion problem to fully leverage this pretraining. We aim to train a policy that is responsive to both long-horizon waypoints and free-form language commands such as “stay on the gravel” or “avoid the puddle”. We find that training naively on VLM-generated commands is not enough to elicit language following, as the image observation is more predictive of the future trajectory than the language command. Instead, we augment the observation with alternative trajectories collected from other times the robot visited a particular location. We show that adding these “counterfactual” trajectories decreases language following error by 0.28m ADE over a naively auto-labeled dataset. Finally, we perform closed loop evaluation in 3D reconstructions of previously unseen environments. We show our dataset annotation method improves the ability of an agent to navigate autonomously to waypoints hundreds of meters away while enabling language-based interventions at important decision points.