Abstract:
Language modeling progress is largely evidenced by steadily rising scores on benchmarks of increasing apparent difficulty. From this, the field infers increasingly general capabilities, many of which presuppose robust world modeling. Interpreting a score, however, requires understanding both the task’s computational requirements and how the test-taker generalizes from them. Unlike humans, who demonstrably generalize well, large language models (LLMs) often do not: they instead learn heuristics fit to the minimal sufficient computational requirements of a task, which may be far simpler than the task appears to demand. This talk extends this analysis to embodiment, where multimodal LLMs (MLLMs) serve as the perception and reasoning core of embodied agents, whose reliable deployment depends on benchmarked capability including a robust world model of the agent’s environment.

We evaluate world model robustness by independently manipulating object count, duplication, trajectory length, and viewpoint change in a controlled synthetic benchmark of multi-frame egocentric trajectories, and find that frontier MLLMs degrade sharply as difficulty increases, while humans remain near ceiling. We then characterize embodied question answering (EQA) task demands through three nested paradigms: selecting relevant observations, clustering nearby observations into local spatial-semantic units, and propagating semantic state across experience. Stratifying questions by the weakest sufficient paradigm yields a difficulty progression. Prominent EQA benchmarks predominantly test only the weakest paradigm, single-frame selection, and so we introduce Campus-Bench, three multi-hour, campus-scale episode histories with questions stratified into selection and propagation regimes, directly comparing model performance on the two over the same episodes. We additionally develop a diagnostic method in which an MLLM incrementally constructs and traverses a hierarchical spatial-semantic memory, executing propagation explicitly. A frontier long-context MLLM performs strongly on selection but collapses on propagation; the same model leveraging our method dramatically improves it while remaining competitive on selection, suggesting models struggle to maintain state internally.

From this, we argue that frontier MLLMs do not yet maintain the robust world models their benchmark scores suggest, and that the field needs to make progress in formalizing tasks’ computational requirements and evaluating along their progressions of difficulty.

Thesis Committee:
Yonatan Bisk (Advisor)
Wennie Tabib
Haochen Zhang