Abstract: The physical properties and dynamics that decide how an object or environment responds to a robot’s actions are often impossible to determine from visual observation alone. An object’s mass distribution and friction, the kinematics of an articulated object: these latent factors dictate the correct action, yet they leave little or no trace in a single image. This partial observability makes a purely feed-forward visual policy fundamentally limited, and even a strong policy will inevitably make mistakes when deployed, whether from this latent ambiguity or imperfect perception.

This thesis argues that both the missing information and the means to recover from error are supplied by the robot’s own history of interaction: by acting, observing the outcome, and reasoning about the mismatch between what was expected and what occurred, an agent can infer the underlying dynamics online and adapt on the fly.

We develop this idea along two complementary axes. Through verification, we introduce HAVE, a History-Aware VErifier that scores action proposals from a generative policy by reasoning about past actions and their outcomes. We prove that any better-than-chance verifier improves expected reward over the generator alone, and validate it across articulated objects, multi-modal doors, and uneven-mass objects in simulation and the real world. Through test-time training, we introduce SCOUT, a dynamics-aware meta-learning framework that couples a policy with a forward dynamics model through a shared belief latent; at deployment an inner loop revises this belief by minimizing the error between expected and observed outcomes, while a meta-learned outer loop ensures the belief steers the policy correctly. SCOUT adapts substantially faster than history-conditioned baselines and directly sim-to-real transfers to real-world tasks.

Together, these methods show that treating interaction history as a rich, structured supervision signal, rather than a sparse scalar reward, yields manipulation policies that adapt quickly and robustly to the hidden physical properties and dynamics of objects and environments, recovering from their own mistakes along the way.

Committee:
Prof. David Held (advisor)
Prof. Andrea Bajcsy
Mihir Prabhudesai