Generalist robot policies—such as vision-language-action models (VLAs)—are trained end-to-end on large observation-action datasets and can perform increasingly diverse manipulation tasks. Yet these policies remain unreliable amid the complexity of real environments and user contexts. A common remedy is to finetune the entire policy for a particular deployment, but this may degrade the pretrained capabilities that make generalist policies valuable. This raises the central question of this thesis: how should robot policies be adapted to particular users and environments while preserving their generality?

This thesis argues that maximizing adaptation while minimizing loss of generality requires identifying what should be adapted and choosing how to adapt it in the least invasive way. For example, a deployment may call for changes to the task specification (what the user wants done), the state representation (which aspects of the scene matter for the task), or the behavioral support (the range of actions the policy can produce). At each of these levels, the desired behavior may already lie within the policy’s pretrained capabilities and only need to be elicited, or the policy may need to acquire new knowledge or skills through additional training.

We study two corresponding adaptation mechanisms: inference-time steering, which guides pretrained action distribution toward the target while keeping policy parameters fixed, and targeted training, which learns additional context or behaviors through limited parameter updates. For steering, we first propose language-based verification methods that check generated actions against user instructions and reasoning plans. Next, to handle incomplete state information in contact-rich manipulation, we introduce a bi-level visual and tactile steering algorithm. Third, to steer the action distribution in open-world settings, we contribute a world model that predicts the consequences of candidate behaviors while balancing fidelity, temporal consistency, and inference efficiency.

Steering, however, can only elicit behaviors the pretrained policy already supports. We turn to targeted training methods with parameter-efficient updates to improve the base policy via learning a compact multi-stream memory that conditions the policy on history-dependent state, and residual policies that incorporate user corrections to expand the policy’s behavioral capabilities. Finally, the proposed work revisits the role of meta-learning for adapting generalist robot policies. We propose separating shared representation learning across tasks from task-specific action learning using limited demonstrations. The goal is adaptation with fewer demonstrations and trainable parameters, with explicit retention of pretrained capabilities

Andrea Bajcsy (Chair)

Guanya Shi

Oliver Kroemer

Anirudha Majumdar,  Princeton University