Beyond Vision-Language-Action Models: Adapting, Steering, and Accelerating Generalist Robot Policies - Robotics Institute Carnegie Mellon University
Loading Events

MSR Thesis Presentation

July

9
Thu
Sungjae Park MSR Student Robotics Institute,
Carnegie Mellon University
Thursday, July 9
2:00 pm to 3:00 pm
GHC 9115
Beyond Vision-Language-Action Models: Adapting, Steering, and Accelerating Generalist Robot Policies

Abstract: Generalist robot policies, vision-language-action models that combine a large pretrained vision-language model backbone with a diffusion or flow-matching action head, are increasingly capable, yet hard to deploy in the real world. Three gaps separate such a policy from a deployable one: a data gap (adapting to a new task still demands task-specific teleoperation data), an inference gap (the policy samples its action distribution with no control over how conservative or diverse the result is), and an architecture gap (a large policy is too slow to replan often, so it must run its action chunks open-loop and cannot react mid-motion). This thesis argues that these gaps can be closed not by training larger models on more robot data, but by changing how a pretrained policy generates its actions, with little or no additional training.

We develop three methods. DemoDiffusion (data) imitates a single human demonstration instead of collecting teleoperation data: it retargets the human hand motion into a coarse robot trajectory, then uses a frozen generalist diffusion policy to refine it into plausible robot actions. It needs no task-specific or paired human-robot data, and succeeds even where the base policy fails outright. Temporal Score Rescaling (inference) rescales the learned score/flow at inference to draw from a sharper or broader distribution than the model was trained on. It is training-free, works with any diffusion or flow model, and improves image generation, depth, pose, and protein design alongside robot policies. πR² (architecture, inference) builds on diffusion forcing to split conditioning into a fast proprioceptive channel and a slow, asynchronously updated vision-language channel, so the policy reacts to fresh proprioception while tolerating stale vision. A latency-adaptive schedule lets a single model handle varying inference latency and emit actions in a single denoising step, making a large policy reactive and real-time, several times faster than the original.

Together, these methods adapt, steer, and accelerate a pretrained policy, taking vision-language-action models beyond what they can do as trained and toward real-world deployment.

Committee: 
Prof. Shubham Tulsiani (advisor)
Prof. Katerina Fragkiadaki
Andrew Wang