Recent progress in robotic foundation models has made broad, reusable robot competence increasingly plausible, but it has also brought a central challenge: how can robots continue to improve upon online deployment? This is especially acute in high-dimensional or contact-rich settings, where reactive, highly dexterous skills are critical but not trivial for BC priors. My thesis studies how to achieve reliable policy improvement from online experience via reinforcement learning (RL), grounded in the premise that scalable robotic adaptation requires reliable learning signals, including accurate critics and distribution-aligned data curation.

In the first part of the talk, I will present TD-M(PC)^2, a model-based RL framework for high-dimensional continuous control. This work identifies a structural mismatch between the model-based exploration and the model-free critic learning as a major source of value overestimation. To address this issue, I introduce constrained policy iteration that reduces out-of-distribution bootstrapping while preserving the advantages of test-time policy improvement via planning. The resulting method yields stable and sample-efficient learning. Second, I present PLD (Probe, Learn, Distill), a post-training framework for vision-language-action models. PLD uses offline warm-start to initiate and stabilize online RL, learning task-specific residual specialists, collecting high-quality, deployment-aligned recovery behaviors, and distilling them back into a pretrained generalist without requiring massive human effort. This enables self-improvement while maintaining generalization across simulation and real-world dexterous manipulation. Finally, I will discuss ongoing work on critic and reward modeling for long-horizon tasks that require precision and dexterity, where reliable evaluation and credit assignment remain significant bottlenecks. Taken together, the thesis suggests that progress in robotic autonomy depends not only on strong foundation priors but also on mechanisms for reliably assessing and bootstrapping experience.