Learning from experience offers a way for language models to improve beyond the solutions available in their training data. Instead of relying only on demonstrations, a model can attempt problems, receive feedback, and learn from the outcomes. This process takes many forms, such as reinforcement learning with verifiable rewards, search at test time, training via self-improvement, and the integration of knowledge that arrives after training. In all of these scenarios, two capabilities matter: (1) exploring a broad range of solutions, and (2) learning from exploration to improve future performance. In practice, both still remain challenging — models often repeat similar attempts even when they fail, and learning from their outcomes is difficult when feedback is unreliable or when updates degrade existing capabilities. This thesis develops methods to address both challenges.

For learning to explore, we study why language models generate a narrow range of solutions and how to encourage broader exploration. We first examine the role of training objectives in simple algorithmic tasks, finding that next-token prediction is myopic, while multi-token objectives and input-side randomness produce more diverse and original outputs. We then study the effect of post-training, which concentrates a model’s reasoning into a small number of modes. We introduce mode-conditioning (ModC), which makes these modes controllable and allocates test-time samples across them.

For learning from exploration, we study how models can provide useful feedback to themselves when exploring and can incorporate new experience without losing existing capabilities. We first introduce self-trained verification (STV), which teaches a model to identify errors in its own solutions. The resulting verifier improves reasoning through iterative refinement, especially on hard problems. We then address the problem of forgetting when learning from new experience. We show that the degradation caused by off-policy training is not inherent to the data, but depends on how the update is applied. We therefore introduce grafting, which learns the update on an earlier checkpoint and transfers it to the post-trained model while masking the directions to which the post-trained model is most sensitive. Grafting outperforms more complex and expensive baselines on both new and old tasks.

Building on these results, the proposed work continues under the two directions. For learning to explore, I will study how to guide exploration on problems beyond a model’s current ability, with hints, reference solutions, and verifier feedback, without the model depending on or exploiting the guidance. For learning from exploration, I will study how new experience can be integrated with pretrained knowledge so that the two compose, and how a model can learn more from less experience by learning how to learn.