Abstract:
Language models increasingly solve hard problems by exploring multiple reasoning paths at training and test time. But making this exploration effective requires balancing two goals that are often in tension: creativity and reliability. On the one hand, models need to generate diverse, non-myopic reasoning strategies rather than repeatedly sampling the same mistakes. On the other hand, they need mechanisms to verify and refine these attempts in a trustworthy way.

In this talk, I will argue that both goals require structured exploration. First, I study creative exploration in language models, showing that next-token prediction can be fundamentally limiting for open-ended reasoning and generation. I then show how explicitly structuring exploration across diverse reasoning modes can substantially improve test-time scaling. Finally, I turn to self-verification for reliable exploration, arguing that iterative improvement is most effective when verification is itself structured and informative, rather than simply making the model think longer. I also discuss evidence from adversarial settings suggesting that reliability does not come automatically from adding more verification components. Together, these results suggest that progress in reasoning will depend not just on more computation, but on learning to use that computation for structured exploration that supports both creativity and reliability.

Committee:
Aditi Raghunathan
Shubham Tulsiani
Daniel Fried
Zhiqiu Lin