Abstract:

Graphical User Interface (GUI) agents need strong planning—what to do next—and grounding—where to click next—to solve user tasks. Yet these agents remain unreliable, and standard metrics such as task success or next-action accuracy often obscure why they fail. In this talk, I argue that reliable GUI agents require fine-grained diagnosis of core agentic capabilities, studied through two complementary directions.

First, I examine planning failures in GUI agents. Existing work can often identify planning errors, but rarely pinpoints which component failed. This is largely because there is no structured breakdown of the skills an agent must master to plan effectively. This thesis identifies several such skills and proposes synthetic data pipelines to diagnose failures across them. Surprisingly, even the strongest agents often fail on basic skills, such as temporal reasoning over webpage sequences. I further show that performance on these fine-grained planning skills correlates with downstream task success, suggesting a cheaper way to predict agent performance without expensive evaluations.

Second, I diagnose failures in GUI grounding. Traditional accuracy-based metrics do not reliably expose hidden failure modes, and I argue that sensitivity-based metrics are needed to compare models. In particular, I find that no model consistently clicks the same UI element, such as a calendar button, across diverse user scenarios, apps, and operating systems. This raises concerns about whether these models truly understand what they click. To operationalize this diagnosis, I propose a novel data generation pipeline and a GUI grounding diagnosis agent that automatically generates diverse instructions, uses model feedback, and iteratively identifies failure cases across state-of-the-art grounding models. Results show a significant mismatch between model rankings under accuracy-based metrics and our sensitivity-based metrics, highlighting the need for more reliable agent performance measures.

Together, this thesis provides diagnostics that can guide model development and advance more robust, trustworthy interactive AI systems. Our future work delineates how each of these diagnostics can serve as a simple intervention to improve existing GUI agent pipelines.

Committee:

Fernando De La Torre, co-chair
Andrea Bajcsy, co-chair
Louis-Philippe Morency
Yinong (Oliver) Wang