Abstract:

Robots operating in open-world environments will need to understand the scope of their capabilities, recognize their limitations, and determine how those capabilities can be expanded. This talk will review research from the Intelligent Autonomous Manipulation Lab on modeling robot capabilities, allocating learning resources to expand them efficiently, and structuring skills for a given scope of tasks. I will also discuss how tactile perception and dexterous hardware can broaden what robots are able to perceive and do.

The main focus will be our work on Grounded Task Axes, a framework for autonomously constructing structured manipulation skills. Rather than representing a skill as a monolithic policy or fixed library element, Grounded Task Axes compose task-tailored behaviors from controllers defined relative to semantically meaningful object keypoints and axes. Our recent work uses vision and language models to generate these skill structures, ground them in observed scenes, and set task-specific parameters. These structured representations support precise controller-level execution, interpretable human correction, and generalization across objects and tasks.