Time: 1:00–3:00 PM ET
Type: RI PhD Thesis Proposal
Zoom Link: LINK
Location: NSH 4305
Title: Grounding Task-Specialized Models in Object Priors for Generalizable Robot Manipulation
Who: Muhammet Yunus Seker
In manipulation skills, GTA-1 exploits task-relevant object keypoints and axes to transfer previously specified skills to novel objects without any retraining. GTA-2 extends this representation through four VLM agents that autonomously decompose task instructions, compose reusable controllers into task-specialized skills, assign execution parameters, and ground task-relevant geometric features in the observed scene. By separating abstract skill synthesis from scene-specific grounding, GTA-2 enables generated skills to be reused across different scene configurations while supporting targeted refinement of individual components. Real-robot evaluations on 14 manipulation tasks demonstrate zero-shot skill generation and generalization across different scene configurations.
Building on these contributions, the proposed research investigates how robots can accumulate and improve their capabilities through continued interaction. It explores agentic orchestration for progressively developing stable, reusable manipulation programs; persistent object-centric scene understanding that retains accumulated knowledge and uses interactive perception and exploration to reduce uncertainty; and compositional observation mechanisms for monitoring execution and supporting adaptation. The broader objective is to enable robots to progressively develop specialized capabilities around recurring objects and interactions while retaining the flexibility to handle unfamiliar situations.
COMMITTEE MEMBERS
Oliver Kroemer (Chair)
Jonathan Francis
Yonatan Bisk
Leslie Pack Kaelbling (Massachusetts Institute of Technology)
