Abstract:
With growing expectations for autonomous robot deployment in unstructured, real-world environments, these systems must operate efficiently while perceiving and interpreting complex scenes to navigate dynamic, cluttered conditions. Robust performance in these settings require handling occlusions, clutter, and ambiguous visual cues; challenges exacerbated by the limited semantic understanding in standard visuomotor policy frameworks. This thesis investigates how incorporating semantic knowledge can enhance the robustness and generalization of perception-manipulation systems, with a particular emphasis on agricultural applications. We propose two complementary research directions to address this gap. First, we introduce Semantics-Guided Training, a method that guides visuomotor policy learning to attend to task-relevant features by regularizing saliency maps with semantic labels. Second, we explore Interactive Perception for Occlusion Handling, which leverages semantic priors to adapt perception and manipulation strategies based on the categorical and spatial characteristics of occluding elements. These approaches build on prior work on occlusion reasoning and visual manipulation, aiming to support more reliable behavior in real-world agricultural tasks. Ongoing and future work investigates the integration of semantic priors into both training and policy execution to improve task performance, sample efficiency, and adaptability across diverse environments.

Thesis Committee Members:
George Kantor (Chair)
Oliver Kroemer
David Held
Maren Bennewitz (University of Bonn)

More Information