Abstract: Manipulation in agricultural and unstructured environments often involves contact-rich interactions with occluded objects. Most deployed systems treat contact as a hazard and rely on vision alone, which limits deployment from real-world field settings. This thesis adopts a different perspective: contact is a strategy for obtaining information where vision cannot provide. Touch can reveal additional spatial, temporal, and semantic cues, along with other task relevant latent states. It argues that augmenting vision with contact sensing, and conditioning learned policies on structured representations of task-relevant hidden states (inferred from sequential contacts), enables reliable manipulation where vision alone cannot disambiguate hidden states. In POMDP settings, we show that structured representations of task-relevant hidden states outperform end-to-end approach conditioned on raw observation history.

The dissertation develops this argument in three movements: establishing the limits of vision-only manipulation through a precise agricultural task where occlusion makes sub-centimeter alignment unreliable; showing that vibrotactile sensing via contact microphone arrays recovers both the spatial and semantic cues; and demonstrating that conditioning policies on structured hidden-state representations yields history-aware adaptive behavior on long-horizon tasks that end-to-end policies cannot reliably solve.

Committee:

Oliver Kroemer (chair), Carnegie Mellon University

George Kantor (co-chair), Carnegie Mellon University

Yonatan Bisk, Carnegie Mellon University