Abstract:

Contact-rich manipulation requires reasoning about complex physical interactions. While vision and force sensing provide useful information, many interaction dynamics remain difficult to observe directly, especially under occlusion, rapid contact transitions, or distributed contact. This thesis explores how vibro-tactile sensing, also known as structure-born acoustic sensing, can serve as a practical and information-rich modality for understanding physical interaction in robotic manipulation.

We present three works that progressively study vibro-tactile sensing for contact-rich manipulation. First, we present acoustic-guided constraint learning for fast non-prehensile transport, where vibro-tactile sensing is used as a binary slip detector to learn motion-dependent friction effects, and incorporate them into optimization-based motion planning. Second, we present a multi-channel acoustic sensing system embedded in a parallel-jaw gripper, which predicts continuous in-hand slip direction and magnitude in real time for closed-loop manipulation. Finally, we introduce a visuo-acoustic framework that combines wearable active acoustic sensing with vision to estimate hand pose and contact during human-object interaction, improving robustness under occlusion and visual ambiguity.

Together, these works demonstrate that vibro-tactile sensing provides a scalable and effective modality for modeling, representing, and perceiving contact-rich interaction in robotic manipulation.

Committee:

Prof. Jeffrey Ichnowski (Chair)

Prof. Christopher G. Atkeson

Prof. Maxim Likhachev

Yufei Wang