Abstract:

Rapid prototyping systems demand both adaptability to new components and precision in physical interaction. In this thesis, we explore how geometry can be used in two complementary ways: as a prompt for detecting novel industrial objects, and to support fine-grained perception in precision tasks.

CAD-Prompted SAM3 enables instance segmentation directly from geometric specification, supporting detection of unseen objects without object-specific training. A geometry-driven manipulation pipeline extends this representation to action, integrating segmentation, pose estimation, and grasp generation for zero-shot robotic manipulation. Eye-in-Finger further leverages task-specific geometry with tool-integrated sensing, enabling sub-millimeter perception accuracy in cluttered assembly tasks.

Together, these works demonstrate how geometry-guided perception provides a scalable approach for rapid prototyping, enabling systems to both adapt to newly introduced components and achieve the precision required for reliable assembly.

Committee:
Changliu Liu (advisor)
Oliver Kroemer
Ruixuan Liu