Abstract:

Dexterous robotic manipulation is becoming increasingly crucial as robots transition from factory manufacturing to everyday human environments such as household assistance and healthcare support. However, operating in these unstructured settings presents significant challenges in both hardware and software. Achieving adaptability across diverse tasks requires multi-fingered robotic hands with high degrees of freedom, capable of dexterous manipulation, together with responsive, autonomous control policies for precise and robust skill execution. This thesis explores integrated systems for dexterous manipulation using multi-fingered robotic hands.

We first introduce DeltaHands, a modular dexterous hand framework based on Delta robots. DeltaHands are highly dexterous and are simple to fabricate using low-cost, off-the-shelf materials. Their modular Delta fingers enable flexible design configurations for different applications, while their parallel and translational kinematics simplify control despite the high degrees of freedom. This framework offers a broad hand design space for dexterous manipulation.

Building on DeltaHands, we develop Tilde, an imitation-learning-based in-hand manipulation system. To collect demonstrations, we propose two teleoperation methods: (1) a vision-based human hand motion tracking interface and (2) a kinematic twin for direct control. Using these demonstrations, we train control policies with diffusion-based imitation learning. We show that the learned policies can be deployed on a real-world DeltaHand to perform a variety of in-hand dexterous manipulation tasks.

To further reduce data collection effort and improve policy generalization, we introduce ExoStart, a real-to-sim-to-real learning pipeline that leverages sensorized exoskeleton demonstrations. By capturing direct human–object interaction without robots in the loop, we use these demonstrations to bootstrap a simulation-based auto-curriculum reinforcement learning method and then transfer the learned policies to real-world robots in zero-shot. Our approach requires fewer than 15 human demonstrations and relies only on sparse reward design, yet enables the learning of diverse and highly dexterous real-world robotic hand behaviors.

Finally, we propose two future directions: (1) enhancing fine-grained manipulation by integrating tactile sensors into fingertips and incorporating multi-sensing modalities into policy learning; (2) exploring hand structure design to enable whole-hand manipulation to improve the grasp stability and strength; and control strategies to transit dexterous fingertip manipulation to powerful grasp.

Thesis Committee Members:

Zeynep Temel (co-chair)

Oliver Kroemer (co-chair)

Nancy Pollard

Oliver Brock (TU Berlin)

Thesis proposal document draft