Robotic Manipulation and Dense Tracking for Complex and Deformable Object Dynamics

Abstract

Many everyday human actions—such as adjusting the grip on a tool or folding clothes—require the ability to manage complex dynamics involving shifting contact, deformability, or high-speed motion. While these tasks are intuitive for humans, object manipulation under such conditions poses significant challenges for robotic systems. Enabling robots to perform complex dynamic manipulations would substantially expand their capabilities and real-world applicability.

This thesis investigates two complementary aspects to address these challenges from both the action and perception perspectives. The first part introduces SWIFT, a trial-and-error optimization method that enables a soft robotic hand to perform a challenging dynamic in-hand manipulation task—pen spinning—without relying on explicit dynamics modeling or simulation. This contribution demonstrates that dynamic, non-prehensile manipulation can be achieved using a surprisingly simple gradient-free optimization approach and highlights the promise of soft robotic systems in executing fast, contact-rich manipulation tasks.

The second part focuses on the perception side of manipulation and addresses the difficulty of accurately capturing the dynamic behavior of deformable objects. This is critical for downstream applications such as object-centric imitation learning or the creation of digital twins. To this end, the thesis presents DeformGS, a dense tracking method that leverages Gaussian splatting and a learned deformation model to accurately and densely track points on deformable objects from visual input.

Together, these two contributions advance the understanding of robotic manipulation in complex dynamic environments and suggest promising directions for the development of more capable, adaptive robotic systems.