Generative Robotics: Self-Supervised Learning for Human-Robot Collaborative Creation

Abstract

Robot automation is generally welcomed for tasks that are dirty, dull, or dangerous, but with expanding robotic capabilities, robots are entering domains that are safe and enjoyable, such as creative industries. Although there is a widespread rejection of automation in creative fields, many people, from amateurs to professionals, would welcome supportive or collaborative creative tools. Supporting creative tasks is challenging with real-world robotics because there are limited relevant datasets, creative tasks are abstract and high-level, and real-world tools and materials are difficult to model and predict. Learning-based robotic intelligence is a promising method for creative support tools, but since the task is so complex, common approaches such as learning from demonstration would require too many samples and reinforcement learning may never converge. In this thesis, we introduce several self-supervised learning techniques to enable a robot to teach itself to support humans in the act of creativity.

We formalize robots that support people in the making of things from high-level goals in the real world as a new field, Generative Robotics. We introduce an approach for supporting 2D visual art-making with paintings and drawings along with 3D clay sculpture from a fixed perspective. Because there are no robotic datasets for collaborative painting and sculpting, we designed our approach to learn from small, self-generated datasets to learn real-world constraints and support collaborative interactions. This thesis contributes (1) a Real2Sim2Real technique that enables a robot to create complex dynamics models from small, self-generated datasets of actions, (2) a method for planning robotic actions for long-horizon tasks in a semantically aligned representation, and (3) a self-supervised learning framework to adapt pretrained models to be compatible with robots and produce collaborative goals. We show how self-supervised learning can enable model-based robot planning approaches to paint collaboratively with humans using various painting mediums. Lastly, we generalize our approach from the painting to the sculpting domain, demonstrating that our approach generalizes to new materials, tools, action representations, and state representations.