Abstract:
Assembly products are ubiquitous in our lives, for example, chairs, tables, couches, drawers, and more. Due to the complex interactions between components, creating such products typically demands significant manual effort in 1) designing the assembly and 2) constructing the product. This thesis seeks to reduce the required manual effort by automating the creation process of assembly products. To this end, we introduce Prompt2Product, an automated pipeline that generates real-world assembly products from natural language prompts. Specifically, we leverage LEGO as the assembly platform and automate the process of creating LEGO assembly structures.

Understanding the physical properties of LEGO assemblies is essential for both designing and constructing LEGO structures. In the first part of the thesis, we build a physics reasoning module that estimates the physical property, i.e., structural stability, of a given LEGO structure. The proposed physics reasoning reliably and efficiently estimates internal stresses at brick connections, enabling accurate assessment of the structural stability of LEGO assemblies. In the second part, we introduce LEGOGPT, the first approach for generating physically buildable LEGO assembly structures from intuitive text prompts. In the third part, we present the design of our robot embodiment and techniques for learning robust manipulation policies. These capabilities enable robots to manipulate tiny LEGO bricks and perform high-precision LEGO assembly tasks. In the fourth part of the thesis, we develop a bimanual robotic system capable of constructing complex, customized, and delicate LEGO structures. To the best of our knowledge, this is the first robotic system capable of performing custom LEGO assembly using standard commercial LEGO bricks.

The completed work demonstrates Prompt2Product, which automates the creation of assembly products from text prompts. We propose further enhancements to extend its functionality. From the design generation standpoint, we propose to extend beyond standardized LEGO bricks and explore methods for generating high-fidelity and functional LEGO assemblies. From the assembly execution perspective, we propose to 1) develop failure detection and recovery mechanisms to improve system robustness, and 2) investigate hierarchical assembly strategies to enhance efficiency and dexterity. From the robot embodiment perspective, we propose to deploy Prompt2Product to more robot embodiments, e.g., humanoid robots.

Thesis Committee Members:
Changliu Liu (Chair)
Oliver Kroemer
Jean Oh
Jun-Yan Zhu
Ken Goldberg (University of California, Berkeley)

More Information