RI PhD Thesis Defense - Ruixuan Liu - Robotics Institute Carnegie Mellon University
Loading Events

RI Event

October

20
Tue
Ruixuan Liu PhD Student Robotics Institute,
Carnegie Mellon University
Tuesday, October 20
3:30 pm to 5:00 pm
Gates Hillman Center 4405
RI PhD Thesis Defense – Ruixuan Liu
Date: October 20, 2026
Time: 3:30–5:30 PM ET
Type:  RI PhD Thesis Defense
Location:GHC 4405
Title: Prompt-to-Product: Towards Robots that Imagine, Reason, and Build
Who: Ruixuan Liu

Abstract:
Recent advances in artificial intelligence (AI) and robotics have opened new possibilities for helping people turn ideas into physical assemblies. Large language models, multimodal foundation models, and other generative models can interpret user intent and generate designs, while robots can perform increasingly complex assembly tasks. Yet these capabilities often remain disconnected. A design that satisfies a user’s request may be physically unstable or beyond a robot’s ability to build. Meanwhile, a robot proficient at individual manipulation tasks may still depend on detailed, step-by-step commands. This thesis pursues the vision of PROMPT-TO-PRODUCT: robots that move beyond prescribed actions to help people imagine new assemblies, reason about their feasibility, and build them. Using brick assembly as a platform, it investigates the complete process from abstract user intent to physical construction. Its central argument is that reasoning about physics, assembly properties, and robot capabilities must bridge design generation and robotic execution, so that physical feasibility informs both the imagined assembly and the process of building it.

The first part of this thesis develops methods for reasoning about the physical properties of brick assemblies, laying the foundation for the design and construction methods that follow. We present STABLELEGO, a computational tool for analyzing structural stability, and a full-stack physics simulator that extends stability analysis to high-fidelity simulation of brick motions and interactions. The second part investigates how abstract user ideas can be translated into concrete assembly designs. We introduce BRICKGPT, which generates stable and buildable interlocking brick assemblies from text prompts. We then develop an agentic framework for zero-shot assembly design from text and image inputs, which moves beyond BRICKGPT’s restricted component library to support diverse components and articulated structures. The third part develops the manipulation and coordination capabilities needed for precise, long-horizon assembly. We present an end-of-arm tool that manipulates small bricks with high precision, and APEX-MR, an asynchronous coordination algorithm that enables multiple robot arms to build complex structures collaboratively. We then develop visuomotor policies through sim-to-real learning, enabling precise assembly with general-purpose grippers on lightweight robots and extending these capabilities beyond industrial platforms. The fourth part integrates reasoning, generation, and manipulation into the complete PROMPT-TO-PRODUCT system. It connects STABLELEGO, BRICKGPT, and the dual-arm construction system BRICKMATIC to transform natural-language descriptions into physical assemblies, with stability reasoning guiding both design generation and construction. End-to-end demonstrations and a user study evaluate the full workflow.
Together, these contributions connect expressive design generation with the physical reasoning and manipulation required for construction. They expand the range of assemblies that can be designed, the physical behaviors that can be simulated, and the robotic platforms that can build them. Ultimately, this thesis advances the vision of robots that imagine, reason, and build alongside people, helping users realize their ideas without specifying every step of design and construction.

COMMITTEE MEMBERS:

Changliu Liu (Chair)

Oliver Kroemer
Jean Oh
Jun-Yan Zhu
Ken Goldberg, UC Berkeley