Think Globally, Solve Locally: Non-sequential Planning for Robotic Manipulation - Robotics Institute Carnegie Mellon University

PhD Thesis Proposal

May

11
Mon
Itamar Mishani PhD Student Robotics Institute,
Carnegie Mellon University
Monday, May 11
8:30 am to 10:00 am
Newell-Simon Hall 1305
Zoom Link Button
Think Globally, Solve Locally: Non-sequential Planning for Robotic Manipulation
Abstract:
Robotic manipulation requires reasoning that bridges local competence over fine-grained dynamics with the  construction of valid long-horizon plans. This mirrors human reasoning, where fast, automatic processes propose locally plausible actions from experience, while slower deliberation integrates them into a whole. Furthermore, evidence from cognitive science suggests that humans do not reason sequentially from start to finish; instead, they anchor on intermediate landmarks, plan from multiple directions, and let local insights reshape global structure.
Yet, current approaches in robotics typically formulate planning as a monolithic, unidirectional search from an initial state toward a goal or in the opposite direction. This thesis argues for a departure from that paradigm. We propose Non-Sequential Deliberative Planning, a framework that distributes deliberation across multiple local regions in the problem space, rather than committing to a systematic, directed reasoning. By exploring simultaneously from these regions, local solvers (whether geometric, heuristic, or generative) operate where they are most effective, while a global search composes them into a coherent plan with formal guarantees.

We instantiate this principle across three algorithmic regimes. For high-dimensional motion planning, we introduce Multi-Graph Search (MGS), which identifies key states as intermediate landmarks and simultaneously grows search trees from each, merging local subgraphs into a global solution with provable completeness and bounded suboptimality guarantees.
Second, for contact-rich manipulation, we present MOSAIC, which treats physics-validated skills as local competences. It composes sequences of skills, such as pushing or grasping, through a non-sequential search that connects local regions of reliable execution. Third, for scenarios where deliberation time is severely limited, we develop methodologies that shift non-sequential reasoning to an offline phase. By integrating manipulation behaviors directly into preprocessing, we generate motions whose manipulation outcomes are provably reliable, and the deliberation time is guaranteed to be within a user-defined time bound.

To complete this thesis, we consider three extensions. First, MOSAIC relies on physics simulation to estimate the outcome of contact-rich interactions during online planning—a significant computational bottleneck. We propose to address this by utilizing an offline phase to learn proxies and construct data structures that enable efficient online planning over long horizons. Second, manipulation skills are typically designed and learned for interactions between a robot and a single object. Real-world deployment, however, brings scenes with many movable objects, and tracking them jointly causes a combinatorial explosion in the planning state space. To address this, we propose to extend our prior framework with partial state planning, in which the global search operates over decoupled, object-centric representations and evaluates multi-object interactions only when necessary. Third, drawing on our prior work in multi-robot coordination—Experience-Accelerated Multi-Robot Planning (xECBS) and Multi-Robot Multi-Model Diffusion (MMD)—we plan to extend the non-sequential planning architecture to multi-arm settings, enabling concurrent execution across multiple manipulators.

Ultimately, this thesis establishes a unified framework for non-sequential decision making, composing fast local competences into globally sound manipulation plans across a range of real-world settings.

Thesis Committee:
  • Prof. Maxim Likhachev (Chair)
  • Prof. Changliu Liu
  • Prof. David Held
  • Prof. Oren Salzman (Technion University)

Thesis proposal document draft