BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Robotics Institute Carnegie Mellon University - ECPv6.15.12.1//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-ORIGINAL-URL:https://www.ri.cmu.edu
X-WR-CALDESC:Events for Robotics Institute Carnegie Mellon University
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20250309T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20251102T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20260308T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20261101T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20270314T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20271107T060000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260911T143000
DTEND;TZID=America/New_York:20260911T153000
DTSTAMP:20260912T191612
CREATED:20260828T164708Z
LAST-MODIFIED:20260911T212228Z
UID:153335-1789137000-1789140600@www.ri.cmu.edu
SUMMARY:Augmenting Bee Colonies with Robotics and AI Technologies for Ecosystem Support
DESCRIPTION:Abstract: Earth’s ecosystems are facing a rapid decline in biodiversity\, with honeybees —keystone pollinators critical to ecosystem stability— being among the most affected. The EU-funded RoboRoyale project addresses this crisis by integrating advanced robotics and AI to augment the beehive\, enabling observation at unprecedented resolutions and scales. Featured on the cover of Science Robotics and receiving the 6th Edge of Government Award at the World Government Summit in 2024 our system tracks the Queen’s behaviors\, colony efficiency\, comb states\, and long-term foraging activities\, while advancing micro-robotic intervention capabilities to support hive health. In this talk\, I will discuss the challenges of developing this system\, share key findings regarding complex social interactions\, and explore the future potential of bio-hybrid research. \nBio: Erol Şahin is a Professor of Computer Engineering at Middle East Technical University (METU) and the founding Director of the Center for Robotics and AI (ROMER). Established with over 5 million Euros in funding\, ROMER spans 25\,000 square feet of state-of-the-art facilities\, including prototyping workshops\, specialized research arenas\, and advanced robotic platforms. Dr. Şahin earned his PhD in Cognitive and Neural Systems from Boston University\, following a BSc in Electrical and Electronics Engineering from Bilkent University and an MSc in Computer Engineering from METU. Before assuming his current position\, he worked as postdoctoral researcher at the Université Libre de Bruxelles.   Between 2013 and 2015\, Dr. Sahin spent two years at the Robotics Institute of Carnegie Mellon University during his sabbatical. His research interests include swarm robotics\, robotic learning\, and human-robot interaction—work that has secured more than 2.5 million Euros from the European Union\, TUBITAK\, and industrial partners. Notably\, his contributions to robotic learning were awarded a 53-DOF iCub humanoid platform through the RobotCub project in 2007. Dr. Şahin has edited several journal special issues and books\, currently serves as an Associate Editor for Adaptive Behavior\, and is a member of the Editorial Board for the Swarm Intelligence journal.
URL:https://www.ri.cmu.edu/event/augmenting-bee-colonies-with-robotics-and-ai-technologies-for-ecosystem-support/
LOCATION:1403 Tepper School Building
CATEGORIES:RI Seminar,Seminar
ATTACH;FMTTYPE=image/jpeg:https://www.ri.cmu.edu/app/uploads/2026/08/erol-sahin.jpg
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260911T120000
DTEND;TZID=America/New_York:20260911T130000
DTSTAMP:20260912T191612
CREATED:20260901T200426Z
LAST-MODIFIED:20260901T200426Z
UID:153397-1789128000-1789131600@www.ri.cmu.edu
SUMMARY:Computational Lensing
DESCRIPTION:Abstract: From the cameras in our phones to the lenses in head-mounted displays\, optics shape both how we capture the world and how we experience virtual reality. Most conventional lenses are designed to bring a single plane into focus. In this talk\, we will discuss a new class of computational lens—referred to as a Split-Lohmann lens—that provides spatially varying control over focal length. This capability is achieved by combining a phase-only spatial light modulator with the cubic phase plates used in Lohmann/Alvarez focus-tunable lenses. The resulting computational lens enables new imaging and display capabilities\, including the ability to (i) make a flat display appear to have three-dimensional shape\, and (ii) capture all-in-focus images of highly non-planar scenes.
URL:https://www.ri.cmu.edu/event/computational-lensing/
LOCATION:Newell-Simon Hall 4305
CATEGORIES:Faculty Events
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260911T100000
DTEND;TZID=America/New_York:20260911T113000
DTSTAMP:20260912T191612
CREATED:20260908T140601Z
LAST-MODIFIED:20260908T140601Z
UID:153484-1789120800-1789126200@www.ri.cmu.edu
SUMMARY:From Simulation to the Real World: A Multi-Level System for Object Navigation with Vision-Language Models
DESCRIPTION:Abstract: \nObject navigation (ObjectNav) asks a robot to find an instance of a target object category in an unknown environment\, which demands perception\, spatial reasoning\, and long-horizon decision making at once. Today’s autonomous robots excel at mapping and moving through space yet lack high-level semantic intelligence\, while vision-language models (VLMs) offer rich commonsense reasoning but limited 3D spatial grounding and long-term spatial consistency. Most existing VLM-based navigation methods treat the model as a black-box oracle\, querying it at every step on unstructured local observations\, which leads to redundant backtracking\, inefficient exploration\, and brittle behavior outside clean simulation. This thesis argues that ObjectNav is a system-level problem rather than a single-policy learning task: its sub-challenges of semantic understanding\, complex spatial structure\, and long-horizon planning should be explicitly decoupled and handled by cooperating modules\, with the VLM asked to reason only at the level where it is reliable. We build such a system and carry it step by step from simulation to floor-scale\, cross-embodiment deployment in the real world. \nWe first develop the core of this system in simulation\, where the robot incrementally organizes what it has seen into a structured scene representation and the VLM reasons only at a high level over it\, while efficient geometry-based exploration handles fine-grained navigation. This design achieves state-of-the-art success rate and navigation efficiency across four widely used benchmarks. We then bring the system into the real world and extend it into three cooperating levels that decouple semantic reasoning\, navigation planning\, and motion control. At the high level\, the structured scene representation summarizes the environment and the VLM provides semantically grounded navigation guidance over it. At the mid level\, a hierarchical room-based navigation strategy reserves VLM guidance for room-level decisions\, which makes effective use of its reasoning while keeping the system efficient. At the low level\, planned waypoints are executed by embodiment-specific motion control. Because only the lowest level depends on the robot\, the same system runs on a custom-built wheeled robot\, the Unitree Go2 quadruped\, and the Unitree G1 humanoid. Across 190 real-world experiments\, it substantially improves success rate and navigates 4-5x more efficiently than existing baselines. To our knowledge\, it is the first system to reliably and efficiently complete floor-scale\, long-range object navigation in complex real-world environments. Together\, these results show that real-world ObjectNav is solved not by a larger model or a single end-to-end policy\, but by a system that balances semantic intelligence with spatial reliability and isolates embodiment-specific control from embodiment-invariant reasoning. \nCommittee:\nJean Oh (advisor)\nJi Zhang\nZhixuan Liu
URL:https://www.ri.cmu.edu/event/from-simulation-to-the-real-world-a-multi-level-system-for-object-navigation-with-vision-language-models/
LOCATION:NSH 3305
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260904T143000
DTEND;TZID=America/New_York:20260904T153000
DTSTAMP:20260912T191612
CREATED:20260828T164202Z
LAST-MODIFIED:20260910T180458Z
UID:153332-1788532200-1788535800@www.ri.cmu.edu
SUMMARY:CANCELED - Trust\, Sensing\, and Learning for Provable Multi-Robot Performance
DESCRIPTION:Seminar Canceled \nThis seminar has been canceled and may be rescheduled for a future date. Please check back for updates. ×Abstract: Multi-robot systems are physically embodied networks — they sense\, move\, and communicate through the physical world. The bar for safe decision-making rises as these systems enter safety-critical\, real-world settings where they must perform well under uncertainty. Our work shows that physicality is a resource against the two kinds of uncertainty they face: intentional (or adversarial)\, where data is manipulated by malicious agents\, and natural\, where aspects of the environment are simply unknown. Most of this talk concerns intentional uncertainty. Here\, one way to exploit physicality is by using communication as a sensor. Because the signals robots exchange are difficult to forge\, they carry evidence that can be cross-validated to yield a quantifiable likelihood that an agent’s data is trustworthy. This is the foundation of cy-trust\, in which stochastic observations of trust model an agent’s trustworthiness probabilistically from physical rather than cryptographic evidence. Each neighbor’s contribution is then weighted by its trust value. Under this framework\, we show that consensus\, distributed optimization\, and other core coordination tasks admit almost-sure convergence with bounded deviation from their nominal performance\, even when malicious agents exceed half of a node’s connectivity\, past the classical Byzantine bound. We support this finding with both theory and hardware experiments under adversarial attack. Against natural uncertainty\, we show that real-time sensing can be folded into rollout-based reinforcement learning\, where the same machinery reweights futures rather than neighbors. We apply this idea to routing a fleet of robots to stochastically appearing demand and\, with Project CETI\, to the first autonomous robotic rendezvous with sperm whales at sea. Finally\, we preview some of our future work combining trust with long-horizon sequential decision-making\, targeting planning that stays provably resilient when the data informing the plan may itself be corrupted. \nBio: Stephanie Gil is the John L. Loeb Associate Professor of Engineering and Applied Sciences at Harvard University and an Associate Faculty member of the Kempner Institute. Her research focuses on trust and coordination in multi-robot systems\, at the intersection of robotics\, communication\, and learning. Her contributions have been recognized through the DARPA Young Faculty Award (2024)\, the Office of Naval Research Young Investigator Award (2021)\, and the National Science Foundation CAREER Award (2019). She was named a 2020 Sloan Research Fellow for her work at the intersection of robotics and communication. She earned her Ph.D. at CSAIL at MIT\, specializing in multi-robot coordination and control\, and her B.S. at Cornell University.
URL:https://www.ri.cmu.edu/event/trust-sensing-and-learning-for-provable-multi-robot-performance/
LOCATION:1403 Tepper School Building
CATEGORIES:RI Seminar,Seminar
ATTACH;FMTTYPE=image/jpeg:https://www.ri.cmu.edu/app/uploads/2026/08/Stephanie-Gil_SQUARW.jpg
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260904T133000
DTEND;TZID=America/New_York:20260904T143000
DTSTAMP:20260912T191612
CREATED:20260904T131506Z
LAST-MODIFIED:20260904T131506Z
UID:153464-1788528600-1788532200@www.ri.cmu.edu
SUMMARY:RI Faculty Business Meeting
DESCRIPTION:Meeting for RI Faculty.\nIn person location – NSH 4305.\nZoom link available via calendar invite.
URL:https://www.ri.cmu.edu/event/ri-faculty-business-meeting-33-2-2-2-2-2-2-2-2-2-2-2-2-2-2-2-2-2-2-2/
LOCATION:Newell-Simon Hall 4305
CATEGORIES:Faculty Events
ATTACH;FMTTYPE=image/png:https://www.ri.cmu.edu/app/uploads/2023/11/ri-new-mark-512-512-transparent.png
ORGANIZER;CN="RI Director's Office":MAILTO:lynnetta@cs.cmu.edu
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260901T113000
DTEND;TZID=America/New_York:20260901T130000
DTSTAMP:20260912T191612
CREATED:20260820T203402Z
LAST-MODIFIED:20260820T203402Z
UID:153235-1788262200-1788267600@www.ri.cmu.edu
SUMMARY:Scalable Vision-Language Models through Unified 2D and 3D Representations
DESCRIPTION:Abstract:\nVision-language models have become remarkably capable on images and short videos\, yet they still struggle with two abilities central to embodied intelligence: spatial understanding and long-range temporal reasoning. A major reason is representational: today’s models process videos as long sequences of 2D patches\, so computation grows with observation length even when the underlying scene changes little. This thesis argues that organizing perception around persistent 3D structure rather than individual frames allows model complexity to scale with scene content rather than observation length\, leading to efficient inference on long videos while providing a stronger foundation for spatial reasoning.\n\nWe introduce a unified representation based on 3D feature clouds that handles both 2D and 3D inputs within a single architecture. Because the same model trains on abundant 2D image-text data alongside available 3D data\, it acquires better spatial understanding without sacrificing 2D performance. We demonstrate this on 3D instance segmentation (ODIN)\, extend it to broader vision-language tasks (UniVLG)\, and scale it to billion-parameter VLMs (Qwen-3D)\, showing consistent improvements in spatial understanding and inference efficiency. \nWe then move beyond static scenes to dynamic videos. We develop 3D scene representations (TrackEverything) that disentangle static and dynamic content\, deduplicate the scene across time\, and track all points in 3D throughout long videos. These representations convert videos into concise spatiotemporal structures that grow with scene complexity rather than video duration\, enabling new capabilities such as being able to track all points across all frames in long videos (1000+ frams). \nFinally\, we outline proposed work on two fronts: leveraging these dynamic 3D representations for spatial and motion reasoning in vision-language models\, and scaling dynamic 3D tracking to real-world video data using heterogeneous supervision beyond synthetic data. \nTogether\, this work makes the case for moving beyond 2D patch representations toward 3D-native models for better and more efficient video understanding. \nThesis Committee:\nKaterina Fragkiadaki (Chair)\nDeva Ramanan\nShubham Tulsiani\nLeonidas Guibas\, Stanford University\n\nThesis Proposal Link
URL:https://www.ri.cmu.edu/event/scalable-vision-language-models-through-unified-2d-and-3d-representations/
LOCATION:3305 Newell-Simon Hall
CATEGORIES:PhD Thesis Proposal,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260831T153000
DTEND;TZID=America/New_York:20260831T163000
DTSTAMP:20260912T191612
CREATED:20260825T175254Z
LAST-MODIFIED:20260825T175617Z
UID:153282-1788190200-1788193800@www.ri.cmu.edu
SUMMARY:My Bitter Lesson with Computer Graphics
DESCRIPTION:Abstract:  In his essay “The Bitter Lesson\,” Richard Sutton argued that general methods leveraging computation ultimately outperform hand-crafted ones. In this talk\, I share my own version of this lesson\, learned the hard way over a decade in computer graphics. Where does the bitter lesson apply to graphics\, and where does it not? I argue that the answer depends on whether a problem genuinely requires 3D\, physics\, and control\, or if it simply produces 2D pixels. I will also share a few advices for graduate students starting their research today\, showing which directions will compound in value over a decade\, and which will be quietly subsumed by the next scale-up. \nBio:  Giljoo Nam is a research scientist at Meta focused on Physical AI\, building systems that understand the structure and behavior of the 3D physical world. His research bridges computer vision and graphics\, with focus areas including generative AI\, 3D reconstruction\, motion tracking\, inverse rendering\, computational imaging\, and human modeling. He earned his Ph.D. in Computer Science from KAIST in 2019.. \nHomepage:   https://giljoonam.github.io/ \n  \nSponsor: \nThe VASC seminar is generously sponsored by HeyGen\, an all-in-one AI-powered video generation platform that leverages advances in computer vision\, generative modeling\, and multimodal learning to make high-quality video creation both scalable and accessible.
URL:https://www.ri.cmu.edu/event/my-bitter-lesson-with-computer-graphics/
LOCATION:3305 Newell-Simon Hall
CATEGORIES:Seminar,VASC Seminar
ATTACH;FMTTYPE=image/jpeg:https://www.ri.cmu.edu/app/uploads/2026/08/8-31-26.jpg
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260828T120000
DTEND;TZID=America/New_York:20260828T130000
DTSTAMP:20260912T191612
CREATED:20260826T182341Z
LAST-MODIFIED:20260826T182341Z
UID:153300-1787918400-1787922000@www.ri.cmu.edu
SUMMARY:Robot Manipulation Capabilities and Grounded Task Axes
DESCRIPTION:Abstract: \nRobots operating in open-world environments will need to understand the scope of their capabilities\, recognize their limitations\, and determine how those capabilities can be expanded. This talk will review research from the Intelligent Autonomous Manipulation Lab on modeling robot capabilities\, allocating learning resources to expand them efficiently\, and structuring skills for a given scope of tasks. I will also discuss how tactile perception and dexterous hardware can broaden what robots are able to perceive and do. \nThe main focus will be our work on Grounded Task Axes\, a framework for autonomously constructing structured manipulation skills. Rather than representing a skill as a monolithic policy or fixed library element\, Grounded Task Axes compose task-tailored behaviors from controllers defined relative to semantically meaningful object keypoints and axes. Our recent work uses vision and language models to generate these skill structures\, ground them in observed scenes\, and set task-specific parameters. These structured representations support precise controller-level execution\, interpretable human correction\, and generalization across objects and tasks.
URL:https://www.ri.cmu.edu/event/robot-manipulation-capabilities-and-grounded-task-axes/
LOCATION:Newell-Simon Hall 4305
CATEGORIES:Faculty Events
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260810T153000
DTEND;TZID=America/New_York:20260810T163000
DTSTAMP:20260912T191612
CREATED:20260805T190409Z
LAST-MODIFIED:20260805T190409Z
UID:153064-1786375800-1786379400@www.ri.cmu.edu
SUMMARY:Using Sound to Steer Light and Light to Measure Sound
DESCRIPTION:Abstract:   Light and sound are traditionally treated as distinct physical phenomena\, yet their interaction provides a powerful mechanism for manipulating and sensing information across imaging\, communication\, and measurement. This talk explores computational acousto-optic systems that co-design acoustics\, optics\, and signal processing to enable programmable control of light and high-speed optical sensing without mechanical motion. \nFirst\, an acousto-optic structured-light system is presented in which ultrasound generates rapidly varying refractive-index patterns that steer laser beams at megahertz rates. Combined with event-based vision\, this approach projects up to two million structured-light planes per second and enables 3D scanning at up to 1\,000 frames per second. \nNext\, the same acousto-optic principle is applied to underwater optical backscatter communication by dynamically steering retroreflected light toward or away from a receiver using ultrasound. This enables low-power underwater devices to achieve megabit-per-second data transmission without mechanically actuated optics. \nFinally\, the talk investigates the complementary direction of using light to measure sound. By combining a continuous-wave laser\, a high-speed photodetector\, and commodity software-defined radios\, phase-based depth measurements are performed at megahertz rates\, recovering microscopic surface vibrations and acoustic signals from remote objects. \nTogether\, these systems demonstrate that acoustic waves can serve as programmable optical elements while optical measurements provide sensitive probes of acoustic phenomena. By tightly integrating acoustics\, optics\, computational imaging\, and signal processing\, this work establishes a unified framework for building fast\, programmable sensing and communication systems that operate beyond the capabilities of conventional mechanically actuated approaches. \nBio:  Dhawal Sirikonda is a PhD candidate at Dartmouth College working with Prof. Adithya Pediredla in the Rendering and Imaging Science Lab (RISc). He builds next-generation ultra-fast LiDAR sensors and novel imaging systems for high-speed perception and communication applications. His research focuses on designing computational imaging systems by leveraging diverse sensor modalities\, including event cameras\, RGB-D sensors\, and emerging hybrid sensing architectures. His current work explores novel acousto-optic imaging systems for fast scanning and communication applications. He is particularly interested in combining machine learning and physics-based computational methods to develop robust real-world sensing systems. Prior to his PhD\, he completed his Master’s degree working with Prof. P. J. Narayanan at the intersection of 3D Vision and Real-Time Graphics. \nHomepage:   https://dhawal.xyz \nSponsor:\nThe VASC seminar is generously sponsored by HeyGen\, an all-in-one AI-powered video generation platform that leverages advances in computer vision\, generative modeling\, and multimodal learning to make high-quality video creation both scalable and accessible.
URL:https://www.ri.cmu.edu/event/using-sound-to-steer-light-and-light-to-measure-sound/
LOCATION:3305 Newell-Simon Hall
CATEGORIES:Seminar,VASC Seminar
ATTACH;FMTTYPE=image/jpeg:https://www.ri.cmu.edu/app/uploads/2026/08/8-10-26.jpeg
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260804T110000
DTEND;TZID=America/New_York:20260804T120000
DTSTAMP:20260912T191612
CREATED:20260728T154143Z
LAST-MODIFIED:20260728T154143Z
UID:152908-1785841200-1785844800@www.ri.cmu.edu
SUMMARY:Towards Modeling GPS Noise via Raytracing
DESCRIPTION:Abstract:\nAutonomous robot navigation relies heavily on visual-inertial state estimation\, which inherently accumulates drift over extended trajectories. Fusing Global Navigation Satellite System (GNSS) signals mitigates this drift\, but evaluating these systems in hardware is resource-intensive. Therefore\, existing works often rely on simplified\, static noise models to generate GNSS measurements. This thesis presents a raytracing-based GPS simulator integrated with Unity using the Flightmare framework to model realistic signal degradation in complex environments. By computing downward ray intersections from real satellite orbital positions\, the simulator dynamically measures signal attenuation through foliage—using Weissberger’s Modified Exponential Decay model—and building obstructions. Instead of relying on geometric position dilution of precision (DOP)\, the system constructs a measurement noise covariance matrix based on Carrier-to-Noise density ratio loss to yield a Weighted Dilution of Precision (WDOP). Experimental validation in simulated canopy and concrete structure environments\, along with hardware comparisons against a u-blox GNSS receiver on the Carnegie Mellon University campus\, demonstrates that WDOP closely mirrors real-world positioning uncertainty where standard geometric DOP models fail. This work offers a high-fidelity simulation tool for advancing autonomous navigation and sensor fusion in signal-degraded environments. \nThesis Committee:\nWennie Tabib (chair)\nWenshan Wang\nJiaoyang Li\nAlbert Xu
URL:https://www.ri.cmu.edu/event/towards-modeling-gps-noise-via-raytracing/
LOCATION:Newell-Simon Hall 3305
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260803T153000
DTEND;TZID=America/New_York:20260803T170000
DTSTAMP:20260912T191612
CREATED:20260727T145526Z
LAST-MODIFIED:20260727T145526Z
UID:152896-1785771000-1785776400@www.ri.cmu.edu
SUMMARY:Simulate to Learn\, Learn to Simulate for Dexterous Robot Control
DESCRIPTION:Abstract:\nSimulation enables robots to learn and evaluate behaviors at scale before real-world deployment. Yet the mismatch between simulation and the physical world remains a fundamental obstacle. This is particularly challenging for dexterous manipulation\, where contact-rich interactions and dynamics variations across objects and robot embodiments are difficult to model. In my thesis research\, I explore how robot learning can scale through simulation and how learned models can make simulation more accurate to the physical world\, through two complementary directions. \nPart I: Differentiable simulation for scalable robot learning. \nFirst\, I present a GPU-parallel differentiable multiphysics simulation and a first-order reinforcement learning algorithm that pairs simulation gradients with entropy regularization\, for smoother policy optimization on locomotion and manipulation tasks. Next\, I introduce hybrid analytic differentiability\, combining implicit differentiation\, auto-differentiation\, and custom analytic Jacobians to compute gradients through contact without modifying forward dynamics. With it\, I develop a production-ready differentiable simulation and show how its gradients support initial value problems\, trajectory optimization\, and system identification. \nPart II: Aligning simulation with the real world across diverse embodiments and tasks. \nFirst\, I introduce an algorithm for iterative real-to-sim alignment. Alongside\, I present a hybrid neural dynamics model that combines learned dynamics correction with analytical inverse dynamics while retaining the simulator’s contact resolution\, to produce physically consistent simulation trajectories. Next\, I build flexible real-time robot I/O infrastructure for synchronized data collection and policy deployment across different robots\, sensors\, and interfaces. \nIn my proposed work\, I will explore how differentiable simulation and differentiable rendering can support real-to-sim reconstruction of simulation environments from multimodal real-world data. In my final project\, I will study how neural dynamics can scale real-to-sim-to-real learning to dexterous hands and humanoid robots requiring high-dimensional continuous control. \n\n\nTogether\, these directions aim to establish a feedback loop in which robot policies and simulators continually improve one another. By turning physical experience into better simulators and using better simulators to train more capable robots\, this loop could scale robot learning across tasks and embodiments in ways that neither simulation nor real-world data can achieve alone.\n\n\nThesis Committee:\nJean Oh (co-chair)\nGuanya Shi (co-chair)\nJeff Ichnowski\nMiles Macklin (NVIDIA)\n\nThesis Link
URL:https://www.ri.cmu.edu/event/simulate-to-learn-learn-to-simulate-for-dexterous-robot-control/
LOCATION:Gates Hillman Center 4405
CATEGORIES:PhD Thesis Proposal,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260730T153000
DTEND;TZID=America/New_York:20260730T163000
DTSTAMP:20260912T191612
CREATED:20260729T174959Z
LAST-MODIFIED:20260729T174959Z
UID:152937-1785425400-1785429000@www.ri.cmu.edu
SUMMARY:Marble: An On-Manifold Approach to Solving Mathematical Programs with Complementarity Constraints
DESCRIPTION:Abstract:\nMany problems in robotics require reasoning over a mix of continuous dynamics and discrete events\, such as making and breaking contact in manipulation and locomotion. These problems are locally well modeled by quadratic programs with complementarity constraints (QPCCs). While very expressive\, QPCCs are non-convex problems\, and few solvers exist for computing fast\, local solutions for use in planning pipelines. In this work\, we develop an open-source solver\, Marble\, designed to solve QPCCs using an on-manifold complementarity relaxation and homotopy technique. The resulting solver avoids many of the classical issues with complementarity constraints and exhibits competitive speed and robustness across QPCC benchmarks and robotics-specific examples compared to existing baselines. \nCommittee:\nZac Manchester (advisor)\nRajan Gill\nArun Bishop
URL:https://www.ri.cmu.edu/event/marble-an-on-manifold-approach-to-solving-mathematical-programs-with-complementarity-constraints/
LOCATION:GHC 6115
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260730T153000
DTEND;TZID=America/New_York:20260730T163000
DTSTAMP:20260912T191612
CREATED:20260713T154004Z
LAST-MODIFIED:20260713T183008Z
UID:151761-1785425400-1785429000@www.ri.cmu.edu
SUMMARY:[MS Thesis Talk] Marble: An On-Manifold Approach to Solving Mathematical Programs with Complementarity Constraints
DESCRIPTION:Date: Thursday\, July 30\, 2026\nTime: 3:30 PM – 4:30 PM\nLocation / ZOOM Link: (GHC 6115 / https://cmu.zoom.us/j/96096959582 ) \nAbstract:\nMany problems in robotics require reasoning over a mix of continuous dynamics and discrete events\, such as making and breaking contact in manipulation and locomotion. These problems are locally well modeled by quadratic programs with complementarity constraints (QPCCs). While very expressive\, QPCCs are non-convex problems\, and few solvers exist for computing fast\, local solutions for use in planning pipelines. In this work\, we develop an open-source solver\, Marble\, designed to solve QPCCs using an on-manifold complementarity relaxation and homotopy technique. The resulting solver avoids many of the classical issues with complementarity constraints and exhibits competitive speed and robustness across QPCC benchmarks and robotics-specific examples compared to existing baselines. \nCommittee:\nZac Manchester (advisor)\nRajan Gill\nArun Bishop \n—\nMICAH REICH \nMSR Student\, Robotics Institute – CMU
URL:https://www.ri.cmu.edu/event/ms-thesis-talk-marble-an-on-manifold-approach-to-solving-mathematical-programs-with-complementarity-constraints/
LOCATION:Gates Hillman Center 6115
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260730T110000
DTEND;TZID=America/New_York:20260730T120000
DTSTAMP:20260912T191612
CREATED:20260727T133112Z
LAST-MODIFIED:20260727T133112Z
UID:152894-1785409200-1785412800@www.ri.cmu.edu
SUMMARY:Risk-Aware Multi-Agent Navigation in Dynamic Smoke Environments
DESCRIPTION:Abstract:\nIn wildfire scenarios\, deploying autonomous drones requires safely anticipating the dynamic behavior of dense smoke to coordinate effectively. Unlike traditional rigid obstacles\, smoke represents a fast-moving\, complex fluid hazard that impairs visual navigation and onboard sensors. In this thesis\, we propose a novel risk-aware\, multi-agent path planning framework that treats dynamic smoke as a continuous physical hazard. By leveraging Probabilistic Fourier Neural Operators (PFNO)\, our architecture forecasts the spatiotemporal behavior of smoke density while simultaneously quantifying its inherent aleatoric uncertainty. These probabilistic predictions are mapped into a safety cost using a Conditional Value-at-Risk (CVaR) metric\, which directly informs a safe variation of a time-varying Model Predictive Path Integral (MPPI) controller to ensure collision-free coordination. We integrate inter-agent safety bounds directly within the sampling-based planner of each drone and shield its output using Control Barrier Functions (CBFs). Furthermore\, because individual agents might possess a limited field of view\, we address the critical challenge of partial observability. We explore a potential initial solution for an active search policy driven by epistemic uncertainty mapping\, demonstrating how the agents collaboratively construct partial global maps to navigate safely. Extensive results demonstrate that this unified multi-agent architecture successfully coordinates the agents\, conservatively overbounding active smoke fronts and minimizing both cumulative and high-density smoke exposure compared to standard reactive baselines. \nCommittee:\nProf. Katia Sycara (Advisor)\nProf. John Dolan\nAndrew Jong
URL:https://www.ri.cmu.edu/event/risk-aware-multi-agent-navigation-in-dynamic-smoke-environments/
LOCATION:Gates 6115
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260730T103000
DTEND;TZID=America/New_York:20260730T113000
DTSTAMP:20260912T191612
CREATED:20260724T004217Z
LAST-MODIFIED:20260724T004217Z
UID:152858-1785407400-1785411000@www.ri.cmu.edu
SUMMARY:Passive Ranging via Differential Defocus
DESCRIPTION:Abstract:   Depth sensors are essential wherever robots\, vehicles\, and wearables must understand 3D scenes. As these applications spread to ever smaller platforms\, like drones\, underwater vehicles\, mobile and wearable devices\, building depth sensors that are more compact and more power-efficient has become a priority in both academia and industry. Today\, two approaches dominate: active systems such as LiDAR\, which spend power projecting light onto the scene\, and passive triangulation methods such as stereo\, which require a wide separation between viewpoints and substantial computation. Neither shrinks gracefully. This talk presents an alternative: differential defocus\, a passive sensing principle that recovers distance from how defocus blur changes between slightly different images\, turning depth estimation into a simple\, closed-form calculation at each pixel. The talk examines this idea through three systems: snapshot depth sensing\, low-computation depth sensing\, and accurate triangulation-based depth sensing. Each project addresses a different requirement of practical depth sensing. \n \nFor snapshot depth sensing\, Focal Split (CVPR 2025)\, a handheld depth-from-differential-defocus camera with fully onboard power and computing\, captures two differently focused images in a single exposure and converts them into depth maps in real time on a Raspberry Pi at under 5 W. For low-computation depth sensing\, depth from coupled optical differentiation (IJCV) provides the underlying theory: each pixel’s distance can be read off from how the image responds to a small\, coupled adjustment of the camera’s optics\, through a closed-form formula that costs only a few hundred operations per pixel\, is markedly more robust to noise than earlier defocus-based methods\, and holds regardless of the aperture design. For accurate triangulation-based depth sensing\, D³S\, the most recent work in this line\, computes distance independently from two physical cues\, defocus and stereo\, and keeps only the estimates on which both agree. This consensus yields depth maps with 1-cm mean error over a 0.3–1.64 m range from a prototype whose two lenses sit just 3.84 mm apart\, a working range that previously required camera separations roughly ten times larger. Together\, these results point to passive\, physics-grounded depth sensing as a practical building block for platforms too small or too power-constrained for today’s depth cameras.\n \nBio:  Junjie Luo is a graduating Ph.D. on the job market in the Elmore Family School of Electrical and Computer Engineering at Purdue University\, advised by Prof. Qi Guo. He develops passive depth sensors that combine unconventional optics with closed-form\, physics-based algorithms to recover accurate 3D information with compact\, low-power cameras. His first-authored research includes Focal Split\, a handheld snapshot depth camera presented at CVPR 2025; depth from coupled optical differentiation\, published in the International Journal of Computer Vision; and a compact metasurface system for simultaneous near-field ranging and far-field imaging\, developed under a Navy STTR project and presented at SPIE Defense + Security 2026. He also works closely with Professor Emma Alexander on computational imaging research. Before joining Purdue\, he earned a BS in Software Engineering from Sun Yat-sen University. More broadly\, he is interested in the co-design of optics and algorithms for reliable 3D imaging with compact hardware.     \n \nHomepage:   https://luo-jun-jie.github.io/\n \nSponsor:\nThe VASC seminar is generously sponsored by HeyGen\, an all-in-one AI-powered video generation platform that leverages advances in computer vision\, generative modeling\, and multimodal learning to make high-quality video creation both scalable and accessible.
URL:https://www.ri.cmu.edu/event/passive-ranging-via-differential-defocus/
LOCATION:3305 Newell-Simon Hall
ATTACH;FMTTYPE=image/jpeg:https://www.ri.cmu.edu/app/uploads/2026/07/7-30-26-scaled.jpeg
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260730T090000
DTEND;TZID=America/New_York:20260730T100000
DTSTAMP:20260912T191612
CREATED:20260723T152807Z
LAST-MODIFIED:20260723T152807Z
UID:152778-1785402000-1785405600@www.ri.cmu.edu
SUMMARY:Reference-Prompted Instance Segmentation for Growing Retail Catalogs
DESCRIPTION:Abstract:\n \nRetail and warehouse perception systems work against a catalog that never stops changing: a segmenter deployed today will eventually be asked to find products that did not exist when it was trained. A closed-set segmenter can only grow its vocabulary by widening its classification head and fine-tuning it\, which risks the classes it already handled\, or by retraining on the whole accumulated catalog\, which costs more with every product added. This thesis studies reference-prompted segmentation\, which specifies the target product with a canonical image rather than a class label or a text description\, so that a product’s identity enters the model as an input rather than as a slot in its output. We fine-tune a reference-conditioned segmenter one new product at a time — twenty-seven sequential steps growing a three-product catalog to thirty — using Learning without Forgetting\, and compare it against a closed-set segmenter given the identical recipe\, budget\, and data. Our method retains its earlier products substantially better than the closed-set baseline. Every dataset in this thesis comes from isaac_datagen\, a standalone system built on Isaac Sim as a synthetic-data engine for retail scenes. Under this protocol and at this scale\, reference prompting lets a perception system keep pace with a growing catalog without ever retraining on all of it.\n\nCommittee:\nJeff Ichnowski (advisor)\nShubham Tulsiani\nBardienus Duisterhof
URL:https://www.ri.cmu.edu/event/reference-prompted-instance-segmentation-for-growing-retail-catalogs/
LOCATION:NSH 3305
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260729T160000
DTEND;TZID=America/New_York:20260729T170000
DTSTAMP:20260912T191612
CREATED:20260724T135742Z
LAST-MODIFIED:20260724T135742Z
UID:152867-1785340800-1785344400@www.ri.cmu.edu
SUMMARY:Exploiting Structure for Real-Time Robot Motion Planning and Control
DESCRIPTION:Abstract: \nReal-time robot motion in dynamic and uncertain environments requires algorithms that can make effective decisions under strict computational constraints. High-fidelity dynamics\, large search spaces\, uncertainty over future outcomes\, and the pursuit of optimal motion each impose substantial computational costs. These challenges become especially pronounced in partially observable and rapidly changing environments\, where a robot must continually update its predictions and react before a previously computed plan becomes obsolete. This thesis presents two model-based frameworks that exploit structure to bridge deliberative planning and reactive control\, enabling efficient closed-loop execution without solving a full planning problem at every step. \nFirst\, we study projectile interception\, where a robot must begin moving after only a few noisy measurements of a fast-moving object’s trajectory. Our framework couples uncertainty estimates from an RGB-D tracking system with a sparse kinodynamic graph of executable motion primitives. As the projectile estimate evolves\, the system rapidly reevaluates candidate motions and executes partial trajectories that remain robust across a distribution of possible future outcomes. This approach maintains millisecond-scale replanning and improves interception performance. \nThe second framework learns the parameters of convex optimization-based safety filters formulated as control barrier function quadratic programs (CBF-QPs). Imitation learning transfers goal-directed behavior from a computationally expensive global planner into a local reactive controller while retaining explicit model-based safety constraints. This combination shifts computation from online global search to offline learning\, enabling efficient closed-loop execution with reduced planning effort. We show that\, under matched online computational budgets\, the learned safety filter improves performance in planar navigation environments with moving obstacles and in constrained manipulator-planning tasks. \nTogether\, these contributions show how structured representations can preserve reasoning about dynamics\, uncertainty\, and safety while enabling efficient real-time robot motion. \n\nCommittee:\nMaxim Likhachev (co-advisor)\nHowie Choset (co-advisor)\nAndrea Bajscy\nItamar Mishani
URL:https://www.ri.cmu.edu/event/exploiting-structure-for-real-time-robot-motion-planning-and-control/
LOCATION:GHC 9115
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260729T143000
DTEND;TZID=America/New_York:20260729T153000
DTSTAMP:20260912T191612
CREATED:20260727T132903Z
LAST-MODIFIED:20260727T132903Z
UID:152892-1785335400-1785339000@www.ri.cmu.edu
SUMMARY:The Limits of Prompt-Based Self-Improvement in Language Model Agents: Meta-Optimization and Failure Attribution
DESCRIPTION:Abstract: \nModern language models’ in-context learning and instruction-following abilities make it possible to adapt agent behavior without updating model weights. This thesis studies online self-improvement\, where an agent encounters each task only once and must accumulate reusable experience across a sequence of tasks\, reflecting real-world deployments in which actions have persistent consequences and retries may be impossible. \nFirst\, we ask whether meta-optimization can be integrated with online self-improvement by allowing the prompt optimizer itself to learn across episodes. Building on Agentic Context Engineering (ACE)\, we evaluate three variants that equip its optimizer with persistent memory. On the AppWorld benchmark\, none reliably improves over ACE when using strong base language models. Analysis suggests that ACE-style adaptation is near saturation in this regime\, leaving little room for meta-optimization to provide further gains. \nTo probe the limits of ACE-style adaptation\, we evaluate it on Toolathlon\, a harder and more heterogeneous tool-use benchmark\, and observe no improvement over non-adapting agents. Auditing these apparent failures reveals ambiguous task specifications\, missing information\, brittle evaluators\, and unstable external services\, rather than cleanly attributable agent errors. Because agent trajectories can span hundreds of thousands of tokens\, we develop an automated\, evidence-grounded pipeline for analyzing trajectories and attributing failures at scale. Together\, these results show that effective online self-improvement depends not only on prompt optimization\, but also on transferable structure across tasks and trustworthy failure attribution. \n\n\nCommittee:\nAran Nayebi (advisor)\nDaniel Fried\nHaochen Zhang
URL:https://www.ri.cmu.edu/event/the-limits-of-prompt-based-self-improvement-in-language-model-agents-meta-optimization-and-failure-attribution/
LOCATION:NSH 3305
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260729T120000
DTEND;TZID=America/New_York:20260729T130000
DTSTAMP:20260912T191612
CREATED:20260722T134829Z
LAST-MODIFIED:20260722T134829Z
UID:152771-1785326400-1785330000@www.ri.cmu.edu
SUMMARY:Efficient cross-embodiment transfer via world models and policy steering
DESCRIPTION:Abstract:\nLearning robot manipulation policies typically requires large amounts of high-quality demonstration data\, which is expensive to collect on real robots. While large robot and human datasets exist\, embodiment gaps make transferring knowledge between platforms challenging. This thesis investigates cross-embodiment transfer—how robots can learn from experience collected on different robots and humans to reduce the need for new demonstrations.\n\nI present Latent Policy Steering (LPS)\, a framework that pretrains a world model across diverse embodiments using embodiment-agnostic visual dynamics represented by optical flow. The pretrained model can be adapted to an unseen robot using less than an hour of teleoperation data. At deployment\, the world model enables the policy to anticipate the consequences of its actions\, identify likely mistakes\, and steer itself back toward safer behaviors through latent-space search. Experiments on both real-world manipulation tasks and simulation benchmarks demonstrate that this approach consistently improves policy performance and outperforms methods relying on embodiment-specific representations.\nIn summary\, LPS demonstrates an effective paradigm for efficient cross-embodiment transfer\, leveraging diverse\, cost-effective robot and human data to adapt to an unseen robot with minimal demonstrations\, reducing data collection costs and enabling more scalable robot learning. \nCommittee:\nJeff Schneider (advisor)\nChristopher Atkeson\nTejus Gupta
URL:https://www.ri.cmu.edu/event/efficient-cross-embodiment-transfer-via-world-models-and-policy-steering/
LOCATION:NSH 3305
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260728T154500
DTEND;TZID=America/New_York:20260728T164500
DTSTAMP:20260912T191612
CREATED:20260723T192143Z
LAST-MODIFIED:20260723T192143Z
UID:152840-1785253500-1785257100@www.ri.cmu.edu
SUMMARY:The Tail and the Judge: Better Policy Gradients and World Models for Reinforcement Learning
DESCRIPTION:Abstract: \nA quiet shift has taken place in how reinforcement learning delivers results.\nIncreasingly\, performance comes not from a single attempt but from selection\nover many: a language model is sampled repeatedly and judged by its best\nverified answer\, and a planning agent imagines many action sequences and\nexecutes the one its learned world model scores highest. Selection can fail\nin exactly two ways. The candidate pool may contain nothing worth choosing\,\nor the judge that scores the candidates may be wrong precisely where it\nmatters. This thesis argues that standard training objectives invite both\nfailures\, and it redesigns them to prevent each. \nThe first part trains the policy for selection itself. Standard methods\nmaximize the average reward of a single attempt\, which spreads learning\neffort evenly across quality levels and neglects the rare\, excellent outcomes\nthat selection exists to find. Extending maximum-likelihood reinforcement\nlearning from binary to continuous rewards\, we introduce a tail-likelihood\nobjective that credits the policy for covering every level of quality and\nconcentrates its gradient on the levels it reaches only rarely. We prove that\nthis objective aligns training with best-of-many deployment at every sampling\nbudget simultaneously\, and that it is optimized by an unbiased\,\nhyperparameter-free estimator amounting to a one-line change in standard\npipelines. Empirically\, the method recovers the supervised gold-standard\ngradient where popular baselines remain permanently misaligned; it lifts\nmaze policies from a near-zero starting success rate to solving most mazes\noptimally; and it is evaluated further on object localization\,\nprogram-speed optimization\, and vision-language grounding. \nThe second part secures the judge. A planner that maximizes predicted return\nwill seek out exactly the regions where its world model is most optimistically\nwrong. We prove that the resulting loss in policy quality is governed by the\nsharpness of the model’s loss landscape\, and we show that sharpness-aware\ntraining of the world model alone\, leaving the planner and policy untouched\,\ndelivers large and consistent gains that transfer across high-dimensional\ncontinuous control and pixel-based discrete control\, and across two different\nmodel-based algorithms. \nGood selection needs something worth choosing and a judge worth trusting.\nProviding both\, and composing them\, is the program of this thesis. \nCommittee:\nProf. Jeff Schneider (advisor)\nProf. Andrea Zanette (advisor)\nProf. Ruslan Salakhutdinov\nWentse Chen
URL:https://www.ri.cmu.edu/event/the-tail-and-the-judge-better-policy-gradients-and-world-models-for-reinforcement-learning/
LOCATION:NSH 3305
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260728T150000
DTEND;TZID=America/New_York:20260728T160000
DTSTAMP:20260912T191612
CREATED:20260721T141849Z
LAST-MODIFIED:20260721T141849Z
UID:152655-1785250800-1785254400@www.ri.cmu.edu
SUMMARY:Robotic Localization and Mapping of Disease in Apple Orchards
DESCRIPTION:Abstract:\nUnited States apple growers lose more than $100 million a year to fire blight\, a bacterial disease caused by Erwinia amylovora that is detrimental to pome fruit trees such as apple and pear.This bacterium infects blossoms\, shoots\, and branches during the bloom season causing tissue to die. Therefore\, the detection and removal of infected tissues over the dormant season is critical to prevent an outbreak in the following spring. However\, the symptoms of fire blight are subtle and difficult to detect\, and finding them still depends almost entirely on manual scouting\, a process that does not scale to large orchards and is prohibitively expensive and time-consuming for growers to perform at the scale required for effective disease management. With the decreasing availability of labor in the agricultural sector\, there is a pressing need for automated solutions that can perform this inspection at scale\, and with high accuracy. \nThis thesis aims to develop a robotic system capable of performing that inspection autonomously. Since the visual cues that distinguish infected tissue are subtle\, easily occluded\, vary with natural lighting\, and are reliably resolved only at close range\, the system relies on active perception: a manipulator positions a camera to acquire discriminative\, task-relevant views of the canopy. We have collected a multi-modal dataset of dormant apple trees\, the first to pair dense near-infrared imagery with flash-illuminated stereo RGB for this task\, and trained detectors to recognize disease symptoms across both modalities. We then introduce a confidence-aware semantic mapping method that fuses these per-view detections into a persistent 3D representation of disease confidence across the canopy\, and a next-best-view planner that actively selects viewpoints to refine the map’s least confident\, most contested predictions. Finally\, we integrated the full pipeline onto Erwin\, a mobile active perception robot built using an Amiga base and an xArm6 arm for manipulation of the camera rig. \n\nWe successfully validated the system both in simulation and in the field on 12 trees at the Penn State Fruit Research and Extension Center under a genuine train–test domain gap\, the semantic planner more than doubles the detection accuracy of a complete planar scan by the halfway point of the inspection\, concentrating its views on the map’s most contested disease evidence. This result demonstrates the central promise of active perception for orchard disease mapping: by deciding where to look next a robot can build maps of orchard disease that are accurate enough to support autonomous disease management\, while minimizing the time and energy spent on inspection. \nThesis Committee:\nProf. Abhisesh Silwal (chair)\nProf. Oliver Kroemer\nItamar Mishani
URL:https://www.ri.cmu.edu/event/robotic-localization-and-mapping-of-disease-in-apple-orchards/
LOCATION:NSH 1109
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260728T133000
DTEND;TZID=America/New_York:20260728T143000
DTSTAMP:20260912T191612
CREATED:20260721T132229Z
LAST-MODIFIED:20260721T132303Z
UID:152653-1785245400-1785249000@www.ri.cmu.edu
SUMMARY:Terrain-Aware Dynamics Models for High-Speed Off-Road Navigation
DESCRIPTION:Abstract:\nHigh-speed autonomy in the real world requires accurate control\, which often relies on dynamics models that capture the complex interaction between a robot and its environment. In off-road regimes\, this terrain interaction dominates the robot’s dynamics\, driven by formidable characteristics such as diverse surface properties\, complex geometries\, environment diversity\, and high-speed instability. This thesis investigates how terrain-aware perception can improve learned dynamics modeling and control at high speed and how such models can be rigorously evaluated before deployment. \nFirst\, we demonstrate how perception representations can make dynamics models terrain-aware\, capturing geometric and semantic details that physics-based models miss and that simplistic learned models overlook. We formulate a representation that extracts the most relevant terrain features given a robot’s motion\, yielding higher prediction accuracy. Second\, we introduce a rigorous evaluation method to mitigate real-world failures. We collect a challenging\, multi-season dataset at speeds up to 13 m/s and mine the most difficult evaluation samples using our benchmarking method. While models appear accurate on average\, our benchmark surfaces the long-tail failure cases where prior models fail catastrophically. \nTogether\, with our verified\, terrain-aware model\, we decrease the worst-case prediction error by 23.8%\, compared to physics-based and learned baselines. We further evaluate on a full-scale ATV platform across high-speed (>10m/s) and geometrically challenging courses with a 34.9% reduction in maximum cross-track error. These results demonstrate the importance of embedding environment context for locomotion-related tasks. \nThesis Committee:\nWenshan Wang (co-chair)\nSebastian Scherer (co-chair)\nAaron Johnson\nAnoushka Alavilli
URL:https://www.ri.cmu.edu/event/terrain-aware-dynamics-models-for-high-speed-off-road-navigation/
LOCATION:GHC 6115
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260728T130000
DTEND;TZID=America/New_York:20260728T140000
DTSTAMP:20260912T191612
CREATED:20260721T191042Z
LAST-MODIFIED:20260721T191042Z
UID:152765-1785243600-1785247200@www.ri.cmu.edu
SUMMARY:Robust Visuomotor Policy Learning in Uncertain World Models
DESCRIPTION:Abstract:\nWorld models have shown promise in robotics by imagining future outcomes of robot actions directly from high-dimensional sensor observations. However\, learning visuomotor policies with world models still faces significant reliability challenges\, as imagined futures from world models often diverge from the outcomes that may actually occur. This unreliability arises from uncertainty in learned world models\, including epistemic uncertainty\, where the model lacks sufficient knowledge in out-of-distribution regions\, and aleatoric uncertainty\, where intrinsic randomness in the system allows multiple plausible outcomes to arise under the same robot action. This thesis develops methods for learning robust visuomotor policies using uncertain world models by explicitly reasoning about and mitigating these uncertainties. \nWe first introduce UNISafe\, an uncertainty-aware latent safety filter for mitigating epistemic uncertainty in learned world models. We propose a principled framework for detecting out-of-distribution world model predictions by quantifying epistemic uncertainty and calibrating an uncertainty threshold with conformal prediction. Moreover\, this out-of-distribution detection is incorporated into Hamilton-Jacobi reachability analysis\, synthesizing the safety filter to proactively avoid regions where world model predictions are unreliable and thereby achieve robust\, safe visuomotor control. We then introduce StressDream\, an inference-time steering method for mitigating aleatoric uncertainty in world models. Instead of relying on nominal samples from the world model\, StressDream actively steers the imaginations of video world models to expose plausible but critical outcomes of robot actions. This enables more robust policy evaluation by uncovering failure modes of robot actions\, as well as improved policy optimization by training policies against challenging but realistic imagined futures. Together\, these methods enable visuomotor policies relying on learned but uncertain world models to achieve robust control in complex\, uncertain environments with high-dimensional sensor observations by explicitly reasoning about the uncertainties of the learned world model. \nThesis Committee:\nAndrea Bajcsy (chair)\nMax Simchowitz\nJeff Schneider\nMichelle Zhao
URL:https://www.ri.cmu.edu/event/robust-visuomotor-policy-learning-in-uncertain-world-models-2/
LOCATION:NSH 3002
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260728T123000
DTEND;TZID=America/New_York:20260728T140000
DTSTAMP:20260912T191612
CREATED:20260721T190755Z
LAST-MODIFIED:20260721T190913Z
UID:152762-1785241800-1785247200@www.ri.cmu.edu
SUMMARY:Robust Visuomotor Policy Learning in Uncertain World Models
DESCRIPTION:Abstract:\nWorld models have shown promise in robotics by imagining future outcomes of robot actions directly from high-dimensional sensor observations. However\, learning visuomotor policies with world models still faces significant reliability challenges\, as imagined futures from world models often diverge from the outcomes that may actually occur. This unreliability arises from uncertainty in learned world models\, including epistemic uncertainty\, where the model lacks sufficient knowledge in out-of-distribution regions\, and aleatoric uncertainty\, where intrinsic randomness in the system allows multiple plausible outcomes to arise under the same robot action. This thesis develops methods for learning robust visuomotor policies using uncertain world models by explicitly reasoning about and mitigating these uncertainties. \nWe first introduce UNISafe\, an uncertainty-aware latent safety filter for mitigating epistemic uncertainty in learned world models. We propose a principled framework for detecting out-of-distribution world model predictions by quantifying epistemic uncertainty and calibrating an uncertainty threshold with conformal prediction. Moreover\, this out-of-distribution detection is incorporated into Hamilton-Jacobi reachability analysis\, synthesizing the safety filter to proactively avoid regions where world model predictions are unreliable and thereby achieve robust\, safe visuomotor control. We then introduce StressDream\, an inference-time steering method for mitigating aleatoric uncertainty in world models. Instead of relying on nominal samples from the world model\, StressDream actively steers the imaginations of video world models to expose plausible but critical outcomes of robot actions. This enables more robust policy evaluation by uncovering failure modes of robot actions\, as well as improved policy optimization by training policies against challenging but realistic imagined futures. Together\, these methods enable visuomotor policies relying on learned but uncertain world models to achieve robust control in complex\, uncertain environments with high-dimensional sensor observations by explicitly reasoning about the uncertainties of the learned world model. \nThesis Committee:\nAndrea Bajcsy (chair)\nMax Simchowitz\nJeff Schneider\nMichelle Zhao
URL:https://www.ri.cmu.edu/event/robust-visuomotor-policy-learning-in-uncertain-world-models/
LOCATION:NSH 3002
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260728T110000
DTEND;TZID=America/New_York:20260728T120000
DTSTAMP:20260912T191612
CREATED:20260722T134629Z
LAST-MODIFIED:20260722T134629Z
UID:152769-1785236400-1785240000@www.ri.cmu.edu
SUMMARY:What needs to be learned in robot learning? A case study: learning battery insertion from a diagram
DESCRIPTION:Abstract:\nManual diagrams are a rich and common knowledge source humans use to learn new skills\, but their use for robot learning is still underexplored. A challenge with instruction diagrams is that they communicate task progression in a sparse\, qualitative visual format\, relying on the learner’s prior physical understanding to fill in unmentioned execution details. This thesis investigates the fundamental question of \emph{what needs to be learned in robot learning} by exploring the interplay between explicit information extracted from instructions and implicit physical knowledge discovered through practice or prior knowledge\, using the cylindrical battery insertion task as a case study. \nFirst\, we present a pipeline that compiles static 2D instruction diagrams into metrically accurate 3D simulation environments. We use Vision-Language Models (VLMs) to extract qualitative scene topology and contact modes\, followed by a geometric optimization solver that certifies and refines metric object dimensions and spatial subgoals. Second\, using the reconstructed task keyframes\, we manually designed a control strategy on physical hardware. This hardware deployment exposes the limitations of purely explicit instructions and reveals crucial implicit details such as “tricks” – open-loop primitives exploiting the task’s physical properties that yield robustness gains even over naive closed-loop methods. Finally\, we investigate whether these implicit physical behaviors can be discovered autonomously by reinforcement and imitation learning. \nIn summary\, this thesis demonstrates a paradigm for bridging human data such as is found on YouTube\, as well as instructional text\, diagrams\, and explicit demonstration videos\, showing a promising approach to make use of the vast knowledge base of humans. \nCommittee:\nChristopher Atkeson (advisor)\nShubham Tulsiani\nYuemin Mao
URL:https://www.ri.cmu.edu/event/what-needs-to-be-learned-in-robot-learning-a-case-study-learning-battery-insertion-from-a-diagram/
LOCATION:GHC 8115
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260727T130000
DTEND;TZID=America/New_York:20260727T140000
DTSTAMP:20260912T191612
CREATED:20260721T190859Z
LAST-MODIFIED:20260721T190859Z
UID:152761-1785157200-1785160800@www.ri.cmu.edu
SUMMARY:From Following to Leading: Adaptive Collaboration and Influence for Multi-Agent Teaming
DESCRIPTION:Abstract: \nAutonomous agents and robots are taking on increasingly collaborative roles alongside people\, from self-driving vehicles that navigate roads alongside human drivers\, to language-model agents that understand user intentions and execute tasks independently. In each of these settings\, success is determined not only by an agent’s individual task competence\, but by its ability to work effectively with others whose preferences and strategies directly shape the outcome of the task. This talk explores collaborative intelligence\, focusing on the requisite components to move from static\, purely reactive agents to proactive collaborators that reason\, adapt to\, and shape the behaviors of their teammates. \nWe introduce TALENTS\, an ad hoc teamwork algorithm that analyzes teammate behavior and dynamically adapts its own policy to best suit them. To accomplish this\, we first learn a latent strategy space from offline trajectory data via a variational autoencoder\, cluster this space into discrete teammate types\, and use a regret-minimization algorithm to infer and track which strategy a partner is following\, allowing the cooperator to adapt online as the partner’s behavior changes over the course of an episode. In both agent-agent evaluations and a 119 participant human-agent study in a modified version of the Overcooked-ai benchmark\, we demonstrate that TALENTS outperforms existing baselines in both quantitative task reward as well as subjective measures of team fluency and trust. \nFinally\, we extend beyond adaptation to examine proactive collaboration through the lens of multi-agent influence. Rather than treating a partner’s strategy as fixed and simply best-responding to it\, we investigate how an agent equipped with knowledge of how its teammate will respond to its actions can deliberately shape that learning process\, motivating partners to shift toward more effective joint conventions. Together\, these contributions establish several important algorithmic foundations needed to build autonomous agents that not only intelligently adapt to humans and other artificial teammates\, but also actively help shape more effective collaboration. \n\nCommittee:\nKatia Sycara (advisor)\nJiaoyang Li\nRenos Zabounidis
URL:https://www.ri.cmu.edu/event/from-following-to-leading-adaptive-collaboration-and-influence-for-multi-agent-teaming/
LOCATION:GHC 8115
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260727T100000
DTEND;TZID=America/New_York:20260727T110000
DTSTAMP:20260912T191612
CREATED:20260722T172447Z
LAST-MODIFIED:20260722T172447Z
UID:152776-1785146400-1785150000@www.ri.cmu.edu
SUMMARY:Hierarchical Manipulation Policies: Adapting to Unseen Objects and Discovering Sub-goals
DESCRIPTION:Abstract:\nA robot that manipulates one object well may still fail on the next. Generalizing across diverse objects and tasks is hard because such objects vary widely in geometry\, articulation\, and interaction dynamics. Hierarchical policies offer a powerful approach: a high-level policy predicts sub-goal end-effector poses\, and a low-level policy generates the actions to reach them. However\, dominant approaches leave the high-level policy brittle to unseen objects and dependent on deterministic sub-goal heuristics that many tasks cannot provide. This thesis asks: How should sub-goals be defined\, represented\, and communicated from the high level to the low level policy so that a single hierarchical policy generalizes across diverse objects and tasks? We address this question through two complementary projects\, both building on a prior hierarchical policy that grounds 3D sub-goal prediction in the observed scene.  \nWe first present demo-conditioned learning for adapting to out-of-distribution objects. Rather than fine-tuning\, the policy is conditioned on a single demonstration provided at test time. We show that reasoning about the demonstration and the current observation jointly in 3D outperforms compressing the demonstration into a latent embedding\, and that a single human hand demonstration can replace a teleoperated robot trajectory\, improving real-world performance on challenging unseen objects. \n  \nWe then present an uncertainty-aware hierarchical framework for tasks where sub-goals cannot be deterministically defined. Common heuristics\, such as gripper open/close transitions or near-zero end-effector velocity\, provide no signal for non-prehensile pushing\, sliding\, or manipulating levers and handles without a discrete grasp event. The framework derives candidate sub-goals through probabilistic changepoint segmentation\, represents the high-level goal distribution as a mixture model over candidate sub-goals\, and conditions the low-level policy on this distribution through goal-aware attention. \n  \nFinally\, this thesis extends hierarchical manipulation policies to challenging settings: adapting to unseen objects\, and modeling sub-goal uncertainty in trajectories that cannot be deterministically segmented. \n\n\nCommittee:\nDavid Held (advisor)\nZackory Erickson (advisor)\nShubham Tulsiani\nJason Liu
URL:https://www.ri.cmu.edu/event/hierarchical-manipulation-policies-adapting-to-unseen-objects-and-discovering-sub-goals/
LOCATION:GHC 8115
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260727T093000
DTEND;TZID=America/New_York:20260727T103000
DTSTAMP:20260912T191612
CREATED:20260721T131907Z
LAST-MODIFIED:20260721T131907Z
UID:152651-1785144600-1785148200@www.ri.cmu.edu
SUMMARY:Understanding Image Intrinsics Through Light and Heat
DESCRIPTION:Abstract:\nThis talk examines the physical interplay between light and heat: what is absent in the visible image—light not reflected by the scene—is absorbed as heat and manifests in the thermal spectrum. This process intrinsically couples illumination\, material properties\, and heat transport. By exploiting this coupling\, a single visible–thermal image pair provides complementary information that enables the decomposition of photometric image intrinsics\, namely incident illumination (shading) and surface reflectance (albedo). More broadly\, modeling the flow of energy across light and heat transport opens new opportunities for vision\, imaging\, and inverse graphics. \nCommittee:\nProf. Srinivasa G. Narasimhan (advisor)\nProf. Aswin C. Sankaranarayanan (advisor)\nProf. Matthew P. O’Toole\nSriram N. Narayanan
URL:https://www.ri.cmu.edu/event/understanding-image-intrinsics-through-light-and-heat/
LOCATION:NSH 3305
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260723T120000
DTEND;TZID=America/New_York:20260723T130000
DTSTAMP:20260912T191612
CREATED:20260716T160912Z
LAST-MODIFIED:20260716T160912Z
UID:152251-1784808000-1784811600@www.ri.cmu.edu
SUMMARY:A Robotic System for Tree Nursery Automation
DESCRIPTION:  \nAbstract:\nThe United States Green Industry faces a persistent labor shortage that motivates the adoption of agricultural automation.  However\, existing systems are not designed for the unstructured\, densely planted environment of a tree nursery. \nThis thesis presents a robotic system intended to alleviate this shortage while remaining usable by non-technical farmers\, built around a map-based representation of the nursery environment. A custom robotic platform\, the mini-Amiga\, and an accompanying LiDAR-camera-IMU sensor rig were developed to satisfy the maneuverability and payload requirements of tight nursery inter-row spacing. Point cloud maps constructed with this platform\, using the GLIM LiDAR-inertial SLAM framework augmented with a custom GNSS georeferencing extension\, were processed with a new constrained Gaussian Mixture Model algorithm to segment individual trees without requiring trunk visibility or large annotated training datasets. \nThe resulting per-tree map was further augmented with photographic colorization and encoded as a hierarchically organized Universal Scene Description (USD) scene\, supporting non-destructive\, multi-mode visualization and per-tree metadata storage intended for intuitive interaction by non-technical operators\, and was used to derive a Nav2-compatible occupancy grid and row-traversal paths intended for autonomous task execution. These results demonstrate that individual nursery trees can be accurately and efficiently segmented from point cloud data\, and that the resulting map can be represented in a form suited to both non-technical human interaction and autonomous navigation\, providing a practical foundation for future work integrating localization and autonomous task execution to fully realize the labor-saving potential of this system. \n\nCommittee:\nGeorge Kantor (advisor)\nMichael Kaess\nEaston Potokar
URL:https://www.ri.cmu.edu/event/a-robotic-system-for-tree-nursery-automation/
LOCATION:NSH 4305
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260721T123000
DTEND;TZID=America/New_York:20260721T133000
DTSTAMP:20260912T191612
CREATED:20260716T132808Z
LAST-MODIFIED:20260716T132808Z
UID:152108-1784637000-1784640600@www.ri.cmu.edu
SUMMARY:Evaluating World Models in Embodied Question Answering through Computational Primitives and Difficulty Progressions
DESCRIPTION:Abstract:\nLanguage modeling progress is largely evidenced by steadily rising scores on benchmarks of increasing apparent difficulty. From this\, the field infers increasingly general capabilities\, many of which presuppose robust world modeling. Interpreting a score\, however\, requires understanding both the task’s computational requirements and how the test-taker generalizes from them. Unlike humans\, who demonstrably generalize well\, large language models (LLMs) often do not: they instead learn heuristics fit to the minimal sufficient computational requirements of a task\, which may be far simpler than the task appears to demand. This talk extends this analysis to embodiment\, where multimodal LLMs (MLLMs) serve as the perception and reasoning core of embodied agents\, whose reliable deployment depends on benchmarked capability including a robust world model of the agent’s environment. \nWe evaluate world model robustness by independently manipulating object count\, duplication\, trajectory length\, and viewpoint change in a controlled synthetic benchmark of multi-frame egocentric trajectories\, and find that frontier MLLMs degrade sharply as difficulty increases\, while humans remain near ceiling. We then characterize embodied question answering (EQA) task demands through three nested paradigms: selecting relevant observations\, clustering nearby observations into local spatial-semantic units\, and propagating semantic state across experience. Stratifying questions by the weakest sufficient paradigm yields a difficulty progression. Prominent EQA benchmarks predominantly test only the weakest paradigm\, single-frame selection\, and so we introduce Campus-Bench\, three multi-hour\, campus-scale episode histories with questions stratified into selection and propagation regimes\, directly comparing model performance on the two over the same episodes. We additionally develop a diagnostic method in which an MLLM incrementally constructs and traverses a hierarchical spatial-semantic memory\, executing propagation explicitly. A frontier long-context MLLM performs strongly on selection but collapses on propagation; the same model leveraging our method dramatically improves it while remaining competitive on selection\, suggesting models struggle to maintain state internally. \nFrom this\, we argue that frontier MLLMs do not yet maintain the robust world models their benchmark scores suggest\, and that the field needs to make progress in formalizing tasks’ computational requirements and evaluating along their progressions of difficulty. \nThesis Committee:\nYonatan Bisk (Advisor)\nWennie Tabib\nHaochen Zhang
URL:https://www.ri.cmu.edu/event/evaluating-world-models-in-embodied-question-answering-through-computational-primitives-and-difficulty-progressions/
LOCATION:NSH 3305
CATEGORIES:MSR Thesis Presentation,Student Talks
END:VEVENT
END:VCALENDAR