The Breakdown:
- A caregiving robot that responds to spoken instructions while performing physical tasks may make robots easier to use and understand.
- The robot explains its actions, listens to feedback and adjusts in real time.
- CMU researchers tested the robot with residents at an independent living facility.
***
Residents at Baptist Senior Family’s Providence Point, a retirement community just a few miles from Carnegie Mellon University, recently tested a new kind of caregiving assistant: a conversational robotic arm designed to understand and respond to their speech.
As a robot scratched a resident’s arm, it announced, “I’m slowing down and increasing the pressure.”
“That’s too light,” the resident responded. “A little more pressure.”
“I’m increasing the pressure slightly,” the robot responded, and then smoothly changed its movements to match the request.
The interaction represents a unique advancement in human-robot interaction. The robot working with residents at Providence Point wasn’t just following commands. It was conversing with them, interpreting what they said and responding in real-time.
The robotic arm adjusts its movements based on user feedback.
“The interaction with the robot through spoken word was amazing to me,” said Jim Strader, a Providence Point resident. “When it adjusted its movement based on what I said — that was the most interesting part, and I was happy to participate.”
These human-robot exchanges lie at the center of a new communication system developed by the Robotic Caregiving and Human Interaction lab (RCHI) in CMU’s Robotics Institute (RI). Researchers are exploring what happens when assistive robots can converse through natural dialogue and adapt their actions according to human preferences. The lab’s project, “Bidirectional Human-Robot Communication for Physical Human-Robot Interaction,” gives robots the ability to speak their intentions, listen for human inputs, and provide verbal responses while physically interacting with someone.
“Natural language is a way that a lot of people communicate with others in their daily lives. We wanted to create an interface that many users could pick up and use without prior training,” said Jim Wang, a Ph.D. student in the RI and lead researcher on the project. “We wanted it to be intuitive for the robot to verbalize its plans and for the user to hear them as the robot moves.”
The team built a system that allows the robot to interpret user speech through a large language model (LLM), grounding spoken commands in the robot’s planned movement trajectory and ongoing conversation. The system enables bidirectional communication, in which the robot listens and adjusts its movements based on user input, while also speaking to confirm an intended adjustment or ask a clarifying question, much as a human caregiver would.
In the early stages of the research, the robot focused on unidirectional narration, where it simply announced its planned motions before executing them. This technique helped users anticipate physical contact and promoted trust. As the project progressed, the team moved beyond narration to full bidirectional communication, where user commands actively shape the robot’s motions and the robot responds with confirmation or clarifying questions.
“ LLMs have developed significantly to understand many phrases, even if they are dependent on context,” Wang said. “That gives our robot the power to interpret even vague instructions or to know that it should ask a follow-up question.”
The team built a filtering system that helps the robot focus only on task-relevant input. Casual remarks are ignored, but feedback about the robot’s performance triggers a response. For example, if a user notes that pressure from the robot arm attachment feels inconsistent, the robot asks a follow-up question to pinpoint where the issue is and adjust its behavior. This selective listening keeps the robot grounded in the task while bringing the interactions closer to natural conversation.
“These findings underscore the importance of transparency in physical human-robot interactions,” Wang said. “Transparency not only calms anxieties humans have about working with a robot, but it also continually builds trust between robot and user. That trust is what makes safe, effective caregiving possible.”
The team’s work was funded by Honda R&D Americas Inc. and was accepted to the 2026 Human-Robot Interaction conference. To learn more about the research, visit the project website.
For More Information: Aaron Aupperlee | 412-268-9068 | aaupperlee@cmu.edu