Abstract:
There is a growing demand for mobile robots to act not only as passive observers but also to actively interact with their environment, especially in cluttered and social settings. Meeting this demand requires navigation systems that can both understand interaction-relevant properties of the environment and make plans accordingly. This thesis presents a modular approach to building such an interactive navigation system by addressing the challenges in the perception and planning components.

For perception, we propose a spatio-temporal SLAM module that constructs and maintains a semantic, instance-level map in real-time. By projecting 2D semantic features into 3D space and performing joint tracking and validation using both geometric and semantic cues from sensor data, the system produces a detailed 3D semantic map. This map includes geometric and semantic object features and continuously updates to reflect dynamic changes in the environment. Compared with baseline methods, the instance-level segmentation average precision in the static scene has improved by around 3x, and has the ability to model dynamic objects. The resulting 4D (3D + time) semantic representation enables the identification of candidate objects for interaction, along with their relevant properties.

For planning, we introduce a real-time interactive planning module based on an enhanced Directed Visibility Graph (DV-graph). This graph enables fast global path planning and supports real-time interaction planning that adapts to new sensory inputs. The planning framework is designed to handle complex, unknown, or partially known environments, allowing the robot to reason about and exploit the dynamics of objects to achieve navigation objectives. The average search time was improved by more than 100 times in large-scale environments, while maintaining the highest success rate and path quality.

The complete interactive navigation system, including both perception and planning modules, has been thoroughly validated through extensive experiments in both simulation and real-world scenarios. All experiments were conducted using onboard computation and in real time. Results demonstrate that the proposed system outperforms previous approaches that rely on offline processing. Furthermore, the system’s modular design makes it suitable for extension to broader applications beyond interactive navigation, such as supporting general-purpose interactions and mobile manipulation tasks.

Committee:
Ji Zhang (co-advisor)