Ground Up Design of a Multi-modal Object Detection System

Abstract

Rapid situational awareness is crucial to enabling a successful response from first responders during an emergency, where time is of the essence. Emergency personnel are often sent into incident scenes to gather information, but this is often a dangerous and slow process. Subterranean environments are particularly challenging due to hazards such as difficult terrain, low visibility, and outdated or incomplete maps. Individual sensors and sensing modalities are unable to reliably identify all categories of objects under these conditions. This work covers the development of a multimodal object detection and localization system to help provide situational awareness in subterranean environments.

We cover the development of two iterations of a modular sensing platform, algorithms to accurately detect and localize objects across multiple sensing modalities, and data transmission techniques to ensure timely updates for human operators from multiple robots in a fleet. Reported information is continually refined with new information from SLAM systems, ensuring global consistency is maintained. We demonstrate that the use of multiple sensors and sensing modalities is advantageous in reporting accurate and timely information. All evaluation is performed with data collected during the DARPA Subterranean Challenge Tunnel Circuit, where the proposed system was used to detect more than twice the number of objects of the next highest performing team, and where we won first place and an award for the most accurate object detected.