Vision-based Aircraft Detection and Tracking for Detect-and-Avoid - Robotics Institute Carnegie Mellon University

Vision-based Aircraft Detection and Tracking for Detect-and-Avoid

Master's Thesis, Tech. Report, CMU-RI-TR-22-11, Robotics Institute, Carnegie Mellon University, May, 2022

Abstract

Detect-and-Avoid (DAA) capabilities are critical for autonomous operations of small unmanned aircraft systems (sUAS). Traditionally DAA systems for large aircraft have been ground and radar-based. Due to the size, weight, and power (SWaP) constraints of sUAS, current DAA systems rely mainly on vision-based sensors and ADS-B (Automatic Dependent Surveillance-Broadcast) transponders. However, not all flying objects have transponders. Therefore, a vision-based DAA capability needs to exist for safe low-altitude autonomous flight.

In this work, we present the vision-based DAA problem, particularly the problem of aircraft detection and tracking. At long distances, the visual appearance of planes and helicopters is usually tiny in the context of the whole image. The problem is detecting tiny objects in high-resolution videos taken from a moving camera. Historically, this problem has been tackled using a multi-stage image processing pipeline involving: (1) ego-motion estimation, (2) background/foreground detection using morphological operations, and (3) tracking using temporal filtering. With the advent of deep learning, modern approaches rely on learning-based object detection methods. However, traditional object detection approaches typically do not scale well for this task due to real-time performance constraints.

Our approach to solving this problem follows a two-stage pipeline: (1) ego-motion estimation and (2) detection and tracking. Both of these stages are fully-convolution neural networks that can scale to large resolution inputs. They are trained on a labeled dataset released by Amazon Prime Air containing 3.3M+ images of airplanes, helicopters, drones, and other flying objects. We also developed our own aircraft data collection systems and designed a custom vision-based DAA payload for in-flight encounters. Through empirical evaluation of real-world data, our approach is compared with two baseline detection and tracking architectures and is shown to be superior. Analyzing our quantitative results in the context of DAA industry standard (ASTM F3442/F3442M - 20) we also show that the proposed method can satisfy the visual DAA surveillance requirements for certain classes of unmanned aircraft with a minimum cruise speed of 60-90kts, minimum turn rate of 21-31deg/s, and a minimum climb rate of 250-500ft/min.

BibTeX

@mastersthesis{Ghosh-2022-131695,
author = {Sourish Ghosh},
title = {Vision-based Aircraft Detection and Tracking for Detect-and-Avoid},
year = {2022},
month = {May},
school = {Carnegie Mellon University},
address = {Pittsburgh, PA},
number = {CMU-RI-TR-22-11},
keywords = {detection, tracking, aircraft, detect and avoid, sense and avoid, computer vision, object detection},
}