/Season-Invariant Semantic Segmentation with A Deep Multimodal Network

Season-Invariant Semantic Segmentation with A Deep Multimodal Network

Dong-Ki Kim, Daniel Maturana, Masashi Uenoyama and Sebastian Scherer
Conference Paper, Field and Service Robotics, September, 2017

Download Publication (PDF)

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.


Semantic scene understanding is a useful capability for autonomous vehicles operating in off-roads. While cameras are the most common sensor used for semantic classification, the performance of methods using camera imagery may suffer when there is significant variation between the train and testing sets caused by illumination, weather, and seasonal variations. On the other hand, 3D information from active sensors such as LiDAR is comparatively invariant to these factors, which motivates us to investigate whether it can be used to improve performance in this scenario. In this paper, we propose a novel multimodal Convolutional Neural Network (CNN) architecture consisting of two streams, 2D and 3D, which are fused by projecting 3D features to image space to achieve a robust pixelwise semantic segmentation. We evaluate our proposed method in a novel off-road terrain classification benchmark, and show a 25% improvement in mean Intersection over Union (IoU) of navigation-related semantic classes, relative to an image-only baseline.

BibTeX Reference
author = {Dong-Ki Kim and Daniel Maturana and Masashi Uenoyama and Sebastian Scherer},
title = {Season-Invariant Semantic Segmentation with A Deep Multimodal Network},
booktitle = {Field and Service Robotics},
year = {2017},
month = {September},
editor = {Prof. Dr. Marco Hutter, Roland Siegwart},
publisher = {Springer International Publishing},