DPSNet: End-to-end Deep Plane Sweep Stereo

Sunghoon Im, Hae-Gon Jeon, Stephen Lin and In So Kweon
Conference Paper, International Conference on Learning Representations (ICLR), December, 2018

View Publication

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.


Multiview stereo aims to reconstruct scene depth from images acquired by a camera under arbitrary motion. Recent methods address this problem through deep learning, which can utilize semantic cues to deal with challenges such as textureless and reflective regions. In this paper, we present a convolutional neural network called DPSNet (Deep Plane Sweep Network) whose design is inspired by best practices of traditional geometry-based approaches. Rather than directly estimating depth and/or optical flow correspondence from image pairs as done in many previous deep learning methods, DPSNet takes a plane sweep approach that involves building a cost volume from deep features using the plane sweep algorithm, regularizing the cost volume via a context-aware cost aggregation, and regressing the depth map from the cost volume. The cost volume is constructed using a differentiable warping process that allows for end-to-end training of the network. Through the effective incorporation of conventional multiview stereo concepts within a deep learning framework, DPSNet achieves state-of-the-art reconstruction results on a variety of challenging datasets.

author = {Sunghoon Im and Hae-Gon Jeon and Stephen Lin and In So Kweon},
title = {DPSNet: End-to-end Deep Plane Sweep Stereo},
booktitle = {International Conference on Learning Representations (ICLR)},
year = {2018},
month = {December},
keywords = {Deep Learning, Stereo, Depth, Geometry},
} 2019-01-07T12:59:40-04:00