Scene disparity estimation with convolutional neural networks

Abstract

Estimation of stereovision disparity maps is important for many applications that require information about objects’ position and geometry. For example, as depth surrogate, disparity maps are essential for objects’ 3D shape reconstruction and indeed other applications that do require three dimensional representation of a scene. Recently, deep learning (DL) methodology has enabled novel approaches for the disparity estimation with some focus on the real-time processing requirement that is critical for applications in robotics and autonomous navigation. Previously, that constraint was not always addressed. Furthermore, for robust disparity estimation the occlusion effects should be explicitly modelled. In the described method the effective detection of occlusion regions is achieved through disparity estimation in both, forward and backward correspondence model with two matching deep networks. These two subnetworks are trained jointly in a single training process. Initially the subnetworks are trained using simulated data with the know ground truth, then to improve generalisation properties the whole model is fine-tuned in an unsupervised fashion on real data. During the unsupervised training the model is equipped with bilinear interpolation warping function to directly measure quality of the correspondence with the disparity maps estimated for both the left and right image. Also, during this phase forward-backward consistency constraint loss function is applied to regularise the disparity estimators for non-occluding pixels. The described network model computes, at the same time, the forward and backward disparity maps as well as corresponding occlusion masks. It showed improved results on simulated and real images with occluded objects, when compared with the results obtained without using the forward-backward consistency constraint loss function.