Algorithms for Super-Resolutoin and Restoration of Noiseless and Noisy Depth Images

Abstract

Over the last decade, along with intensity images depth images are also gaining popularity because of its demand in applications like robot navigation, augmented reality, 3DTV, etc. The distinctive characteristic of depth image is that each pixel value represents the distance from the camera position, unlike optical image where each pixel represent intensity values. The prominent features of depth images are the edges and the corners, but it lacks texture unlike optical images. The modern high-end depth cameras provide depth map with higher spatial resolution and higher bit-width, but they are bulky and expensive. However, on the other hand, the commercial low-end depth cameras provide lower spatial resolution, smaller bit width, and are relatively inexpensive. Moreover, the depth images captured by such cameras are noisy and may have some missing regions. To deal with problems like noise and missing regions in the images, the image processing methods like image denoising and image inpainting can be used. Super-resolution (SR) methods address the problem of lower spatial resolution by taking low-resolution (LR) input image and produce high-resolution (HR) image with minimal perturbation in image details. In literature, several super-resolution (SR) and depth reconstruction (DR) methods have been proposed to address the problems associated with these low-end depth camera. We propose few methods to address the above mentioned issues related to the process of super resolution and restoration of depth images. Wavelets have been used for decades for image compression, image denoising and image enhancement because of its better localization in time (space) and frequency. In the proposed work, a wavelet transform based single depth image SR method has been proposed. It uses discrete wavelet transform (DWT), stationary wavelet transform (SWT), and the image gradient. The proposed method is an intermediate stage for obtaining the high-frequency contents from different subbands obtained through DWT, viiSWT and gradient operations on the input LR image and estimates the SR image. For super-resolution by larger factors, i.e. ×4 or ×8 or higher, the guided approach has been used in literature which makes use of the corresponding HR guidance colour image which are easy to capture. In this work, we propose a HR colour-image guided depth image SR method that makes use of the segment cues from the HR colour image. The cues are obtained by segmentation of the HR colour image using popular segmentation methods such as mean-shift algorithm (MS) or simple linear iterative clustering (SLIC) segmentation algorithms. Like other guidance image based methods, it is assumed that the prominent edges in the depth image coincides with the edges in the HR guidance colour image. The median of a segment in the initial estimated depth image corresponding to the segment in the guiding HR colour image is computed. This median value replaces the depth value in that identified segment of the initial estimated depth image. After processing all the segments, we get a final SR output with better edge details and reduced noise. Bilateral filtering can be applied as post processing to smooth the variations at the abutting segment regions. The initial estimate of the SR depth image is derived from LR depth image using the following two approaches. The first one is with bicubic interpolation to the required spatial resolution and the SR process which uses this is referred as LRBicSR method in this work. The other method maps the LR points on to the HR grid and super resolves; this method is referred to as LRSR method. Processing of sparse depth images involves two stages namely DR and SR in that order. This framework of DR followed by SR is called as DRSR method and is challenging. The sparse depth images used for processing may have sparseness range between 1% and 15% of the total pixels. Processing of very sparse depth images of the order of 1% is highly challenging and has been reasonably reconstructed. The corresponding RGB images have been used for guiding the reconstruction process. Two approaches have been proposed to estimate the unknown depth values in the sparse depth input. First one being the plane fitting approach (PFit) and the other being the median filling approach (MFill). This work also shows that guidance based methods are useful in overcoming the effect of noise in depth images and inpainting of the missing regions in viiithe depth images. Literature contains SR methods for intensity images that use a set of training images to learn the HR-LR relationship. In this work, a learning based method has been proposed where algorithm learns the image details from the HR and LR pairs of training images using Gaussian mixture model (GMM). It has been observed from the conducted experiments that, for larger SR factors, the learned parameters do not help much in learning the finer details. So, hierarchical approach has been proposed for such factors and the approach tend to give better SR image quality. The anisotropic total generalized variation method (ATGV) available in the literature is an iterative method and the quality of the SR image so obtained with this method is dependent on the number of iterations used. A simple and less computationally intensive Residual interpolation method (RI) has been used as a preprocessor for ATGV. The computational complexity of RI is comparable to the computational intensity of classical bicubic interpolation method. RI provides a better initial estimate to the ATGV. It has been observed that the proposal of cascading the RI as a preprocessor reduces the number of iterations, converges faster to achieve the better SR image quality. For experimentation, we have used the freely available Middlebury depth dataset, which has depth images along with their corresponding registered colour image. Another dataset used is Kitti dataset which has depth images of outdoor scenes. Real-time depth images captured from Kinect camera and ToF camera has also been used in the experiments to show the robustness of the proposed methods. The LR image is generated from the ground truth (GT) image by blurring, downsampling and adding noise to it. Several performance metrics e.g. peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and root mean square error (RMSE) have been used to evaluate the performance.