Combined use of ToF sensors and standard cameras for 3D video acquisition

3D video applications usually require the availability of high resolution depth and color information. Depth information can be acquired at video rate by Time-of-Flight matrix sensors, but these devices usually have a limited resolution and image quality. A common solution to this issue is the combined use of ToF sensors and color cameras. This paper firstly presents a generalized multi-camera calibration technique that aims at calibrating together the ToF sensor with two synchronized cameras exploiting the color information from both type of sensors but also the depth measures of the ToF sensor. In the second part of the work we will present an ad-hoc interpolation technique to obtain high resolution depth information exploiting side information from the color camera together with the ToF measures and a novel surface prediction scheme. Finally we will show how the high resolution depth map obtained with the proposed approach can be used in order to warp the available video streams to arbitrary viewpoints in 3D video applications. Experimental results have shown how the proposed method allows to obtain a more accurate calibration and to improve the quality of the depth data and warped views if compared with standard approaches.