Image-based rendering for visualisation of 3D scenes in near real-time

Abstract

In this research work, a software and hardware prototype for real—time 3D visualisation is developed. The proposed system takes two input videos to interpolate virtual in—between views, which are then combined into 3D videos after processing
for viewing on a 3D monitor. The core section of this research work is based on view morphing, a type of image based rendering. The image based rendering is a technique used to render a scene from a number of source images. According to
the knowledge of geometric information of captured scenes, the image based rendering technique can be classified into three categories: rendering without geometry, rendering with implicit geometry and rendering with explicit geometry. The view
morphing technique, a subset of the second category, requires less geometric information and a few source images of captured scenes. These reduce the complexities of both computation and hardware configuration of the proposed system, moreover, the quality of interpolated virtual in—between views by view morphing technique is good enough for visualisation applications. In this thesis, the research work is presented from two aspects: the algorithmic and the system's points of view separately.
The algorithmic development and optimisation consist of the procedures of automatically interpolating virtual in—between views from two source images. The work begun with two cameras calibration with the objective of finding out the geometric
relationship in 3D space between the two cameras. Image rectification is followed to project two source images into two parallel planes. This enables to obtain physically valid virtual in—between views and also reduces the computational cost for correspondence estimation. Subsequently, stereo matching is applied to establish feature correspondences between the two rectified source images. A novel feature based correspondence estimation algorithm is proposed to raise the level of the computational efficiency and the reliability. After that, interval interpolation is used to synthesise virtual in—between views. Finally, image derectification is applied to obtain final interpolated virtual in—between views.
A novel pseudo real—time 3D visualisation system is proposed in the system development and optimisation. The proposed system has been developed using the TI (Texas Instruments) DM642 EVM board which is a standalone Digital Media Processing board. The system also includes a stereo video capture module consisting of two PAL cameras and the X3D-19 DISPLAY AD 3D display unit for visualisation of 3D video output. The core algorithm utilises images captured from the cameras and generates 6 virtual in—between views using interpolation techniques. The combined views (eight views of 2D images) are displayed on the 3D monitor using a proprietary method developed specifically for X31) monitors. The advantage of the proposed system is that real 3D impressions are able to be visualised in front of the 3D monitor in near real—time without any special glasses. The proposed system has been evaluated on a number of real scenes. The experimental results indicate that the performance of the proposed 3D visualisation system is about 4.7 FPS.