The purpose of our work is to develop a complete image based modeling and rendering software pipeline to extract photorealistic 3D models from a set of pictures or video sequences captured from a physical object. No constraints are imposed on the camera position and focus and the geometry of the object is unknown.

First, pictures of the physical object are acquired and digitized. Next, important features are detected (i.e. corners and silhouettes of the object). The typical dataset consists of 10 to 20 still images or a few seconds of video stream and an additional 10 to 50 features per frame. The second step is to automatically calculate the 3D position of the tracked features, the 3D position and orientation of cameras and their internal parameters (such as the focal length). This is a very complex problem and we have successfully developed algorithms that perform these calculations automatically using a non-linear least squares numerical approach.

Beginning the calibrated cameras and a subset of the images, we perform a spatial carving operation on a piece of virtual material (imagine an artist sculpting a raw block of marble). To do so we divide the space into small cubes (voxels) and iteratively remove those voxels that, when projected to each of the views, do not maintain a constant color value. The carving algorithm is computationally very intensive and we exploit the hardware acceleration of the actual video-cards to reduce the run time, making this method suitable for a standard PC.

In the last step, the volumetric model will be converted into an application specific format, generally a triangular mesh. An optional mesh simplification can be performed to achieve the maximum polygon count for the application. The images are being turned into texture maps to further enhance the model.

This system targets applications in areas such as Virtual Reality, Telepresence, Computer Gaming and Special Effects. All these environments require high quality 3D models for increased realism. Our approach allows us to obtain data directly from real world objects using images in a fast and automatic way.