RI | Research | Projects | 2D->3D Face Model Construction

Text only version of this site

2D->3D Face Model Construction This project is no longer active. Head: Jing Xiao Contact: Jing Xiao Mailing address: Carnegie Mellon University Robotics Institute 5000 Forbes Avenue Pittsburgh, PA 15213 Associated center: VASC Associated labs/groups: Human Identification at a Distance and Face Group

Jump to: Project Description | Personnel | Publications

Project Description

We develop a linear algorithm that uniquely recovers the 3D non-rigid shapes and poses of a human face from a 2D monocular video. The non-rigid shapes can be regarded as a weighted combination of certain shape bases. We thus construct a 3D morphable model of the face using the recovered shape bases. Previous work on this problem utilized only orthonormality constraints on the camera rotations (rotation constraints). We prove that using only the rotation constraints results in ambiguous and invalid solutions. To eliminate the ambiguity, we present a set of novel constraints on the shape bases (basis constraints). It is proved that, under the weak-perspective projection model and the non-degenerate situations, enforcing both the basis and rotation constraints leads to a closed-form solution to reconstructing the non-rigid shape and motion. The accuracy and robustness of the approach is evaluated quantitatively on synthetic data and qualitatively on real video sequences. Once we have constructed a 3D face model, it can be used to track the face in 3D in real-time using our 3D AAM fitting algorithms.

The upper-left window shows the input video with tracked feature points. It contains head rotation and a few facial expressions.
The upper-middle window demonstrates the reconstructed 3D meshes formed by the feature points.
The upper-right window shows the reconstructed 3D locations of those feature points.

According to the reconstruction, we stabilized the face appearance into both the profile views and the frontal view, as shown in the lower windows. The non-rigid facial deformations such as mouth opening and eye closure were recovered.

Past members

	Name	Title	Email Address
	Simon Baker	Adjunct Faculty (Adjunct)
	Jinxiang Chai	PhD Student, RI
	Jeffrey Cohn	Adjunct Faculty (Adjunct)	jeffcohn@cs.cmu.edu
	Takeo Kanade	U.A. and Helen Whitaker University Prof.	tk@cs.cmu.edu
	Jing Xiao	PhD Student, RI

Publications

Note: This list may not be comprehensive. It contains only those publications in the RI publications database. Entries are listed in reverse chronological order.

• Non-Rigid Shape and Motion Recovery: Degenerate Deformations
  J. Xiao and T. Kanade
  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, June, 2004, pp. 668 - 675. [Abstract]
  Download: pdf [511 KB] copyrighted

• Real-Time Combined 2D+3D Active Appearance Models
  J. Xiao, S. Baker, I. Matthews, and T. Kanade
  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2, June, 2004, pp. 535 - 542. [Abstract]
  Download: pdf [5306 KB] copyrighted

• A Closed-Form Solution to Non-Rigid Shape and Motion Recovery
  J. Xiao, J. Chai, and T. Kanade
  The 8th European Conference on Computer Vision (ECCV 2004), May, 2004. [Abstract]
  Download: pdf [802 KB] copyrighted

• A Closed-Form Solution to Nonrigid Shape and Motion Recovery
  J. Xiao, J. Chai, and T. Kanade
  tech. report CMU-RI-TR-03-16, Robotics Institute, Carnegie Mellon University, June, 2003. [Abstract]
  Download: pdf [840 KB] copyrighted