2D->3D Face Model Construction

We develop a linear algorithm that uniquely recovers the 3D non-rigid shapes and poses of a human face from a 2D monocular video. The non-rigid shapes can be regarded as a weighted combination of certain shape bases. We thus construct a 3D morphable model of the face using the recovered shape bases. Previous work on this problem utilized only orthonormality constraints on the camera rotations (rotation constraints). We prove that using only the rotation constraints results in ambiguous and invalid solutions. To eliminate the ambiguity, we present a set of novel constraints on the shape bases (basis constraints). It is proved that, under the weak-perspective projection model and the non-degenerate situations, enforcing both the basis and rotation constraints leads to a closed-form solution to reconstructing the non-rigid shape and motion. The accuracy and robustness of the approach is evaluated quantitatively on synthetic data and qualitatively on real video sequences. Once we have constructed a 3D face model, it can be used to track the face in 3D in real-time using our 3D AAM fitting algorithms.

The upper-left window shows the input video with tracked feature points. It contains head rotation and a few facial expressions.
The upper-middle window demonstrates the reconstructed 3D meshes formed by the feature points.
The upper-right window shows the reconstructed 3D locations of those feature points.

According to the reconstruction, we stabilized the face appearance into both the profile views and the frontal view, as shown in the lower windows. The non-rigid facial deformations such as mouth opening and eye closure were recovered.