Carnegie Mellon University
Temporal Segmentation of Facial Behavior

Fernando De la Torre Frade, Joan Campoy, Zara Ambadar, and J. F. Cohn
International Conference on Computer Vision, October, 2007.

  • Adobe portable document format (pdf) (3MB)
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Temporal segmentation of facial gestures in spontaneous facial behavior recorded in real-world settings is an important, unsolved, and relatively unexplored problem in facial image analysis. Several issues contribute to the challenge of this task. These include non-frontal pose, moderate to large out-of-plane head motion, large variability in the temporal scale of facial gestures, and the exponential nature of possible facial action combinations. To address these challenges, we propose a two-step approach to temporally segment facial behavior. The first step uses spectral graph techniques to cluster shape and appearance features invariant to some geometric transformations. The second step groups the clusters into temporally coherent facial gestures. We evaluated this method in facial behavior recorded during face-to-face interactions. The video data were originally collected to answer substantive questions in psychology without concern for algorithm development. The method achieved moderate convergent validity with manual FACS (Facial Action Coding System) annotation. Further, when used to preprocess video for manual FACS annotation, the method significantly improves productivity, thus addressing the need for ground-truth data for facial image analysis. Moreover, we were also able to detect unusual facial behavior.

Temporal segmentation, Clustering, face analysis, Facial expression recognition

Associated Center(s) / Consortia: Vision and Autonomous Systems Center
Associated Lab(s) / Group(s): People Image Analysis Consortium, Face Group, Component Analysis
Associated Project(s): Facial Expression Analysis and Spatio-Temporal Facial Expression Segmentation
Number of pages: 8

Text Reference
Fernando De la Torre Frade, Joan Campoy, Zara Ambadar, and J. F. Cohn, "Temporal Segmentation of Facial Behavior," International Conference on Computer Vision, October, 2007.

BibTeX Reference
   author = "Fernando {De la Torre Frade} and Joan Campoy and Zara Ambadar and J. F. Cohn",
   title = "Temporal Segmentation of Facial Behavior",
   booktitle = "International Conference on Computer Vision",
   month = "October",
   year = "2007",