Recognizing Facial Actions by Combining Geometric Features and Regional Appearance Patterns

Abstract

In facial expression analysis, two principle approaches to extract facial features are geometric feature-based methods and appearance-based methods such as Gabor filters. In this paper, we combine these approaches in a feature-based system to recognize Facial Action Coding System (FACS) action units (AUs) in a complex database. The geometric facial features (including mouth, eyes, brows, and cheeks) are extracted using multi-state facial component models. After extraction, these features are represented parametricly. The regional facial appearance patterns are captured using a set of multi-scale and multi-orientation Gabor wavelet filters at specific locations. For the upper face, we recognize 8 AUs and neutral expression. The database consists of 606 image sequences from 107 adults of European, African, and Asian ancestry. AUs occur both alone and in combinations. Average recognition rate is 87.6% by using geometric facial features alone, 32% by using regional appearance patterns alone, 89.6% by combining both features, and 92.7% after refinement. For the lower face, we recognize 13 AUs and neutral expression. The database consists of 514 image sequences from 180 adults of European, African, and Asian ancestry. AUs occur both alone and in combinations. Average recognition rate is 84.7% by using geometric facial features alone, 82% by combining both features, and 87.4% after refinement.