The face is the fundamental part of day to day interpersonal communication. Humans use the face along with facial expressions to denote consciously their emotional states (anger, surprise, stress, etc) or subconsciously (yawn, lip biting), to accompany and enhance the meaning of their thoughts (wink) or exchange thoughts without talking (head nodes, look exchanges). Facial expressions are the result of the deformations in a human face due to muscle movement. The importance of automating the task to analyse facial expressions by computing systems is apparent and can be beneficial to many different scientific subjects such as psychology, neurology, psychiatry as well as applications for everyday life such as driver monitoring systems, automated tutoring systems or smart environments and human-computer interaction. Although humans are able to identify changes in facial expressions easily and effortlessly even in complicated scenes, the same is not an easy task to be undertaken by a machine. Moreover computing systems must share the same robustness and accuracy with a human so that these systems could be used in a real-world scenario and provide adequate aid.

Advances in topics such as face detection, face tracking and recognition, psychological studies as well as the processing power of modern computer systems make the automatic analysis of facial expressions possible for use with real world examples where responsiveness (i.e. real time processing) is required along with sensitivity (i.e. being able to detect various day to day emotional states and visual cues) and the ability to tolerate head movements or sudden changes.

For an effective automatic facial expression recognition (AFER) system there are several characteristics that must be present so that it can be efficient. These are outlined in the figure below.

Face detection and identification of prominent features is a crucial step for an AFER system. It is the first step for any system that carries the automatic tag and the performance of this step in terms of accuracy determines the overall accuracy of the system. Various approaches are presented in the literature in terms of static or temporal identification of the face or identification of prominent features such as eyes in contrast to identifying the presence of a face in a scene.

When the face is located it must be modelled so that it can be represented in an appropriate manner, in a machine readable form. The facial representation could be based on the facial geometry that encompasses some unique features of homogeneity and diversion across humans. It could also be based in characteristics that appear after some transformation with mathematical expressions modelling texture, position and gray-level information. After that the feature vector is built by extracting features. It can be represented either holistically or locally. Holistic approach treats the face as a whole. The processing of the face and the mathematical information applies to the whole face without regarding any special prominent features of it. On the other hand the local approach treats each prominent feature of the face in a different way and the feature extraction process is applied in selected locations in the image which are often called fiducial points. Lastly there are systems proposed in scientific literature that combine the two approaches, treating the face in a hybrid manner. There is also a distinction in terms of the presence of temporal information or not, that is if the feature extraction process occurs in image sequences or static images.

Classification is the last step for an AFER system. The facial actions or the deformations due to facial movement are categorised either as prototypic, basic emotions or as Action Units (AUs). Basic emotions are derived from the work of Ekman who concluded that there are six emotions that are unique and are shared across different communities irrespective of cultural diversions. The six basic emotions are: disgust, fear, joy, surprise, sadness and anger. Action Units describe facial deformations for each facial muscle. There are a total of 44 AUs where the majority involves the contraction or relaxation of facial muscles and the rest involve miscellaneous actions such as “tongue show” or “bite lip”.