Automatic analysis of facial affect: A survey of registration, representation, and recognition

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 37, Issue 6, 2015, Pages 1113-1133

  Sariyanidi, Evangelos ^a   Gunes, Hatice ^a   Cavallaro, Andrea ^a  

^a QUEEN MARY UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

Affect sensing and analysis; facial expressions; facial representations; registration; survey

Indexed keywords

ENCODING (SYMBOLS); HUMAN COMPUTER INTERACTION; SURVEYING; SURVEYS;

AFFECT SENSING AND ANALYSIS; COMPREHENSIVE ANALYSIS; DIMENSIONALITY REDUCTION; FACIAL EXPRESSIONS; FACIAL REPRESENTATIONS; FUNDAMENTAL COMPONENT; ILLUMINATION VARIATION; REGISTRATION;

FACE RECOGNITION;

ALGORITHM; AUTOMATED PATTERN RECOGNITION; FACE; FACIAL EXPRESSION; HUMAN; PHYSIOLOGY; PROCEDURES;

ALGORITHMS; FACE; FACIAL EXPRESSION; HUMANS; PATTERN RECOGNITION, AUTOMATED;

EID: 84929192741     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2014.2366127     Document Type: Review

References (191)

1. R. Adolphs, "Recognizing emotion from facial expressions: Psychological and neurological mechanisms," Behav. Cognitive Neurosci. Rev., vol. 1, no. 1, pp. 21-62, 2002.
2. R. Adolphs, "Perception and emotion: How we recognize facial expressions," Current Directions Psychol. Sci., vol. 15, no. 5, pp. 222-226, 2006.
3. T. Ahonen, A. Hadid, and M. Pietikainen, "Face description with local binary patterns: Application to face recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 12, pp. 2037-2041, Dec. 2006.
4. Z. Ambadar, J. W. Schooler, and J. Cohn, "Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expressions," Psychol. Sci., vol. 16, no. 5, pp. 403-410, 2005.
5. N. Ambady and R. Rosenthal, "Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis," Psychol. Bull., vol. 11, no. 2, pp. 256-274, 1992.
6. J. Angulo and F. Meyer, "Morphological exploration of shape spaces," in Mathematical Morphology and Its Applications to Image and Signal Processing. New York, NY, USA: Springer, 2009, pp. 226-237.
7. T. Baenziger, M. Mortillaro, and K. R. Scherer, "Introducing the Geneva multimodal expression corpus for experimental research on emotion perception," Emotion, vol. 12, no. 5, pp. 1161-1179, 2012.
8. S. Baker and I. Matthews, "Lucas-Kanade 20 years on: A unifying framework," Int. J. Computer Vision, vol. 56, no. 3, pp. 221-255, 2004.
9. T. Baltrusaitis, N. Banda, and P. Robinson, "Dimensional affect recognition using continuous conditional random fields," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2013, pp. 1-8.
10. M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Automatic recognition of facial actions in spontaneous expressions," J. Multimedia, vol. 1, no. 6, pp. 22-35, 2006.
11. S. Berretti, B. B. Amor, M. Daoudi, and A. Del Bimbo, "3D facial expression recognition using SIFT descriptors of automatically detected keypoints," Vis. Comput., vol. 27, no. 11, pp. 1021-1036, 2011.
12. C. M. Bishop, and N. M. Nasrabadi, Pattern Recognition and Machine Learning. New York, NY, USA: Springer, 2006.
13. Y.-L. Boureau, N. Le Roux, F. Bach, J. Ponce, and Y. LeCun, "Ask the locals: Multi-way local pooling for image recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 2651-2658.
14. Y.-L. Boureau, J. Ponce, and Y. LeCun, "A theoretical analysis of feature pooling in visual recognition," in Proc. Int. Conf. Mach. Learn., 2010, pp. 111-118.
15. S. Brahnam, C.-F. Chuang, R. S. Sexton, and F. Y. Shih, "Machine assessment of neonatal facial expressions of acute pain," Decision Support Syst., vol. 43, no. 4, pp. 1242-1254, 2007.
16. S. Butler, J. Tanaka, M. Kaiser, and R. Le Grand, "Mixed emotions: Holistic and analytic perception of facial expressions," J. Vis., vol. 9, no. 8, p. 496, 2009.
17. A. J. Calder, A. M. Burton, P. Miller, A. W. Young, and S. Akamatsu, "A principal component analysis of facial expressions," Vis. Res., vol. 41, no. 9, pp. 1179-1208, 2001.
18. A. Calder, G. Rhodes, M. Johnson, and J. Haxby, Oxford Handbook of Face Perception. Oxford, U.K.: Oxford Univ. Press, 2011.
19. E. Candes and M. Wakin, "An introduction to compressive sampling," IEEE Signal Process. Mag., vol. 25, no. 2, pp. 21-30, Mar. 2008.
20. O. Çeliktutan, S. Ulukaya, and B. Sankur, "A comparative study of face landmarking techniques," EURASIP J. Image Video Process., vol. 2013, no. 1, p. 13, 2013.
21. K.-Y. Chang, C.-S. Chen, and Y.-P. Hung, "Intensity rank estimation of facial expressions based on a single image," in IEEE Int. Conf. Syst., Man, Cybern., 2013, pp. 3157-3162.
22. Y. Chang, C. Hu, and M. Turk, "Manifold of facial expression," in Proc. IEEE Int. Workshop Anal. Model. Faces Gestures, 2003, pp. 28-35.
23. S. Chew, P. Lucey, S. Lucey, J. Saragih, J. Cohn, I. Matthews, and S. Sridharan, "In the pursuit of effective affective computing: The relationship between features and registration," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 4, pp. 1006-1016, Aug. 2012.
24. W.-S. Chu, F. De La Torre, and J. Cohn, "Selective transfer machine for personalized facial action unit detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 3515-3522.
25. J. Cohn and K. L. Schmidt, "The timing of facial motion in posed and spontaneous smiles," Int. J. Wavelets, Multiresolution Inf. Process., vol. 2, no. 2, pp. 121-132, 2004.
26. T. F. Cootes, G. J. Edwards, and C. J. Taylor, "Active appearance models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 6, pp. 681-685, Jun. 2001.
27. S. Cotter, "Sparse representation for accurate classification of corrupted and occluded facial expressions," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2010, pp. 838-841.
28. G. W. Cottrell, M. N. Dailey, C. Padgett, and R. Adolphs, "Is all face processing holistic? The view from UCSD," in Computational, Geometric, and Process Perspectives on Facial Cognition. Florence, KY, USA: Psychology Press, 2001, pp. 347-396.
29. A. Cruz, B. Bhanu, and S. Yang, "A psychologically-inspired match-score fusion mode for video-based facial expression recognition," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 341-350.
30. M. Dahmane and J. Meunier, "Continuous emotion recognition using Gabor energy filters," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 351-358.
31. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., vol. 1, 2005, pp. 886-893.
32. C. Darwin, The Expression of the Emotions in Man and Animals. Oxford, U.K.: Oxford Univ. Press, 1998.
33. F. De la Torre, T. Simon, Z. Ambadar, and J. F. Cohn, "Fast-FACS: A computer-assisted system to increase speed and reliability of manual FACS coding," in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 57-66.
34. A. Dhall, A. Asthana, R. Goecke, and T. Gedeon, "Emotion recognition using PHOG and LPQ features," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2011, pp. 866-871.
35. X. Ding, W.-S. Chu, F. De La Torre, J. F. Cohn, and Q. Wang, "Facial action unit event detection by cascade of tasks," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2400-2407.
36. E. Douglas-Cowie, N. Campbell, R. Cowie, and P. Roach, "Emotional speech: Towards a new generation of databases," Speech Commun., vol. 40, no. 1, pp. 33-60, 2003.
37. P. Ekman, J. Campos, R. Davidson, and F. D. Waals, Emotions Inside Out (ser. Annals of the New York Academy of Sciences), vol. 1000. New York, NY, USA: New York Acad. Sci., 2003.
38. P. Ekman, W. Friesen, and J. Hager, The Facial Action Coding System, 2nd ed. London, U.K.: Weidenfeld and Nicolson, 2002.
39. P. Etyngier, F. Segonne, and R. Keriven, "Shape priors using manifold learning techniques," in Proc. IEEE Int. Conf. Comput. Vis., 2007, pp. 1-8.
40. W. A. Freiwald, D. Y. Tsao, and M. S. Livingstone, "A face feature space in the macaque temporal lobe," Nature Neurosci., vol. 12, no. 9, pp. 1187-1196, 2009.
41. J. Friedman, T. Hastie, and R. Tibshirani, "Additive logistic regression: A statistical view of boosting," Ann. Statist., vol. 28, no. 2, pp. 337-407, 2000.
42. (2013, Aug. 6). Gabor Filtering Implementation. [Online]. Available: http://luks.fe.uni-lj.si/en/staff/vitomir/
43. M. Glodek, S. Tschechne, G. Layher, M. Schels, T. Brosch, S. Scherer, M. Kächele, M. Schmidt, H. Neumann, G. Palm, and F. Schwenker, "Multiple classifier systems for the classification of audio-visual emotional states," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 359-368.
44. R. C. Gonzalez and R. E. Woods, Digital Image Processing. Englewood Cliffs, NJ, USA: Prentice-Hall, 2008.
45. T. Gritti, C. Shan, V. Jeanne, and R. Braspenning, "Local features based facial expression recognition with face registration errors," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2008, pp. 1-8.
46. R. Gross, I. Matthews, and S. Baker, "Generic vs. person specific active appearance models," Image Vis. Comput., vol. 23, no. 12, pp. 1080-1093, 2005.
47. R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker, "Multipie," Image Vis. Comput., vol. 28, no. 5, pp. 807-813, 2010.
48. H. Gunes and B. Schuller, "Categorical and dimensional affect analysis in continuous input: Current trends and future directions," Image Vis. Comput., vol. 31, no. 2, pp. 120-136, 2013.
49. Z. Hammal and J. F. Cohn, "Automatic detection of pain intensity," in Proc. ACM Int. Conf. Multimodal Interfaces, 2012, pp. 47-52.
50. P. O. Hoyer, "Non-negative matrix factorization with sparseness constraints," J. Mach. Learn. Res., vol. 5, pp. 1457-1469, 2004.
51. K.-C. Huang, S.-Y. Huang, and Y.-H. Kuo, "Emotion recognition based on a novel triangular facial feature extraction method," in Proc. Int. Joint Conf. Neural Networks, 2010, pp. 1-6.
52. X. Huang, G. Zhao, W. Zheng, and M. Pietikäinen, "Towards a dynamic expression recognition system under facial occlusion," Pattern Recognit. Lett., vol. 33, no. 16, pp. 2181-2191, 2012.
53. D. H. Hubel and T. N. Wiesel, "Receptive fields, binocular interaction and functional architecture in the cat's visual cortex," J. Physiol., vol. 160, no. 1, pp. 106-154, 1962.
54. L. A. Jeni, J. Girard, J. Cohn, and F. De La Torre, "Continuous AU intensity estimation using localized, sparse facial feature space," in Proc. IEEE Int. Conf. Autom. Face and Gesture Recognit. Workshops, 2013, pp. 1-7.
55. L. A. Jeni, J. F. Cohn, and F. De La Torre, "Facing imbalanced datarecommendations for the use of performance metrics," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2013, pp. 245-251.
56. B. Jiang, M. Valstar, B. Martinez, and M. Pantic, "Dynamic appearance descriptor approach to facial actions temporal modelling," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 44, no. 2, pp. 161-174, Feb. 2014.
57. B. Jiang, M. Valstar, and M. Pantic, "Action unit detection using sparse appearance descriptors in space-time video volumes," in Proc. IEEE Int. Conf.Autom. FaceGesture Recognit., 2011, pp. 314-321.
58. M. Kaiser, B. Kwolek, C. Staub, and G. Rigoll, "Registration of 3D facial surfaces using covariance matrix pyramids," in Proc. IEEE Int. Conf. Robot. Autom., 2010, pp. 1002-1007.
59. S. Kaltwang, O. Rudovic, and M. Pantic, "Continuous pain intensity estimation from facial expressions," in Proc. Int. Symp. Adv. Vis. Comput., 2012, pp. 368-377.
60. J.-K. Kamarainen, V. Kyrki, and H. Kalviainen, "Invariance properties of Gabor filter-based features-Overview and applications," IEEE Trans. Image Process., vol. 15, no. 5, pp. 1088-1099, May 2006.
61. T. Kanade, J. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2000, pp. 46-53.
62. S. Koelstra, M. Pantic, and I. Patras, "A dynamic texture-based approach to recognition of facial actions and their temporal models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 1940-1954, Nov. 2010.
63. M. Kyperountas, A. Tefas, and I. Pitas, "Salient feature and reliable classifier selection for facial expression classification," Pattern Recognit., vol. 43, no. 3, pp. 972-986, 2010.
64. M. Lades, J. Vorbruggen, J. Buhmann, J. Lange, C. von der Malsburg, R. Wurtz, and W. Konen, "Distortion invariant object recognition in the dynamic link architecture," IEEE Trans. Comput., vol. 42, no. 3, pp. 300-311, Mar. 1993.
65. S. Lazebnik, C. Schmid, and J. Ponce, "Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., vol. 2, 2006, pp. 2169-2178.
66. (2013, Aug. 6). LBP and LBP-TOP Implementations. [Online]. Available: http://www.cse.oulu.fi/CMV/Downloads/ LPQMatlab
67. (2013, Aug. 6). LBP Implementation. [Online]. Available: http:// www.cse.oulu.fi/CMV/Downloads/LBPMatlab
68. (2013, Aug. 6). LBP-TOP Implementation. [Online]. Available: http://www.cse.oulu.fi/CMV/Downloads/LBPMatlab
69. Y. LeCun, "Learning invariant feature hierarchies," in Proc. Eur. Conf. Comput. Vis. Workshops Demonstrations, 2012, vol. 7583, pp. 496-505.
70. Y. LeCun, K. Kavukcuoglu, and C. Farabet, "Convolutional networks and applications in vision," in Proc. IEEE Int. Symp. Circuits Syst., 2010, pp. 253-256.
71. S. Lee, "Symmetry-driven shape description for image retrieval," Image Vis. Comput., vol. 31, no. 4, pp. 357-363, 2013.
72. X. Li, T. Pfister, X. Huang, G. Zhao, and M. Pietikäinen, "A spontaneous micro facial expression database: Inducement, collection and baseline," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2013, pp. 1-6.
73. Y. Li, J. Chen, Y. Zhao, and Q. Ji, "Data-free prior model for facial action unit recognition," IEEE Trans. Affective Comput., vol. 4, no. 2, pp. 127-141, Apr.-Jun. 2013.
74. C.-T. Liao, H.-J. Chuang, C.-H. Duan, and S.-H. Lai, "Learning spatial weighting for facial expression analysis via constrained quadratic programming," Pattern Recognit., vol. 46, pp. 3103-3116, 2013.
75. S. Liao, W. Fan, A. Chung, and D.-Y. Yeung, "Facial expression recognition using advanced local binary patterns, Tsallis entropies and global appearance features," in Proc. IEEE Int. Conf. Image Process., 2006, pp. 665-668.
76. G. C. Littleword, M. S. Bartlett, and K. Lee, "Automatic coding of facial expressions displayed during posed and genuine pain," Image Vis. Comput., vol. 27, no. 12, pp. 1797-1803, 2009.
77. G. Littlewort, J. Whitehill, T.-F. Wu, N. Butko, P. Ruvolo, J. Movellan, and M. Bartlett, "The motion in emotion - a CERT based approach to the FERA emotion challenge," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2011, pp. 897-902.
78. C. Liu, J. Yuen, and A. Torralba, "SIFT flow: Dense correspondence across scenes and its applications," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 978-994, May 2011.
79. F. Long, T. Wu, J. R. Movellan, M. S. Bartlett, and G. Littlewort, "Learning spatiotemporal features by using independent component analysis with application to facial expression recognition," Neurocomputing, vol. 93, no. 0, pp. 126-132, 2012.
80. D. G. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
81. (2013, Aug. 6). LPQ-TOP Implementation. [Online]. Available: http://www.cse.oulu.fi/CMV/Downloads/LPQMatlab
82. P. Lucey, J. Cohn, I. Matthews, S. Lucey, S. Sridharan, J. Howlett, and K. Prkachin, "Automatically detecting pain in video through facial action units," IEEE Trans. Syst., Man Cybern. B, Cybern., vol. 41, no. 3, pp. 664-674, Jun. 2011.
83. P. Lucey, J. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, "The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2010, pp. 94-101.
84. P. Lucey, J. Cohn, K. M. Prkachin, P. E. Solomon, S. Chew, and I. Matthews, "Painful monitoring: Automatic pain monitoring using the UNBC-McMaster shoulder pain expression archive database," Image Vis. Comput., vol. 30, no. 3, pp. 197-205, 2012.
85. S. Lucey, A. B. Ashraf, and J. Cohn, "Investigating spontaneous facial action recognition through AAM representations of the face," in Face Recognition Book. Mamendorf, Germany: Pro Literatur Verlag, 2007.
86. M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, "Coding facial expressions with Gabor wavelets," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 1998, pp. 200-205.
87. M. Mahoor, S. Cadavid, D. Messinger, and J. Cohn, "A framework for automated measurement of the intensity of non-posed facial action units," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2009, pp. 74-80.
88. M. H. Mahoor, M. Zhou, K. L. Veon, S. M. Mavadati, and J. F. Cohn, "Facial action unit recognition with sparse representation," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2011, pp. 336-342.
89. A. Martinez, "Deciphering the face," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2011, pp. 7-12.
90. S. Mavadati, M. Mahoor, K. Bartlett, P. Trinh, and J. Cohn, "DISFA: A spontaneous facial action intensity database," IEEE Trans. Affective Comput., vol. 4, no. 2, pp. 151-160, Jul.-Sep. 2013.
91. G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder, "The SEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent," IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 5-17, Jan.-Mar. 2012.
92. A. Metallinou and S. Narayanan, "Annotation and processing of continuous emotional attributes: Challenges and opportunities," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., Apr. 2013, pp. 1-8.
93. E. Meyers and L. Wolf, "Using biologically inspired features for face processing," Int. J. Comput. Vis., vol. 76, no. 1, pp. 93-104, 2008.
94. S. Moore and R. Bowden, "Local binary patterns for multi-view facial expression recognition," Comput. Vis. Image Understanding, vol. 115, no. 4, pp. 541-558, 2011.
95. D. Neth, "Facial configuration and the perception of facial expression," Ph.D. dissertation, Ohio State University, 2007.
96. D. Neth and A. M. Martinez, "A computational shape-based model of anger and sadness justifies a configural representation of faces," Vis. Res., vol. 50, no. 17, pp. 1693-1711, 2010.
97. M. Nicolaou, H. Gunes, and M. Pantic, "Output-associative RVM regression for dimensional and continuous emotion prediction," Image Vis. Comput., vol. 30, no. 3, pp. 186-196, 2012.
98. M. Nicolaou, V. Pavlovic, and M. Pantic, "Dynamic probabilistic CCA for analysis of affective behaviour," in Proc. Eur. Conf. Comput. Vis., 2012, pp. 98-111.
99. J. Nicolle, V. Rapp, K. Bailly, L. Prevost, and M. Chetouani, "Robust continuous prediction of human emotions using multiscale dynamic cues," in Proc. ACM Int. Conf. Multimodal Interaction, 2012, pp. 501-508.
100. S. Nikitidis, A. Tefas, N. Nikolaidis, and I. Pitas, "Facial expression recognition using clustering discriminant non-negative matrix factorization," in Proc. IEEE Int. Conf. Image Process., 2011, pp. 3001-3004.
101. S. Nikitidis, A. Tefas, N. Nikolaidis, and I. Pitas, "Subclass discriminant nonnegative matrix factorization for facial image analysis," Pattern Recognit., vol. 45, no. 12, pp. 4080-4091, 2012.
102. V. Ojansivu and J. Heikkilä, "Blur insensitive texture classification using local phase quantization," in Proc. Int. Conf. Image Signal Process., 2008, pp. 236-243.
103. C. Orrite, A. Gan, and G. Rogez, "HOG-based decision tree for facial expression classification," in Pattern Recognition and Image Analysis (ser. Lecture Notes in Computer Science), vol. 5524. Berlin, Germany: Springer, 2009, pp. 176-183.
104. M. Pantic and I. Patras, "Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences," IEEE Trans. Systems, Man Cybern. B, Cybern., vol. 36, no. 2, pp. 433-449, Apr. 2006.
105. M. Pantic, M. Valstar, R. Rademaker, and L. Maat, "Web-based database for facial expression analysis," in Proc. IEEE Int.'l Conf. Multimedia Expo., 2005, p. 5.
106. S. Park, G. Mohammadi, R. Artstein, and L.-P. Morency, "Crowdsourcing micro-level multimedia annotations: The challenges of evaluation and interface," in Proc. ACM Multimedia Workshops, 2012, pp. 29-34.
107. I. Patras and M. Pantic, "Particle filtering with factorized likelihoods for tracking facial features," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2004, pp. 97-102.
108. N. Pinto, D. D. Cox, and J. J. DiCarlo, "Why is real-world visual object recognition hard?" PLoS Comput. Biol., vol. 4, no. 1, p. e27, 2008.
109. S. J. Prince, J. Warrell, J. H. Elder, and F. M. Felisberti, "Tied factor analysis for face recognition across large pose differences," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 6, pp. 970-984, Jun. 2008.
110. R. Ptucha and A. Savakis, "Facial expression recognition using facial features and manifold learning," Adv. Vis. Comput., vol. 6455, pp. 301-309, 2010.
111. M. Ranzato, J. Susskind, V. Mnih, and G. Hinton, "On deep generative models with applications to recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 2857-2864.
112. S. Rifai, Y. Bengio, A. Courville, P. Vincent, and M. Mirza, "Disentangling factors of variation for facial expression recognition," in Proc. Eur. Conf. Comput. Vis., 2012, pp. 808-822.
113. B. Romera-Paredes, A. Argyriou, N. Berthouze, and M. Pontil, "Exploiting unrelated tasks in multi-task learning," in Int. Conf. Artif. Intell. Statist., 2012, pp. 951-959.
114. R. Rubinstein, A. Bruckstein, and M. Elad, "Dictionaries for sparse representation modeling," Proc. IEEE, vol. 98, no. 6, pp. 1045-1057, Jun. 2010.
115. O. Rudovic, V. Pavlovic, and M. Pantic, "Multi-output Laplacian dynamic ordinal regression for facial expression recognition and intensity estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 2634-2641.
116. G. Sandbach, S. Zafeiriou, and M. Pantic, "Markov random field structures for facial action unit intensity estimation," in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2013, pp. 738-745.
117. G. Sandbach, S. Zafeiriou, M. Pantic, and D. Rueckert, "Recognition of 3D facial expression dynamics," Image Vis. Comput., vol. 30, no. 10, pp. 762-773, 2012.
118. G. Sandbach, S. Zafeiriou, M. Pantic, and L. Yin, "Static and dynamic 3D facial expression recognition: A comprehensive survey," Image Vis. Comput., vol. 30, no. 10, pp. 683-697, 2012.
119. J. Saragih, S. Lucey, and J. Cohn, "Face alignment through subspace constrained mean-shifts," in Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 1034-1041.
120. E. Sariyanidi, V. Dagli, S. C. Tek, B. Tunc, and M. Gökmen, "Local Zernike Moments: A new representation for face recognition," in Proc. IEEE Int. Conf. Image Process., 2012, pp. 585-588.
121. E. Sariyanidi, H. Gunes, M. Gökmen, and A. Cavallaro, "Local Zernike moment representations for facial affect recognition," in Proc. British Machine Vision Conf., 2013, pp. 108.1-108.13.
122. A. Savran, N. Alyüz, H. Dibeklioglu, O. Çeliktutan, B. Gökberk, B. Sankur, and L. Akarun, "Bosphorus database for 3D face analysis," in Biometrics and Identity Management. Berlin, Germany: Springer, 2008, pp. 47-56.
123. A. Savran, H. Cao, M. Shah, A. Nenkova, and R. Verma, "Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering," in Proc. ACM Int. Conf. Multimodal Interaction, 2012, pp. 485-492.
124. A. Savran, B. Sankur, and M. Taha Bilge, "Comparative evaluation of 3D vs. 2D modality for automatic detection of facial action units," Pattern Recognit., vol. 45, no. 2, pp. 767-782, 2012.
125. A. Savran, B. Sankur, and M. Taha Bilge, "Regression-based intensity estimation of facial action units," Image Vis. Comput., vol. 30, no. 10, pp. 774-784, 2012.
126. B. Schuller, M. Valstar, R. Cowie, and M. Pantic, "AVEC 2012-The continuous audio/visual emotion challenge," in Proc. ACM Int. Conf. Multimodal Interaction, 2012, pp. 361-362.
127. B. Schuller, M. Valstar, F. Eyben, G. McKeown, R. Cowie, and M. Pantic, "AVEC 2011-The first international audio/visual emotion challenge," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 415-424.
128. T. Senechal, V. Rapp, H. Salam, R. Seguier, K. Bailly, and L. Prevost, "Facial action recognition combining heterogeneous features via multikernel learning," IEEE Trans. Systems, Man, Cybern. B, Cybern., vol. 42, no. 4, pp. 993-1005, Aug. 2012.
129. T. Senechal, V. Rapp, and L. Prevost, "Facial feature tracking for emotional dynamic analysis," in Adv. Concepts Intell. Vis. Syst., vol. 6915, pp. 495-506, 2011.
130. T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio, "Robust object recognition with cortex-like mechanisms," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 3, pp. 411-426, Mar. 2007.
131. T. Sha, M. Song, J. Bu, C. Chen, and D. Tao, "Feature level analysis for 3D facial expression recognition," Neurocomputing, vol. 74, no. 12, pp. 2135-2141, 2011.
132. C. Shan, S. Gong, and P. W. McOwan, "Facial expression recognition based on local binary patterns: A comprehensive study," Image Vis. Comput., vol. 27, no. 6, pp. 803-816, 2009.
133. C. Shan, S. Gong, and P. McOwan, "Appearance manifold of facial expression," in Proc. Int. Conf. Comput. Vis. Human-Comput. Interaction, 2005, vol. 3766, pp. 221-230.
134. C. Shan, S. Gong, and P. McOwan, "A comprehensive empirical study on linear subspace methods for facial expression analysis," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2006, p. 153.
135. K. Sikka, T. Wu, J. Susskind, and M. Bartlett, "Exploring bag of words architectures in the facial expression domain," in Proc. Eur. Conf. Comput. Vis. Workshops Demonstrations, 2012, pp. 250-259.
136. T. Simon, M. H. Nguyen, F. De la Torre, and J. Cohn, "Action unit detection with segment-based SVMs," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2010, pp. 2737-2744.
137. I. Sneddon, M. McRorie, G. McKeown, and J. Hanratty, "The Belfast induced natural emotion database," IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 32-41, Jan.-Mar. 2012.
138. G. Stratou, A. Ghosh, P. Debevec, and L. Morency, "Effect of illumination on automatic expression recognition: A novel 3D relightable facial database," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2011, pp. 611-618.
139. J. W. Tanaka and I. Gordon, "Features, configuration, and holistic face processing," in Oxford Handbook of Face Perception. Oxford, U.K.: Oxford Univ. Press, 2011, p. 177.
140. H. Tang and T. Huang, "3D facial expression recognition based on properties of line segments connecting facial feature points," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., Sep. 2008, pp. 1-6.
141. G. Taylor, R. Fergus, Y. LeCun, and C. Bregler, "Convolutional learning of spatio-temporal features," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 140-153.
142. M. R. Teague, "Image analysis via the general theory of moments," J. Opt. Soc. Amer., vol. 70, no. 8, pp. 920-930, 1980.
143. The MPLab GENKI Database. [Online]. Available: http://mplab. ucsd.edu, 2009.
144. Y.-L. Tian, T. Kanade, and J. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 97-115, Feb. 2001.
145. Y. Tong, J. Chen, and Q. Ji, "A unified probabilistic framework for spontaneous facial action modeling and understanding," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 2, pp. 258-273, Feb. 2010.
146. Y. Tong, W. Liao, and Q. Ji, "Facial action unit recognition by exploiting their dynamic and semantic relationships," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 10, pp. 1683-1699, Oct. 2007.
147. H. J. Towner, "Analysis of feature points and physiological data for facial expression inference," Ph.D. dissertation, University College London, 2007.
148. B. Tunç, V. Dagl, and M. Gökmen, "Class dependent factor analysis and its application to face recognition," Pattern Recognit., vol. 45, no. 12, pp. 4092-4102, 2012.
149. M. Turk and A. Pentland, "Eigenfaces for recognition," J. Cognitive Neurosci., vol. 3, no. 1, pp. 71-86, 1991.
150. G. Tzimiropoulos, V. Argyriou, S. Zafeiriou, and T. Stathaki, "Robust FFT-based scale-invariant image registration with image gradients," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 10, pp. 1899-1906, Oct. 2010.
151. M. Valstar, H. Gunes, and M. Pantic, "How to distinguish posed from spontaneous smiles using geometric features," in Proc. ACM Int. Conf. Multimodal Interfaces, 2007, pp. 38-45.
152. M. Valstar, B. Martinez, X. Binefa, and M. Pantic, "Facial point detection using boosted regression and graph models," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2010, pp. 2729-2736.
153. M. Valstar and M. Pantic, "Induced disgust, happiness and surprise: An addition to the MMI facial expression database," in Proc. Int. Conf. Language Resources Eval. Workshop Emotion, 2010, pp. 65-70.
154. M. Valstar and M. Pantic, "Fully automatic recognition of the temporal phases of facial actions," IEEE Trans. Systems, Man, Cybern. B, Cybern., vol. 42, no. 1, pp. 28-43, Feb. 2012.
155. M. Valstar, M. Pantic, Z. Ambadar, and J. Cohn, "Spontaneous vs. posed facial behavior: Automatic analysis of brow actions," in Proc. ACM Int. Conf. Multimodal Interfaces, 2006, pp. 162-170.
156. M. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, "The first facial expression recognition and analysis challenge," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2011, pp. 921-926.
157. M. Valstar and M. Pantic, "Combined support vector machines and hidden Markov models for modeling facial action temporal dynamics," in Proc. Int. Conf. Human-Comput. Interaction, 2007, vol. 4796, pp. 118-127.
158. M. Valstar, B. Schuller, K. Smith, F. Eyben, B. Jiang, S. Bilakhia, S. Schnieder, R. Cowie, and M. Pantic, "AVEC 2013-The continuous audio/visual emotion and depression recognition challenge," in Proc. ACM Int. Conf. Multimodal Interfaces, 2013, pp. 3-10.
159. M. O. A. Vasilescu and D. Terzopoulos, "Multilinear analysis of image ensembles: Tensorfaces," in Proc. Eur. Conf. Comput. Vis., 2002, pp. 447-460.
160. A. Vedaldi and B. Fulkerson. (2008). VLFeat: An open and portable library of computer vision algorithms. [Online]. Available: http://www.vlfeat.org/
161. P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., vol. 1, 2001, pp. 511-518.
162. D. Vukadinovic and M. Pantic, "Fully automatic facial feature point detection using Gabor feature based boosted classifiers," in IEEE Int. Conf. Syst., Man Cybern., 2005, vol. 2, pp. 1692-1698.
163. Z. Wang, Y. Li, S. Wang, and Q. Ji, "Capturing global semantic relationships for facial action unit recognition," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 3304-3311.
164. J. Westermeyer, "A social interactional theory of emotions," Amer. J. Psychiatry, vol. 136, no. 6, pp. 870-870, 1979.
165. M. White, "Parts and wholes in expression recognition," Cognition Emotion, vol. 14, no. 1, pp. 39-60, 2000.
166. L. Wiskott, J.-M. Fellous, N. Kuiger, and C. von der Malsburg, "Face recognition by elastic bunch graph matching," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 7, pp. 775-779, Jul. 1997.
167. T. Wu, M. Bartlett, and J. Movellan, "Facial expression recognition using Gabor motion energy filters," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2010, pp. 42-47.
168. T. Wu, N. Butko, P. Ruvolo, J. Whitehill, M. Bartlett, and J. R. Movellan, "Action unit recognition transfer across datasets," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2011, pp. 889-896.
169. X. Xiong and F. De la Torre, "Supervised descent method and its applications to face alignment," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 532-539.
170. H. Yang and I. Patras, "Sieving regression forest votes for facial feature detection in the wild," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1936-1943.
171. P. Yang, Q. Liu, and D. Metaxas, "Similarity features for facial event analysis," in Proc. Eur. Conf. Comput. Vis., 2008, vol. 5302, pp. 685-696.
172. P. Yang, Q. Liu, and D. N. Metaxas, "Boosting encoded dynamic features for facial expression recognition," Pattern Recognit. Lett., vol. 30, no. 2, pp. 132-139, 2009.
173. P. Yang, Q. Liu, and D. N. Metaxas, "Dynamic soft encoded patterns for facial event analysis," Comput. Vis. Image Understanding, vol. 115, no. 3, pp. 456-465, 2011.
174. P. Yang, Q. Liu, and D. Metaxas, "Boosting coded dynamic features for facial action units and facial expression recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2007, pp. 1-6.
175. S. Yang and B. Bhanu, "Facial expression recognition using emotion avatar image," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2011, pp. 866-871.
176. L. Yin, X. Wei, Y. Sun, J. Wang, and M. Rosato, "A 3D facial expression database for facial behavior research," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2006, pp. 211-216.
177. S. Zafeiriou and M. Petrou, "Sparse representations for facial expressions recognition via l1 optimization," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2010, pp. 32-39.
178. N. Zaker, M. Mahoor, W. Mattson, D. Messinger, and J. Cohn, "Intensity measurement of spontaneous facial actions: Evaluation of different image representations," in Proc. IEEE Int. Conf. Dev. Learn. Epigenetic Robot., 2012, pp. 1-2.
179. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
180. D. Zhang and G. Lu, "Review of shape representation and description techniques," Pattern Recognit., vol. 37, no. 1, pp. 1-19, 2004.
181. L. Zhang, Y. Tong, and Q. Ji, "Active image labeling and its application to facial action labeling," in Proc. Eur. Conf. Comput. Vis., 2008, pp. 706-719.
182. L. Zhang and D. Tjondronegoro, "Facial expression recognition using facial movement features," IEEE Trans. Affective Comput., vol. 2, no. 4, pp. 219-229, Oct.-Dec. 2011.
183. L. Zhang and G. W. Cottrell, "When holistic processing is not enough: Local features save the day," in Proc. Annu. Conf. Cognitive Sci. Soc., 2004, pp. 1506-1511.
184. S. Zhang, Y. Zhan, M. Dewan, J. Huang, D. N. Metaxas, and X. S. Zhou, "Towards robust and effective shape modeling: Sparse shape composition," Med. Image Anal., vol. 16, no. 1, pp. 265-277, 2012.
185. G. Zhao and M. Pietikäinen, "Dynamic texture recognition using local binary patterns with an application to facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 915-928, Jun. 2007.
186. G. Zhao and M. Pietikäinen1, "Boosted multi-resolution spatiotemporal descriptors for facial expression recognition," Pattern Recognit. Lett., vol. 30, no. 12, pp. 1117-1127, 2009.
187. Q. Zhao, D. Zhang, and H. Lu, "Supervised LLE in ICA space for facial expression recognition," in Proc. Int. Conf. Neural Netw. Brain, vol. 3, 2005, pp. 1970-1975.
188. R. Zhi, M. Flierl, Q. Ruan, and W. Kleijn, "Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition," IEEE Trans. Systems, Man, Cybern. B, Cybern., vol. 41, no. 1, pp. 38-52, Feb. 2011.
189. L. Zhong, Q. Liu, P. Yang, B. Liu, J. Huang, and D. Metaxas, "Learning active facial patches for expression analysis," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 2562-2569.
190. X. Zhu and D. Ramanan, "Face detection, pose estimation, and landmark localization in the wild," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 2879-2886.
191. Y. Zhu, F. De la Torre, J. Cohn, and Y.-J. Zhang, "Dynamic cascades with bidirectional bootstrapping for action unit detection in spontaneous facial behavior," IEEE Trans. Affective Comput., vol. 2, no. 2, pp. 79-91, Apr.-Jun. 2011.