Meta-analysis of the first facial expression recognition challenge

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 42, Issue 4, 2012, Pages 966-979

  Valstar, Michel F ^a,b   Mehu, Marc ^c   Jiang, Bihan ^a   Pantic, Maja ^a,d   Scherer, Klaus ^c  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^c UNIVERSITY OF GENEVA   (Switzerland)

^d UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

Challenges; discrete emotion recognition; facial action coding system (FACS) analysis; facial expression analysis

Indexed keywords

ACTION UNITS; AUTOMATIC FACIAL EXPRESSION RECOGNITION; AUTOMATIC RECOGNITION; CHALLENGES; EMOTION RECOGNITION; EVALUATION PROTOCOL; FACE AND GESTURE RECOGNITION; FACIAL ACTION CODING SYSTEM; FACIAL EXPRESSION ANALYSIS; FACIAL EXPRESSION RECOGNITION; FACIAL EXPRESSIONS; LEVEL PLAYING FIELDS; META-ANALYSIS; POSSIBLE FUTURES;

COMPUTER KEYBOARDS; GESTURE RECOGNITION;

FACE RECOGNITION;

EID: 84864118983     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2012.2200675     Document Type: Article

References (69)

1. T. Ahonen, A. Hadid, and M. Pietikäinene, "Face description with local binary patterns: Application to face recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 12, pp. 2037-2041, Dec. 2006. (Pubitemid 46405714)
2. N. Ambady and R. Rosenthal, "Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis," Psychol. Bull., vol. 11, no. 2, pp. 256-274, Mar. 1992.
3. A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, Z. Ambadar, K. M. Prkachin, and P. E. Solomon, "The painful face-Pain expression recognition using active appearance models," Image Vis. Comput., vol. 27, no. 12, pp. 1788- 1796, Nov. 2009.
4. A. Asthana, J. Saragih, M. Wagner, and R. Goecke, "Evaulating aam fitting methods for facial expression recognition," in Proc. Int. Conf. Affective Comput. Intell. Interact., 2009, pp. 1-8.
5. T. Baltrusaitis, D. McDuff, N. Banda, M. Mahmoud, R. El Kaliouby, P. Robinson, and R. Picard, "Real-time inference of mental states from facial expressions and upper body gestures," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 909-914.
6. T. Bänziger and K. R. Scherer, "Introducing the geneva multimodal emotion portrayal (gemep) corpus," in Blueprint for Affective Computing: A Sourcebook, K. R. Scherer, T. Bänziger, and E. B. Roesch, Eds. Oxford, U.K.: Oxford Univ. Press, 2010, ser. Series in Affective Science, ch. 6.1, pp. 271-294.
7. M. Bartlett, G. Littlewort-Ford, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Fully automatic facial action recognition in spontaneous behaviour," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2006, pp. 223-230. (Pubitemid 44711354)
8. S. W. Chew, P. Lucey, S. Lucey, J. Saragih, J. Cohn, and S. Sridharan, "Person-independent facial expression detection using constrained local models," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 915-920.
9. J. Cohn and P. Ekman, "Measuring facial action by manual coding, facial emg, and automatic facial image analysis," in Handbook of Nonverbal Behavior Research Methods in the Affective Sciences, R. R. K. S. J. A. Harrigan, Ed. New York: Oxford Univ. Press, 2005, pp. 9-64.
10. J. F. Cohn, "Foundations of human computing: Facial expression and emotion," in Proc. ACM Int.l Conf. Multimodal Interf., 2006, vol. 1, pp. 610-616.
11. D. Cunningham, M. Kleiner, C.Wallraven, and H. Bülthoff, "The components of conversational facial expressions," in Proc. ACM Int. Symp. Appl. Perception Graph. Vis., 2004, pp. 143-149. (Pubitemid 40290754)
12. M. Dahmane and J. Meunier, "Emotion recognition using dynamic gridbased hog features," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 884-888.
13. A. C. de C. Williams, "Facial expression of pain: An evolutionary account," Behav. Brain Sci., vol. 25, no. 4, pp. 439-488, Aug. 2002.
14. A. Dhall, A. Asthana, R. Goecke, and T. Gedeon, "Emotion recognition using phog and lpq features," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 878-883.
15. P. Ekman, Face of Man: Universal Expression in a New Guinea Village. New York: Garland, 1982.
16. P. Ekman, "An argument for basic emotions," Cog. Emotion, vol. 6, no. 3/4, pp. 169-200, May 1992.
17. P. Ekman and W. Friesen, "The repertoire of nonverbal behavioral categories-Origins, usage and coding," Semiotica, vol. 1, no. 1, pp. 49- 98, 1969.
18. P. Ekman,W. V. Friesen, and J. C. Hager, FACS Manual. Salt Lake City, UT: A Human Face, May 2002.
19. R. El Kaliouby and P. Robinson, "Generalization of a vision-based computational model of mind-reading," in Proc. 1st Int. Conf. Affective Comput. Intell. Interact., 2005, pp. 582-589. (Pubitemid 43763378)
20. B. Fasel and J. Luettin, "Automatic facial expression analysis: A survey," Pattern Recognit., vol. 36, no. 1, pp. 259-275, Jan. 2003.
21. T. Gehrig and H. Ekenel, "Facial action unit detection using kernel partial least squares," in Proc. IEEE ICCV, 2011, pp. 2092-2099.
22. H. Gunes and M. Pantic, "Automatic, dimensional and continuous emotion recognition," Int. J. Synthetic Emotions, vol. 1, no. 1, pp. 68-99, Jan./Jun. 2010.
23. H. Gunes, B. Schuller,M. Pantic, and R. Cowie, "Emotion representation, analysis and synthesis in continuous space: A survey," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2011, pp. 827-834.
24. B. Jiang, M. Valstar, and M. Pantic, "Action unit detection using sparse appearance descriptors in space-time video volumes," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2011, pp. 314-321.
25. T. Kanade, "Picture processing system by computer complex and recognition of human faces," Ph.D. dissertation, Tokyo Univ., Tokyo, Japan, 1973.
26. T. Kanade, J. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in Proc. 4th IEEE Int. Conf. Autom. Face Gesture Recog., 2000, pp. 46-53.
27. S. Koelstra, M. Pantic, and I. Patras, "A dynamic texture based approach to recognition of facial actions and their temporal models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 1940-1954, Nov. 2010.
28. I. Kotsia and I. Pitas, "Facial expression recognition in image sequences using geometric deformation features and support vector machines," IEEE Trans. Image Process., vol. 16, no. 1, pp. 172-187, Jan. 2007. (Pubitemid 46437480)
29. F. D. la Torre and J. F. Cohn, "Facial expression analysis," in Guide to Visual Analysis of Humans: Looking at People, V. K. T. B. Moeslund, A. Hilton, and L. Sigal, Eds. New York: Springer-Verlag, 2011.
30. G. Littlewort, J. Whitehill, T. Wu, N. Butko, P. Ruvolo, J. Movellan, and M. Bartlett, "The motion in emotion-a cert based approach to the fera emotion challenge," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 897-902.
31. G. Littlewort, J. Whitehill, T. Wu, I. Fasel, M. Frank, J. Movellan, and M. Bartlett, "The computer expression recognition toolbox (cert)," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2011, pp. 298-305.
32. C. Liu, J. Yuen, and A. Torralba, "Sift flow: Dense correspondence across scenes and its applications," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 978-994, May 2010.
33. M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, "Coding facial expressions with gabor wavelets," in Proc. 3rd IEEE Int. Conf. Autom. Face Gesture Recog., 1998, pp. 200-205.
34. H. Meng, B. Romera-Paredes, and N. Berthouze, "Emotion recognition by two view SVM2k classifier on dynamic facial expression features," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 854-859.
35. S. Moore and R. Bowden, "Local binary patterns for multi-view facial expression recognition," Comput. Vis. Image Understand., vol. 115, no. 4, pp. 541-558, Apr. 2011.
36. T. Ojala, M. Pietikainen, and D. Harwood, "A comparative study of texture measures with classification based on featured distribution," Pattern Recognit., vol. 29, no. 1, pp. 51-59, Jan. 1996.
37. T. Ojala, M. Pietikainen, and T. Maenpaa, "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971-987, Jul. 2002. (Pubitemid 34835471)
38. M. Pantic and M. Bartlett, "Machine analysis of facial expressions," in Face Recognition, K. Delac and M. Grgic, Eds. Vienna, Austria: I-Tech Educ. Publ., 2007, pp. 377-416.
39. M. Pantic, A. Nijholt, A. Pentland, and T. Huang, "Human-centred intelligent human-computer interaction (hci2): How far are we from attaining it?" Int. J. Auton. Adapt. Commun. Syst., vol. 1, no. 2, pp. 168-187, Aug. 2008.
40. M. Pantic and L. Rothkrantz, "Automatic analysis of facial expressions: The state of the art," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 12, pp. 1424-1445, Dec. 2000. (Pubitemid 32088469)
41. M. Pantic and L. J. M. Rothkrantz, "Toward an affect-sensitive multimodal human-computer interaction," Proc. IEEE, vol. 91, no. 9, pp. 1370-1390, Sep. 2003. (Pubitemid 40890819)
42. M. Pantic, M. F. Valstar, R. Rademaker, and L. Maat, "Web-based database for facial expression analysis," in Proc. Int. Conf. Multimedia Expo., 2005, pp. 317-321. (Pubitemid 44668868)
43. A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet, "Simplemkl," J. Mach. Learn. Res., vol. 9, pp. 2491-2521, 2008.
44. A. Samal and P. A. Iyengar, "Automatic recognition and analysis of human faces and facial expressions: A survey," Pattern Recognit., vol. 25, no. 1, pp. 65-77, Jan. 1992.
45. J. Saragih, S. Lucey, and J. Cohn, "Face alignment through subspace constrained mean-shifts," in Proc. Int. Conf. Comput. Vis., 2009, pp. 1034-1041.
46. B. Schuller, M. Valstar, F. Eyben, G. McKeown, R. Cowie, and M. Pantic, "Avec 2011 the first international audio/visual emotion challenge," in Proc. ACII, 2011, pp. 415-424.
47. N. Sebe, M. Lew, I. Cohen, T. Gevers, and T. Huang, "Authentic facial expression analysis," Image Vis. Comput., vol. 25, no. 12, pp. 1856-1863, Dec. 2007. (Pubitemid 47333322)
48. T. Senechal, V. Rapp, L. Prevost, H. Salam, R. Seguier, and K. Bailly, "Combining lgbp histograms with aam coefficients in the multi-kernel svm framework to detect facial action units," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 860-865.
49. C. Shan, S. Gong, and P. McOwan, "Facial expression recognition based on local binary patterns: A comprehensive study," Image Vis. Comput. J., vol. 27, no. 6, pp. 803-816, 2009.
50. T. Simon, M. H. Nguyen, F. De La Torre, and J. Cohn, "Action unit detection with segment-based SVMS," in Proc. IEEE Conf. CVPR, Jun. 2010, pp. 2737-2744.
51. R. Srivastava, S. Roy, S. Yan, and T. Sim, "Accumulated motion images for facial expression recognition in videos," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 903-908.
52. J. Sung and D. Kim, "Real-time facial expression recognition using staam and layered gda classifier," Image Vis. Comput., vol. 27, no. 9, pp. 1313-1325, Aug. 2009.
53. U. Tariq, K.-H. Lin, Z. Li, X. Zhou, Z. Wang, V. Le, T. Huang, X. Lv, and T. Han, "Emotion recognition from an ensemble of features," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 872-877.
54. Y. Tian, T. Kanade, and J. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 97-115, Feb. 2001. (Pubitemid 32254704)
55. Y. L. Tian, T. Kanade, and J. F. Cohn, Handbook of Face Recognition. New York: Springer-Verlag, 2005.
56. Y. Tong, J. Chen, and Q. Ji, "A unified probabilistic framework for spontaneous facial action modeling and understanding," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 2, pp. 258-273, Dec. 2010.
57. M. Valstar andM. Pantic, "Combined support vector machines and hidden Markov models for modeling facial action temporal dynamics," in Proc. ICCV-HCI, 2007, pp. 118-127.
58. M. Valstar and M. Pantic, "Induced disgust, happiness and surprise: An addition to the mmi facial expression database," in Proc. 3rd Int. Workshop EMOTION, 2010, pp. 65-70, (Satellite of LREC): Corpora for Research on Emotion and Affect.
59. M. Valstar and M. Pantic, "Fully automatic recognition of the temporal phases of facial actions," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 42, no. 1, pp. 28-43, Feb. 2012.
60. M. F. Valstar and M. Pantic, "Biologically vs. logic inspired encoding of facial actions and emotions in video," in Proc. IEEE Int. Conf.Multimedia Expo., 2006, pp. 325-328. (Pubitemid 46679720)
61. P. Viola and M. Jones, "Robust real-time object detection," Int. J. Comput. Vis., vol. 57, no. 2, pp. 137-154, 2002.
62. J. Whitehill and C. Omlin, "Haar features for facs au recognition," in Proc. Int'l. Conf. Automatic Face and Gesture Recognition, 2006, pp. 97-101. (Pubitemid 44711334)
63. T. Wu, N. Butko, P. Ruvolo, J. Whitehill, M. Bartlett, and J. Movellan, "Action unit recognition transfer across datasets," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 889-896.
64. S. Yang and B. Bhanu, "Facial expression recognition using emotion avatar image," in Proc. IEEE Int. Conf. Autom. Face Gesture Anal., 2011, pp. 866-871.
65. S. Zafeiriou and M. Petrou, "Nonlinear non-negative component analysis algorithms," IEEE Trans. Image Process., vol. 19, no. 4, pp. 1050-1066, Apr. 2010.
66. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
67. G. Zhao and M. Pietikainen, "Dynamic texture recognition using local binary pattern with an application to facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 915-928, Jun. 2007. (Pubitemid 46667407)
68. R. Zhi, M. Flierl, Q. Ruan, and W. Kleijn, "Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 1, pp. 38-52, Feb. 2011.
69. Y. Zhu, F. De la Torre, J. Cohn, and Y. Zhang, "Dynamic cascades with bidirectional bootstrapping for action unit detection in spontaneous facial behavior," IEEE Trans. Affective Comput., vol. 2, no. 2, pp. 79-91, Apr.-Jun. 2011.