Fully automatic recognition of the temporal phases of facial actions

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 42, Issue 1, 2012, Pages 28-43

  Valstar, Michel F ^a   Pantic, Maja ^a,b  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

Facial expression analysis; GentleBoost; particle filtering; spatiotemporal facial behavior analysis; support vector machine (SVM)

Indexed keywords

FACIAL EXPRESSION ANALYSIS; GENTLEBOOST; PARTICLE FILTERING; SPATIOTEMPORAL FACIAL BEHAVIOR ANALYSIS; SUPPORT VECTOR;

HIDDEN MARKOV MODELS; SIGNAL FILTERING AND PREDICTION; SUPPORT VECTOR MACHINES;

FACE RECOGNITION;

ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; COMPUTER ASSISTED DIAGNOSIS; FACIAL EXPRESSION; HUMAN; IMAGE SUBTRACTION; METHODOLOGY; PHOTOGRAPHY; VIDEORECORDING;

ARTIFICIAL INTELLIGENCE; FACIAL EXPRESSION; HUMANS; IMAGE INTERPRETATION, COMPUTER-ASSISTED; PATTERN RECOGNITION, AUTOMATED; PHOTOGRAPHY; SUBTRACTION TECHNIQUE; VIDEO RECORDING;

EID: 84856294719     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2011.2163710     Document Type: Article

References (87)

1. P. Ekman and E. L. Rosenberg, What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System. Oxford, U.K.: Oxford Univ. Press, 2005.
2. J. Russell and J. Fernandez-Dols, The Psychology of Facial Expression. New York: Cambridge Univ. Press, 1997.
3. B. Golomb and T. Sejnowski, "Benefits of machine understanding of facial expressions," in NSF Report-Facial Expression Understanding, P. Ekman, T. Huang, T. Sejnowski, and J. Hager, Eds. Salt Lake City, UT, 1997, pp. 55-71.
4. M. Pantic, "Face for ambient interface," in Ambient Intelligence in Everyday Life, vol. 3864, Lecture Notes on Artificial Intelligence. Berlin, Germany: Springer-Verlag, 2006, pp. 32-66.
5. M. Cohen, Perspectives on the Face. Oxford, U.K.: Oxford Univ. Press, 2006.
6. A. Young, Face and Mind. Oxford, U.K.: Oxford Univ. Press, 1998.
7. M. Pantic and L. J. M. Rothkrantz, "Toward an affect-sensitive mul-timodal human-computer interaction," Proc. IEEE, vol. 91, no. 9, pp. 1370-1390, Sep. 2003. (Pubitemid 40890819)
8. Z. Zeng, M. Pantic, G. Roisman, and T. Huang, "A survey of affect recognition methods: Audio, visual and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
9. Y. L. Tian, T. Kanade, and J. F. Cohn, Handbook of Face Recognition. New York: Springer-Verlag, 2005.
10. M. Pantic and M. Bartlett, "Machine analysis of facial expressions," in Face Recognition, K. Delac and M. Grgic, Eds. Vienna, Austria: I-Tech Educ. Publishing, 2007, pp. 377-416.
11. J. F. Cohn, "Foundations of human computing: Facial expression and emotion," in Proc. ACM Int. Conf. Multimodal Interfaces, 2006, vol. 1, pp. 610-616.
12. P. Ekman, Face of Man: Universal Expression in a New Guinea Village. New York: Garland, 1982.
13. W.-F. Liu, J.-L. Lu, Z.-F. Wang, and H.-J. Song, "An expression space model for facial expression analysis," in Proc. CISP, May 2008, vol. 2, pp. 680-684.
14. K. Anderson and P. McOwan, "A real-time automated system for recognition of human facial expressions," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 36, no. 1, pp. 96-105, Feb. 2006. (Pubitemid 43174144)
15. S. Park, J. Shin, and D. Kim, "Facial expression analysis with facial expression deformation," in Proc. 19th ICPR, Dec. 2008, pp. 1-4.
16. I. Kotsia and I. Pitas, "Facial expression recognition in image sequences using geometric deformation features and support vector machines," IEEE Trans. Image Process., vol. 16, no. 1, pp. 172-187, Jan. 2007. (Pubitemid 46437480)
17. R. E. Kaliouby and P. Robinson, "Real-time inference of complex mental states from facial expressions and head gestures," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2004, vol. 3, p. 154.
18. A. Ashraf, S. Lucey, T. Chen, J. Cohn, and K. Prkachin, "The painful face-Pain expression recognition using active appearance models," in Proc. Int. Conf. Multimedia Interfaces, 2007, pp. 9-14.
19. G. Littlewort, M. Bartlett, and K. Lee, "Faces of pain: Automated measurement of spontaneous facial expressions of genuine and posed pain," in Proc. Int. Conf. Multimodal Interfaces, 2007, pp. 15-21.
20. H. Gu and Q. Ji, "Information extraction from image sequences of real-world facial expressions," Mach. Vis. Appl., vol. 16, no. 2, pp. 105-115, Feb. 2005. (Pubitemid 40536233)
21. J. Cohn and P. Ekman, "Measuring facial action by manual coding, facial EMG, and automatic facial image analysis," in Handbook of Nonverbal Behavior Research Methods in the Affective Sciences, J. A. Harrigan, R. Rosenthal, and K. Scherer, Eds. New York: Oxford Univ. Press, 2005, pp. 9-64.
22. P. Ekman, W. V. Friesen, and J. C. Hager, Facial Action Coding System. Salt Lake City, UT: A Human Face, 2002.
23. D. Cunningham, M. Kleiner, C. Wallraven, and H. Bülthoff, "The components of conversational facial expressions," in Proc. ACM Int. Symp. Appl. Perception Graph. Vis., 2004, pp. 143-149. (Pubitemid 40290754)
24. A. C. de C. Williams, "Facial expression of pain: An evolutionary account," Behav. Brain Sci., vol. 25, no. 4, pp. 439-488, Aug. 2002.
25. P. Ekman and W. Friesen, "The repertoire of nonverbal behavioral categories-Origins, usage and coding," Semiotica, vol. 1, pp. 49-98, 1969.
26. M. F. Valstar and M. Pantic, "Biologically vs. logic inspired encoding of facial actions and emotions in video," in Proc. IEEE Int. Conf. Multimedia Expo, 2006, pp. 325-328. (Pubitemid 46679720)
27. M. S. Bartlett, P. Viola, T. Sejnowski, B. Golomb, J. Larsen, J. Hager, and P. Ekman, "Classifying facial actions," in Proc. Adv. Neural Inf. Process. Syst., 1996, vol. 8, pp. 823-829.
28. J. Lien, T. Kanade, J. Cohn, and C. Li, "Subtly different facial expression recognition and expression intensity estimation," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 1998, pp. 853-859.
29. M. Pantic, L. J. M. Rothkrantz, and H. Koppelaar, "Automation of nonverbal communication of facial expressions," in Proc. Conf. Euromedia, 1998, pp. 86-93.
30. M. S. Bartlett, J. C. Hager, P. Ekman, and T. J. Sejnowski, "Measuring facial expressions by computer image analysis," Psychophysiology, vol. 36, no. 2, pp. 253-263, Mar. 1999. (Pubitemid 29130001)
31. Y. Tian, T. Kanade, and J. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 97-115, Feb. 2001. (Pubitemid 32254704)
32. M. Pantic and L. Rothkrantz, "Facial action recognition for facial expression analysis from static face images," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 34, no. 3, pp. 1449-1461, Jun. 2004.
33. M. Pantic and I. Patras, "Dynamics of facial expressions-Recognition of facial actions and their temporal segments from face profile image sequences," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 36, no. 2, pp. 433-449, Apr. 2006.
34. J. F. Cohn, J. F. Reed, Z. Ambadar, J. Xiao, and T. Moriyama, "Automatic analysis and recognition of brow actions in spontaneous facial behavior," in Proc. IEEE Int. Conf. Syst., Man, Cybern., 2004, vol. 1, pp. 610-616. (Pubitemid 40415926)
35. M. S. Bartlett, G. Littlewort, M. G. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Fully automatic facial action recognition in spontaneous behavior," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2006, pp. 223-230. (Pubitemid 44711354)
36. M. F. Valstar, M. Pantic, Z. Ambadar, and J. F. Cohn, "Spontaneous vs. posed facial behavior: Automatic analysis of brow actions," in Proc. ACM Int. Conf. Multimodal Interfaces, 2006, pp. 162-170. (Pubitemid 47129473)
37. M. F. Valstar, M. Pantic, and H. Gunes, "A multimodal approach to automatic recognition of posed vs. spontaneous smiles," in Proc. ACM Int. Conf. Multimodal Interfaces, 2007, pp. 38-45.
38. Z. Ambadar, J. Schooler, and J. F. Cohn, "Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expressions," Psychol. Sci., vol. 16, no. 5, pp. 403-410, May 2005. (Pubitemid 40763688)
39. L. Gralewski, N. Campbell, and I. Voak, "Using a tensor framework for the analyisis of facial dynamics," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2006, pp. 217-222. (Pubitemid 44711353)
40. Y. Tong, W. Liao, and Q. Ji, "Facial action unit recognition by exploiting their dynamic and semantic relationships," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 10, pp. 1683-1699, Oct. 2007.
41. Y. Zhang and Q. Ji, "Active and dynamic information fusion for facial expression understanding from image sequence," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 5, pp. 699-714, May 2005. (Pubitemid 40608403)
42. G. Littlewort, M. Bartlett, I. Fasel, J. Susskind, and J. Movellan, "Dynamics of facial expression extracted automatically from video," Image Vis. Comput., vol. 24, no. 6, pp. 615-625, Jun. 2006. (Pubitemid 43867835)
43. S. Koelstra and M. Pantic, "Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2008, pp. 1-8.
44. M. Pantic and I. Patras, "Detecting facial actions and their temporal segments in nearly frontal-view face image sequences," in Proc. IEEE Int. Conf. Syst., Man, Cybern., 2005, pp. 3358-3363. (Pubitemid 41667331)
45. M. F. Valstar, I. Patras, and M. Pantic, "Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2005, vol. 3, pp. 76-84.
46. M. F. Valstar and M. Pantic, "Fully automatic facial action unit detection and temporal analysis," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2006, p. 149.
47. M. F. Valstar and M. Pantic, "Combined support vector machines and hidden Markov models for modeling facial action temporal dynamics," in Proc. IEEE Workshop Human Comput. Interaction, vol. 4796, Lecture Notes on Computer Science, 2007, pp. 118-127.
48. G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski, "Classifying facial actions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 10, pp. 974-989, Oct. 1999. (Pubitemid 129304131)
49. B. Jiang, M. Valstar, and M. Pantic, "Action unit detection using sparse appearance descriptors in space-time video volumes," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2011, pp. 314-321.
50. E. Hjelmås and B. Low, "Face detection: A survey," Comput. Vis. Image Understand., vol. 83, no. 3, pp. 236-274, 2001. (Pubitemid 33024376)
51. M. Yang, D. Kriegman, and N. Ahuja, "Detecting faces in images: A survey," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 1, pp. 34-58, Jan. 2002. (Pubitemid 34325615)
52. S. Li and A. Jain, Handbook of Face Recognition. New York: Springer-Verlag, 2005.
53. P. Viola and M. Jones, "Robust real-time object detection," Int. J. Comput. Vi s., vol. 57, no. 2, pp. 137-154, May 2004.
54. I. Fasel, B. Fortenberry, and J. Movellan, "A generative framework for real time object detection and classification," Comput. Vis. Image Understand., vol. 98, no. 1, pp. 181-210, Apr. 2005.
55. T. Kanade, J. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2000, pp. 46-53.
56. D. Vukadinovic and M. Pantic, "Fully automatic facial feature point detection using Gabor feature based boosted features," in Proc. IEEE Int. Conf. Syst., Man, Cybern., 2005, pp. 1692-1698. (Pubitemid 41667548)
57. E. Holden and R. Owens, "Automatic facial point detection," in Proc. Asian Conf. Comput. Vis., 2002, pp. 731-736.
58. L. Chen, L. Zhang, H. Zhang, and M. Abdel-Mottaleb, "3D shape constraint for facial feature localization using probabilistic-like output," in Proc. IEEE Int. Workshop Anal. Model. Faces Gestures, 2004, pp. 302-307.
59. D. Cristinacce and T. Cootes, "A comparison of shape constrained facial feature detectors," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2004, pp. 375-380.
60. D. Cristinacce and T. Cootes, "Facial feature detection and tracking with automatic template selection," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2006, pp. 429-434. (Pubitemid 44711387)
61. S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. Black, and R. Szeliski, "A database and evaluation methodology for optical flow," Int. J. Comput. Vi s., vol. 92, no. 1, pp. 1-31, Mar. 2011.
62. J. Barron, D. Fleet, and S. Beauchemin, "Performance of optical flow techniques," Int. J. Comput. Vis., vol. 12, no. 1, pp. 43-78, Feb. 1994.
63. B. McCane, K. Novins, D. Crannitch, and B. Galvin, "On benchmarking optical flow," Comput. Vis. Image Understand., vol. 84, no. 1, pp. 126-143, Oct. 2001. (Pubitemid 34220912)
64. B. Lucas and T. Kanade, "An iterative image registration technique with an application to stereo vision," in Proc. Int. Joint Conf. Artif. Intell., 1981, pp. 674-679. (Pubitemid 14478744)
65. J. Xiao, S. Baker, I. Matthews, and T. Kanade, "Real-time combined 2D+3D active appearance models," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog., 2004, vol. 2, pp. 535-542.
66. R. Kalman, "A new approach to linear filtering and prediction problems," Trans. ASME, J. Basic Eng., vol. 82, pp. 35-45, 1960.
67. C. Andrieu, A. Doucet, S. Singh, and V. Tadic, "Particle methods for change detection, system identification, and control," Proc. IEEE, vol. 92, no. 3, pp. 423-438, Mar. 2004. (Pubitemid 40890751)
68. S. Hamlaoui and F. Davoine, "Facial action tracking using particle filters and active appearance models," in Proc. IEEE Int. Conf. Face Gesture Recog., 2005, pp. 165-169.
69. J. McCall and M. Trivedi, "Facial action coding using multiple visual cues and a hierarchy of particle filters," in Proc. Int. Conf. Comput. Vis. Pattern Recog., 2006, vol. 3, p. 150.
70. I. Patras and M. Pantic, "Particle filtering with factorized likelihoods for tracking facial features," in Proc. Int. Conf. Autom. Face Gesture Recog., 2004, pp. 97-102.
71. M. K. Pitt and N. Shephard, "Filtering via simulation: Auxiliary particle filters," J. Amer. Stat. Assoc., vol. 94, no. 446, pp. 590-599, Jun. 1999.
72. I. Patras and M. Pantic, "Tracking deformable motion," in Proc. Int. Conf. Syst., Man, Cybern., 2005, pp. 1066-1071. (Pubitemid 41667447)
73. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-Based Learning. New York: Cambridge Univ. Press, 2000.
74. J. Friedman, T. Hastie, and R. Tibshirani, "Additive logistic regression: A statistical view of boosting," Ann. Statist., vol. 28, no. 2, pp. 337-374, 2000.
75. J. F. Cohn and K. L. Schmidt, "The timing of facial motion in posed and spontaneous smiles," J. Wavelets, Multi-Resolution Inf. Process., vol. 2, no. 2, pp. 121-132, 2004.
76. H. Bourlard and N. Morgan, "Hybrid HMM/ANN systems for speech recognition: Overview and new research directions," in Adaptive Processing of Sequences and Data Structures, Lecture Notes in Artificial Intelligence. New York: Springer-Verlag, 1998, pp. 389-417. (Pubitemid 128056038)
77. S. Kruger, M. Schaffner, M. Katz, E. Andelic, and A. Wendemuth, "Speech recognition with support vector machines in a hybrid system," in Proc. Interspeech, 2005, pp. 993-996. (Pubitemid 43908231)
78. J. Platt, "Probabilistic outputs for support vector machines and comparison to regularized likelihood methods," in Advances in Large Margin Classifiers, A. Smola, P. Bartlett, B. Schoelkopf, and D. Schuurmans, Eds. Cambridge, MA: MIT Press, 2000, pp. 61-74.
79. M. Pantic, M. F. Valstar, R. Rademaker, and L. Maat, "Web-based database for facial expression analysis," in Proc. Int. Conf. Multimedia Expo, 2005, pp. 317-321. (Pubitemid 44668868)
80. M. F. Valstar and M. Pantic, "Induced disgust, happiness and surprise: An addition to the MMI facial expression database," in Proc. Lang. Resour. Eval. Conf., 2010, pp. 317-321.
81. E. Rosenberg, P. Ekman, and J. Blumenthal, "Facial expression and the affective component of cynical hostility in male coronary heart disease patients," Health Psychol., vol. 17, no. 4, pp. 376-380, Jul. 1998. (Pubitemid 28345946)
82. T. Kirchner, M. Sayette, J. Cohn, R. Moreland, and J. Levine, "Effects of alcohol on group formation among male social drinkers," J. Stud. Alcohol, vol. 67, no. 5, pp. 785-794, Sep. 2006.
83. J. Cohen, "A coefficient of agreement for nominal scales," Educ. Psychol. Meas., vol. 20, no. 1, pp. 37-46, Apr. 1960.
84. R. Feris, J. Gemmell, K. Toyama, and V. Kruger, "Hierarchical wavelet networks for facial feature localization," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2002, pp. 118-123.
85. P. Ekman, "Darwin, deception, and facial expression," Ann. New York Acad. Sci., vol. 1000, pp. 105-221, 2003.
86. P. Yang, Q. Liu, and D. N. Metaxas, "Boosting encoded dynamic features for facial expression recognition," Pattern Recognit. Lett., vol. 30, no. 2, pp. 132-139, Jan. 2009.
87. J. Whitehill and C. Omlin, "Haar features for FACS AU recognition," in Proc. 7th Int. Conf. FGR, Apr. 2006, pp. 97-101. (Pubitemid 44711334)