A unified probabilistic framework for spontaneous facial action modeling and understanding

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 32, Issue 2, 2010, Pages 258-273

  Tong, Yan ^a   Chen, Jixu ^b   Ji, Qiang ^b  

^a GE GLOBAL RESEARCH   (United States)

^b RENSSELAER POLYTECHNIC INSTITUTE   (United States)

Author keywords

Bayesian networks.; Face pose estimation; Facial action analysis; Facial action coding system; Facial action unit recognition

Indexed keywords

DYNAMIC BAYESIAN NETWORK; FACE POSE ESTIMATION; FACIAL ACTION; FACIAL ACTION CODING SYSTEM; FACIAL ACTION UNIT RECOGNITION; FACIAL DEFORMATIONS; FACIAL EXPRESSION RECOGNITION; FACIAL EXPRESSIONS; FACIAL MOTIONS; HEAD MOVEMENTS; HUMAN COMMUNICATIONS; IMAGE MEASUREMENTS; MACHINE LEARNING METHODS; MODEL-BASED; MUSCULAR MOVEMENTS; PRIOR KNOWLEDGE; PROBABILISTIC FRAMEWORK; PROBABILISTIC INFERENCE; SPATIOTEMPORAL INTERACTIONS; TRAINING DATA; VISUAL MEASUREMENTS;

BAYESIAN NETWORKS; COMPUTER KEYBOARDS; DISTRIBUTED PARAMETER NETWORKS; GESTURE RECOGNITION; INFERENCE ENGINES; INTELLIGENT NETWORKS; SIGNAL ENCODING; SPEECH RECOGNITION;

FACE RECOGNITION;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; BAYES THEOREM; BIOMETRY; FACE; FACTUAL DATABASE; HISTOLOGY; HUMAN; METHODOLOGY; STATISTICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BAYES THEOREM; BIOMETRIC IDENTIFICATION; DATABASES, FACTUAL; FACE; HUMANS; MODELS, STATISTICAL;

EID: 73849113366     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2008.293     Document Type: Article

References (54)

1. M. Pantic and M. Bartlett, "Machine Analysis of Facial Expres sions," Face Recognition, K. Delac and M. Grgic, eds., pp. 377-416, I-Tech Education and Publishing, 2007.
2. P. Ekman and W.V. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, 1978.
3. K. Scherer and P. Ekman, Handbook of Methods in Nonverbal Behavior Research. Cambridge Univ. Press, 1982.
4. J.F. Cohn and K. Schmidt, "The Timing of Facial Motion in Posed and Spontaneous Smiles," Int'l J. Wavelets, Multiresolution, and Information Processing, vol.2, pp. 1-12, Mar. 2004.
5. M. Pantic and I. Patras, "Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences," IEEE Trans. Systems, Man, and Cybernetics\-Part B: Cybernetics, vol.36, no.2, pp. 433-449, Apr. 2006.
6. Y. Tong, W. Liao, and Q. Ji, "Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships," IEEE Trans. Pattern Analysis and Machine Intelligence, vol.29, no.10, pp. 1683-1699, Oct. 2007.
7. M. Pantic and L.J.M. Rothkrantz, "Automatic Analysis of Facial Expressions: The State of the Art," IEEE Trans. Pattern Analysis and Machine Intelligence, vol.22, no.12, pp. 1424-1445, Dec. 2000. (Pubitemid 32088469)
8. B. Fasel and J. Luettin, "Automatic Facial Expression Analysis: A Survey," Pattern Recognition, vol.36, no.1, pp. 259-275, 2003.
9. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor, "Emotion Recognition in Human-Computer Interaction," IEEE Signal Processing Magazine, vol.18, no.1, pp. 32-80, Jan. 2001.
10. Y. Tian, T. Kanade, and J. Cohn, "Facial Expression Analysis," Handbook of Face Recognition, S. Li and A. Jain, eds., Springer, 2004.
11. M. Pantic, A. Pentland, A. Nijholt, and T.S. Huang, "Human Computing Machine Understanding of Human Behavior: A Survey," Artificial Intelligence for Human Computing, T.S. Huang, A. Nijholt, M. Pantic, and A. Pentland, eds., Springer Verlag, 2007.
12. J. Wang, L. Yin, X. Wei, and Y. Sun, "3D Facial Expression Recognition Based on Primitive Surface Feature Distribution," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol.2, pp. 1399-1406, 2006. (Pubitemid 44931484)
13. Y. Chang, M. Vieira, M. Turk, and L. Velho, "Automatic 3D Facial Expression Analysis in Videos," Proc. Analysis and Modelling of Faces and Gestures, pp. 293-307, 2005.
14. B. Braathen, M.S. Bartlett, G.C. Littlewort, E. Smith, and J.R. Movellan, "An Approach to Automatic Recognition of Spontaneous Facial Actions," Proc. Fifth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 345-350, 2002.
15. M.S. Bartlett, G.C. Littlewort, M.G. Frank, C. Lainscsek, I.R. Fasel, and J.R. Movellan, "Automatic Recognition of Facial Actions in Spontaneous Expressions," J. Multimedia, vol.1, no.6, pp. 22-35, Sept. 2006.
16. F. Dornaika and F. Davoine, "Simultaneous Facial Action Tracking and Expression Recognition Using a Particle Filter," Proc. Int'l Conf. Computer Vision, vol.2, pp. 1733-1738, 2005.
17. Y. Zhang and Q. Ji, "Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences," IEEE Trans. Pattern Analysis and Machine Intelligence, vol.27, no.5, pp. 699-714, May 2005.
18. M.F. Valstar, I. Patras, and M. Pantic, "Facial Action Unit Detection Using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, Workshop Vision for Human-Computer Interaction, June 2005.
19. R. El Kaliouby and P. Robinson, "Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures," RealTime Vision for HCI, pp. 181-200, Springer Verlag, 2005.
20. B. Bascle and A. Blake, "Separability of Pose and Expression in Facial Tracking and Animation," Proc. Int'l Conf. Computer Vision, pp. 323-328, 1998.
21. Z. Zhu and Q. Ji, "Robust Real-Time Face Pose and Facial Expression Recovery," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol.1, pp. 681-688, 2006.
22. M. Anisetti, V. Bellandi, E. Damiani, and F. Beverina, "3D Expressive Face Model-Based Tracking Algorithm," Proc. Signal Processing, Pattern Recognition, and Applications, pp. 111-116, 2006.
23. M.A.O. Vasilescu and D. Terzopoulos, "Multilinear Analysis of Image Ensembles: Tensorfaces," Proc. European Conf. Computer Vision, pp. 447-460, 2002.
24. T.K. Marks, J. Hershey, J.C. Roddey, and J.R. Movellan, "Joint Tracking of Pose, Expression, and Texture Using Conditionally Gaussian Filters," Advances in Neural Information Processing Systems, vol.17, pp. 889-896, The MIT Press, 2005.
25. A. Kapoor, Y. Qi, and R.W. Picard, "Fully Automatic Upper Facial Action Recognition," Proc. IEEE Int'l Workshop Analysis and Modeling of Faces and Gestures, pp. 195-202, 2003.
26. J.F. Cohn, L.I. Reed, Z. Ambadar, J. Xiao, and T. Moriyama, "Automatic Analysis and Recognition of Brow Actions and Head Motion in Spontaneous Facial Behavior," Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, vol.1, pp. 610-616, 2004.
27. N. Sebe, M. Lew, I. Cohen, S. Yafei, T. Gevers, and T. Huang, "Authentic Facial Expression Analysis," Proc. Sixth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 517-522, 2004.
28. M.F. Valstar, M. Pantic, Z. Ambadar, and J.F. Cohn, "Spontaneous vs. Posed Facial Behavior: Automatic Analysis of Brow Actions," Proc. Eighth Int'l Conf. Multimodal Interfaces, pp. 162-170, 2006.
29. G. Littlewort, M.S. Bartlett, and K. Lee, "Faces of Pain: Automated Measurement of Spontaneous Facial Expressions of Genuine and Posed Pain," Proc. Ninth Int'l Conf. Multimodal Interfaces, pp. 15-21, 2007.
30. Z. Zeng, Y. Fu, G. Roisman, Z. Wen, Y. Hu, and T.S. Huang, "Spontaneous Emotional Facial Expression Detection," J. Multimedia, vol.1, no.5, pp. 1-8, 2006.
31. S. Ioannou, A. Raouzaiou, V. Tzouvaras, T. Mailis, K. Karpouzis, and S. Kollias, "Emotion Recognition through Facial Expression Analysis Based on a Neurofuzzy Method," Neural Networks, vol.18, no.4, pp. 423-435, 2005.
32. S. Lucey, A.B. Ashraf, and J. Cohn, "Investigating Spontaneous Facial Action Recognition through AAM Representations of the Face," Face Recognition Book, K. Kurihara, ed., Pro Literatur Verlag, Apr. 2007.
33. J. Russell and J. Fernandez-Dols, The Psychology of Facial Expression. Cambridge Univ. Press, 1997.
34. J.N. Bassili, "Emotion Recognition: The Role of Facial Movement and the Relative Importance of Upper and Lower Areas of the Face," J. Personality and Social Psychology, vol.37, no.11, pp. 2049-2058, 1979.
35. P. Ekman and E. Rosenberg, What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford Univ. Press, 2005.
36. P. Ekman, W.V. Friesen, and J.C. Hager, Facial Action Coding System: The Manual. Research Nexus Division, Network Information Research Corp., 2002.
37. P. Wang and Q. Ji, "Multi-View Face and Eye Detection Using Discriminant Features," Computer Vision and Image Understanding, vol.105, no.2, pp. 99-111, Feb. 2007.
38. Z. Zhu and Q. Ji, "Robust Pose Invariant Facial Feature Detection and Tracking in Real-Time," Proc. Int'l Conf. Pattern Recognition, vol.1, pp. 1092-1095, 2006. (Pubitemid 46532089)
39. Y. Tong, Y. Wang, Z. Zhu, and Q. Ji, "Robust Facial Feature Tracking under Varying Face Pose and Facial Expression," Pattern Recognition, vol.40, no.11, pp. 3195-3208, Nov. 2007.
40. K. Murphy, "Inference and Learning in Hybrid Bayesian Networks," Technical Report CSD-98-990, Dept. of Computer Science, Univ. of California Berkeley, 1998.
41. P. Ekman, Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. Norton, 1985.
42. K. Schmidt and J. Cohn, "Dynamics of Facial Expression: Normative Characteristics and Individual Differences," Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 728-731, 2001.
43. S. Nishio, K. Koyama, and T. Nakamura, "Temporal Differences in Eye and Mouth Movements Classifying Facial Expressions of Smiles" Proc. Third IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 206-211, Apr. 1998.
44. T. Dean and K. Kanazawa, "Probabilistic Temporal Reasoning," Proc. Seventh Nat'l Conf. Artificial Intelligence, pp. 524-528, 1988.
45. T. Kanade, J.F. Cohn, and Y. Tian, "Comprehensive Database for Facial Expression Analysis," Proc. Fourth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 46-53, 2000.
46. E. Douglas-Cowie, R. Cowie, and M. Schroeder, "The Description of Naturally Occurring Emotional Speech," Proc. 15th Int'l Congress of Phonetic Sciences, 2003.
47. Multiple Aspects of Discourse Research Lab, http://madre searchlab.org/ 2009.
48. G. Schwarz, "Estimating the Dimension of a Model," Annals of Statistics, vol.6, pp. 461-464, 1978.
49. D. Heckerman, D. Geiger, and D.M. Chickering, "Learning Bayesian Networks: The Combination of Knowledge and Statistical Data," Machine Learning, vol.20, no.3, pp. 197-243, 1995.
50. S.J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Prentice-Hall, 1995.
51. Y. Tong and Q. Ji, "Learning Bayesian Networks with Qualitative Constraints," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2008.
52. C.P. de Campos, Y. Tong, and Q. Ji, "Exploiting Qualitative Constraints for Learning Bayesian Network Parameters," Proc. European Conf. Computer Vision, 2008.
53. K.B. Korb and A.E. Nicholson, Bayesian Artificial Intelligence. Chapman and Hall/CRC, 2004.
54. K. Murphy, "The Bayes Net Toolbox for Matlab," Computing Science and Statistics, vol.33, pp. 331-350, 2001.