Simultaneous facial feature tracking and facial expression recognition

IEEE Transactions on Image Processing

Volumn 22, Issue 7, 2013, Pages 2559-2573

  Li, Yongqiang ^a   Wang, Shangfei ^b   Zhao, Yongping ^a   Ji, Qiang ^c  

^a HARBIN INSTITUTE OF TECHNOLOGY   (China)

^b UNIVERSITY OF SCIENCE AND TECHNOLOGY OF CHINA   (China)

^c RENSSELAER POLYTECHNIC INSTITUTE   (United States)

Author keywords

Bayesian network; expression recognition; facial action unit recognition; facial feature tracking; simultaneous tracking and recognition

Indexed keywords

DYNAMIC BAYESIAN NETWORKS; EXPRESSION RECOGNITION; FACIAL ACTION CODING SYSTEM; FACIAL ACTION UNIT RECOGNITION; FACIAL EXPRESSION RECOGNITION; FACIAL FEATURE TRACKING; MACHINE LEARNING METHODS; SIMULTANEOUS TRACKING AND RECOGNITION;

BAYESIAN NETWORKS; GESTURE RECOGNITION; LEARNING SYSTEMS; MUSCLE;

FACE RECOGNITION;

ALGORITHM; ARTICLE; BAYES THEOREM; BIOMETRY; FACE; FACIAL EXPRESSION; HISTOLOGY; HUMAN; IMAGE PROCESSING; METHODOLOGY; PHYSIOLOGY; ANATOMY AND HISTOLOGY; PROCEDURES;

ALGORITHMS; BAYES THEOREM; BIOMETRIC IDENTIFICATION; FACE; FACIAL EXPRESSION; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED;

EID: 84877901074     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2013.2253477     Document Type: Article

References (54)

1. P. Ekman and W. V. Friesen, Facial Action Coding System (FACS): Manual. Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
2. Z. Zhu, Q. Ji, K. Fujimura, and K. Lee, "Combining Kalman filtering and mean shift for real time eye tracking under active IR illumination," in Proc. IEEE Int. Conf. Pattern Recognit., vol. 4. Aug. 2002, pp. 318-321. (Pubitemid 44851007)
3. T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, "Active shape models-their training and application," Comput. Vis. Image Understand., vol. 61, no. 1, pp. 38-59, 1995.
4. T. F. Cootes, G. J. Edwards, and C. Taylor, "Active appearance models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 6, pp. 681-685, Jun. 2001. (Pubitemid 32585279)
5. X. W. Hou, S. Z. Li, H. J. Zhang, and Q. S. Cheng, "Direct appearance models," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., vol. 1. Dec. 2001, pp. 828-833.
6. S. J. McKenna, S. Gong, R. P. Würtz, J. Tanner, and D. Banin, "Tracking facial feature points with Gabor wavelets and shape models," in Proc. Int. Conf. Audio-Video-Based Biometric Person Authent., vol. 1206. Mar. 1997, pp. 35-42. (Pubitemid 127046614)
7. M. Rogers and J. Graham, "Robust active shape model search," in Proc. Eur. Conf. Comput. Vis., 2002, pp. 517-530.
8. Y. Tong, Y. Wang, Z. Zhu, and Q. Ji, "Robust facial feature tracking under varying face pose and facial expression," Pattern Recognit., vol. 40, no. 11, pp. 3195-3208, 2007. (Pubitemid 46990444)
9. J. J. Lien, T. Kanade, J. F. Cohn, and C. Li, "Detection, tracking, and classification of action units in facial expression," J. Robot. Auto. Syst., vol. 31, no. 3, pp. 131-146, 2000.
10. G. Donato, M. S. Bartlett, J. C. Hager, P. Ekman, and T. J. Sejnowski, "Classifying facial actions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 10, pp. 974-989, Oct. 1999. (Pubitemid 129304131)
11. R. Lienhart and J. Maydt, "An extended set of Haar-like features for rapid object detection," in Proc. IEEE Int. Conf. Image Process., vol. 1. Sep. 2002, pp. 900-903.
12. B. A. Draper, K. Baek, M. S. Bartlett, and J. R. Beveridge, "Recognizing faces with PCA and ICA," Comput. Vis. Image Understand., vol. 91, nos. 1-2, pp. 115-137, 2003.
13. Cassio P. de Campos and Q. Ji, "Efficient structure learning of Bayesian networks using constraints," J. Mach. Learn. Res., vol. 12, pp. 663-689, Mar. 2011.
14. M. S. Bartlett, G. Littlewort, M. G. Frank, C. Lainscsek, I. Fasel, and J. R. Movellan, "Recognizing facial expression: Machine learning and application to spontaneous behavior," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., vol. 2. Jun. 2005, pp. 568-573.
15. G. Donato, M. S. Bartlett, J. C. Hager, P. Ekman, and T. J. Sejnowski, "Classifying facial actions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 10, pp. 974-989, Oct. 1999. (Pubitemid 129304131)
16. Y. Tian, T. Kanade, and J. F. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 97-115, Feb. 2001. (Pubitemid 32254704)
17. G. Zhao and M. Pietikainen, "Boosted multi-resolution spatiotemporal descriptors for facial expression recognition," Pattern Recognit. Lett., vol. 30, no. 12, pp. 1117-1127, 2009.
18. M. Valstar and M. Pantic, "Combined support vector machines and hidden Markov models for modeling facial action temporal dynamics," in Proc. IEEE Int. Conf. Human-Comput. Interact., vol. 4796. Oct. 2007, pp. 118-127.
19. M. Valstar and M. Pantic, "Fully automatic facial action unit detection and temporal analysis," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshop, Jun. 2006, p. 149.
20. A. Kapoor, Y. Qi, and R. W. Picard, "Fully automatic upper facial action recognition," in Proc. IEEE Int. Workshop Anal. Model. Faces Gestures, Oct. 2003, pp. 195-202.
21. M. Pantic and I. Patras, "Dynamics of facial expressions: Recogniton of facial actions and their temporal segments from face profile image sequences," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 36, no. 2, pp. 433-449, Apr. 2006.
22. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
23. J. J. Lien, T. Kanade, J. F. Cohn, and C. Li, "Detection, tracking, and classification of action units in facial expression," J. Robot. Auto. Syst., vol. 31, no. 3, pp. 131-146, 2000.
24. Y. Tong, J. Chen, and Q. Ji, "A unified probabilistic framework for spontaneous facial activity modeling and understanding," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 2, pp. 258-273, Feb. 2010.
25. Y. Tong, W. Liao, and Q. Ji, "Facial action unit recognition by exploiting their dynamic and semantic relationships," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 10, pp. 1683-1699, Oct. 2007.
26. F. Dornaika and F. Davoine, "Simultaneous facial action tracking and expression recognition in the presence of head motion," Int. J. Comput. Vis., vol. 76, no. 3, pp. 257-281, 2008.
27. K. Schwerdt and J. L. Crowley, "Robust face tracking using color," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., Mar. 2000, pp. 90-95.
28. Y. B. Shalom and X. Li, Estimation, Tracking: Principles, Techniques, and Software. Norwood, MA, USA: Artech House, 1993.
29. Z. Ghahramani and G. E. Hinton, "Variational learning for switching state-space models," Neural Computat., vol. 12, no. 4, pp. 831-864, 2000.
30. J. Chen and Q. Ji, "A hierarchical framework for simultaneous facial activity tracking," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., Mar. 2011, pp. 679-686.
31. C. Shan, S. Gong, and P. W. McOwan, "Facial expression recognition based on local binary patterns: A comprehensive study," Image Vis. Comput., vol. 27, no. 6, pp. 803-816, 2009.
32. G. Zhao and M. Pietikainen, "Dynamic texture recognition using local binary patterns with an application to facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 915-928, Jun. 2007. (Pubitemid 46667407)
33. M. H. Mahoor, M. Zhou, K. L. Veon, S. M. Mavadati, and J. F. Cohn, "Facial action unit recognition with sparse representation," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., Mar. 2011, pp. 336-342.
34. S. W. Chew, R. Rana, P. Lucey, S. Lucey, and S. Sridharan, "Sparse Temporal Representations for Facial Expression Recognition," in Advances in Image and Video Technology (Lecture Notes in Computer Science), vol. 7088. New York, NY, USA: Springer-Verlag, 2012, pp. 311-322.
35. K. Schmidt and J. Cohn, "Dynamics of facial expression: Normative characteristics and individual differences," in Proc. IEEE Int. Conf. Multimedia Expo, Aug. 2001, pp. 728-731.
36. Y. Chang, C. Hu, R. Feris, and M. Turk, "Manifold-based analysis of facial expression," J. Image Vis. Comput., vol. 24, no. 6, pp. 605-614, 2006.
37. J. Whitehill and C. W. Omlin, "Haar features for FACS AU recognition," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., Apr. 2006, pp. 97-101. (Pubitemid 44711334)
38. S. Lucey, A. Ashraf, and J. Cohn, Investigating Spontaneous Facial Action Recognition Through AAM Representations of the Face, K. Kurihara, Ed. Augsburg, Germany: Pro Literatur Verlag, 2007, pp. 395-406.
39. L. Zhang and D. Tjondronegoro, "Facial expression recognition using facial movement features," IEEE Trans. Affect. Comput., vol. 2, no. 4, pp. 219-229, Oct.-Dec. 2011.
40. G. Schwarz, "Estimating the dimension of a model," Ann. Stat., vol. 6, no. 2, pp. 461-464, 1978.
41. D. Heckerman, D. Geiger, and D. M. Chickering, "Learning Bayesian networks: The combination of knowledge and statistical data," Mach. Learn., vol. 20, no. 3, pp. 197-243, 1995.
42. K. B. Korb and A. E. Nicholson, Bayesian Artificial Intelligence. London, U.K.: Chapman & Hall, 2004.
43. M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Automatic recognition of facial actions in spontaneous expressions," J. Multimedia, vol. 1, no. 6, pp. 22-35, 2006.
44. T. Kanade, J. Cohn, and Y. L. Tian, "Comprehensive database for facial expression analysis," in Proc. 4th IEEE Int. Conf. Autom. Face Gesture Recognit., Mar. 2000, pp. 46-53.
45. S. Koelstra, M. Pantic, and I. Patras, "A dynamic texture-based approach to recognition of facial actions and their temporal models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 1940-1954, Nov. 2010.
46. P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, "The extended Cohn-Kande dataset (CK+): A complete facial expression dataset for action unit and emotion-specified expression," in Proc. 3rd IEEE Int. Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 94-101.
47. M. Valstar and M. Pantic, "Fully automatic recognition of the temporal phases of facial actions," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 1, pp. 28-43, Feb. 2012.
48. I. Patras and M. Pantic, "Particle filtering with factorized likelihoods for tracking facial features," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., May 2004, pp. 97-102.
49. H. Dibeklioglu, A. A. Salah, and T. Gevers, "A statistical method for 2-D facial landmarking," IEEE Trans. Image Process., vol. 21, no. 2, pp. 844-858, Feb. 2012.
50. I. Cohen, N. Sebe, A. Garg, L. S. Chen, and T. S. Huang, "Facial expression recognition from video sequences: Temporal and static modeling," Comput. Vis. Image Understand., vol. 91, nos. 1-2, pp. 160-187, 2003.
51. Y. Zhang and Q. Ji, "Active and dynamic information fusion for facial expression understanding from image sequences," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 5, pp. 699-714, May 2005. (Pubitemid 40608403)
52. U. Kjærulff, "dHugin: A computational system for dynamic time-sliced Bayesian networks," Int. J. Forecast., vol. 11, no. 1, pp. 89-111, 1995.
53. M. Pantic, M. Valstar, R. Rademaker, and L. Maat, "Web-based database for facial expression analysis," in Proc. IEEE Int. Conf. Multimedia Expo, Jul. 2005, pp. 317-321. (Pubitemid 44668868)
54. C. E. Izard, Human Emotions. New York, NY, USA: Plenum, 1977.