Understanding discrete facial expressions in video using an emotion avatar image

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 42, Issue 4, 2012, Pages 980-992

  Yang, Songfan ^a   Bhanu, Bir ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Avatar reference; emotion avatar image (EAI); face registration; person independent emotion recognition; Scale invariant feature transform (SIFT) flow

Indexed keywords

AVATAR REFERENCE; EMOTION AVATAR IMAGE (EAI); EMOTION RECOGNITION; FACE REGISTRATION; SCALE INVARIANT FEATURE TRANSFORMS;

GESTURE RECOGNITION;

FACE RECOGNITION;

EID: 84864140728     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2012.2192269     Document Type: Article

References (52)

1. J. A. Russell and J. M. Fernández-Dols, The Psychology of Facial Expression. Cambridge, U.K.: Cambridge Univ. Press, 1997.
2. M. Pantic and L. Rothkrantz, "Automatic analysis of facial expressions: The state of the art," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 12, pp. 1424-1445, Dec. 2000. (Pubitemid 32088469)
3. J. Yu and B. Bhanu, "Evolutionary feature synthesis for facial expression recognition," Pattern Recog. Lett., vol. 27, no. 11, pp. 1289-1298, Aug. 2006.
4. G. Zhao and M. Pietikäinen, "Dynamic texture recognition using local binary patterns with an application to facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 915-928, Jun. 2007. (Pubitemid 46667407)
5. T. Wu, M. Bartlett, and J. Movellan, "Facial expression recognition using Gabor motion energy filters," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog. Workshop Human Commun. Behav. Anal., Jun. 2010, pp. 42-47.
6. Y. Tong, J. Chen, and Q. Ji, "A unified probabilistic framework for spontaneous facial action modeling and understanding," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 2, pp. 258-273, Feb. 2010.
7. P. Yang, Q. Liu, and D. Metaxas, "Boosting coded dynamic features for facial action units and facial expression recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2007, pp. 1-6.
8. Z. Ambadar, J. Schooler, and J. Cohn, "Deciphering the enigmatic face: The importance of facial dynamics to interpreting subtle facial expressions," Psychol. Sci., vol. 16, no. 5, pp. 403-410, May 2005. (Pubitemid 40763688)
9. J. N. Bassili, "Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face," Pers. Social Psychol., vol. 37, no. 11, pp. 2049-2058, Nov. 1979.
10. G. Littlewort, J. Whitehill, T. Wu, I. Fasel, M. Frank, J. Movellan, and M. Bartlett, "Computer expression recognition toolbox," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., Mar. 2011, pp. 298-305.
11. R. El Kaliouby and P. Robinson, "Mind reading machines: Automated inference of cognitive mental states from video," in Proc. IEEE Int. Conf. Syst., Man Cybern., 2004, vol. 1, pp. 682-688. (Pubitemid 40415938)
12. M. Pantic and L. Rothkrantz, "Facial action recognition for facial expression analysis from static face images," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 34, no. 3, pp. 1449-1461, Jun. 2004.
13. M. Pantic, M. Valstar, R. Rademaker, and L. Maat, "Web-based database for facial expression analysis," in Proc. IEEE Int. Conf.Multimedia Expo., 2005, p. 5.
14. T. Kanade, J. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2000, pp. 46-53.
15. S. M. Boker, J. F. Cohn, B.-J. Theobald, I. Matthews, T. R. Brick, and J. R. Spiesaff, "Effects of damping head movement and facial expression in dyadic conversation using real-time facial expression tracking and synthesized avatars," Philosoph. Trans. B Roy. Soc., vol. 364, no. 1535, pp. 3485-3495, Dec. 2009.
16. Challenge Data. [Online]. Available: http://sspnet.eu/fera2011/ fera2011data
17. T. Cootes, G. Edwards, and C. Taylor, "Active appearance models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 6, pp. 681-685, Jun. 2001. (Pubitemid 32585279)
18. I. Matthews and S. Baker, "Active appearance models revisited," Int. J. Comput. Vis., vol. 60, no. 2, pp. 135-164, Nov. 2004.
19. S. Lucey, I. Matthews, C. Hu, Z. Ambadar, F. de la Torre, and J. Cohn, "AAM derived face representations for robust facial action recognition," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2006, pp. 155-160.
20. J. Saragih, S. Lucey, and J. Cohn, "Face alignment through subspace constrained mean-shifts," in Proc. IEEE Int. Conf. Comput. Vis., Sep. 29- Oct. 2, 2009, pp. 1034-1041.
21. G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski, "Classifying facial actions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 10, pp. 974-989, Oct. 1999. (Pubitemid 129304131)
22. T. Ojala, M. Pietikäinen, and T. Maenpaa, "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971-987, Jul. 2002. (Pubitemid 34835471)
23. G. Littlewort, M. Bartlett, I. Fasel, J. Susskind, and J. Movellan, "Dynamics of facial expression extracted automatically from video," in Image Vis. Comput., Jun. 2006, vol. 24, no. 6, pp. 615-625. (Pubitemid 43867835)
24. V. Ojansivu and J. Heikkilä, "Blur insensitive texture classification using local phase quantization," in Proc. ICISP, Berlin, Germany, 2008, pp. 236-243.
25. C. Liu, J. Yuen, and A. Torralba, "SIFT Flow: Dense correspondence across scenes and its applications," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 978-994, May 2011.
26. Y. Yacoob and L. Davis, "Recognizing human facial expressions from long image sequences using optical flow," IEEE Trans. Pattern Anal. Mach. Intell., vol. 18, no. 6, pp. 636-642, Jun. 1996. (Pubitemid 126773899)
27. I. Essa and A. Pentland, "Coding, analysis, interpretation, and recognition of facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 7, pp. 757-763, Jul. 1997. (Pubitemid 127762267)
28. M. Wang, Y. Iwai, and M. Yachida, "Expression recognition from timesequential facial images by use of expression change model," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., Apr. 1998, pp. 324-329.
29. C. Hu, Y. Chang, R. Feris, andM. Turk, "Manifold based analysis of facial expression," Image Vis. Comput., vol. 24, no. 6, pp. 605-614, Jun. 2006. (Pubitemid 43867834)
30. M. F. Valstar and M. Pantic, "Combined support vector machines and hidden Markov models for modeling facial action temporal dynamics," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog. Workshop Human Comput. Interact., Oct. 2007, pp. 118-127.
31. M. Pantic and I. Patras, "Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 36, no. 2, pp. 433-449, Apr. 2006.
32. M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Fully automatic facial action recognition in spontaneous behavior," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2006, pp. 223-230. (Pubitemid 44711354)
33. Y.-I. Tian, T. Kanade, and J. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 97-115, Feb. 2001. (Pubitemid 32254704)
34. F. Zhou, F. De la Torre, and J. Cohn, "Unsupervised discovery of facial events," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2010, pp. 2574-2581.
35. D. Lowe, "Object recognition from local scale-invariant features," in Proc. IEEE Int. Conf. Comput. Vis., 1999, vol. 2, pp. 1150-1157. (Pubitemid 32079997)
36. FERA2011 Challenge. [Online]. Available: http://sspnet.eu/fera2011
37. S. Yang and B. Bhanu, "Facial expression recognition using emotion avatar image," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 866-871.
38. U. Tariq, K.-H. Lin, Z. Li, X. Zhou, Z. Wang, V. Le, T. S. Huang, X. Lv, and T. X. Han, "Emotion recognition from an ensemble of features," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 872-877.
39. T. Gehrig and H. Ekenel, "A common framework for real-time emotion recognition and facial action unit eetection," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog. Workshop Human Comput. Interact., Jun. 2011, pp. 1-6.
40. G. Littlewort, J. Whitehill, T.-F. Wu, N. Butko, P. Ruvolo, J. Movellan, and M. Bartlett, "The Motion in Emotion-A CERT based approach to the FERA emotion challenge," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 897-902.
41. A. Dhall, A. Asthana, R. Goecke, and T. Gedeon, "Emotion recognition using PHOG and LPQ features," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 878-883.
42. H. Meng, B. Romera-Paredes, and N. Bianchi-Berthouze, "Emotion recognition by two view SVM 2K classifier on dynamic facial expression features," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 854-859.
43. H. Meng, J. Shawe-Taylor, S. Szedmak, and J. Farquhar, "Support vector machine to synthesise kernels," in Deterministic and Statistical Methods in Machine Learning, J. Winkler, M. Niranjan, and N. Lawrence, Eds. Berlin/Heidelberg, Germany: Springer-Verlag, 2005, pp. 242-255.
44. M. Dahmane and J. Meunier, "Emotion recognition using dynamic gridbased HoG features," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 884-888.
45. R. Srivastava, S. Roy, S. Yan, and T. Sim, "Accumulated motion images for facial expression recognition in videos," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 903-908.
46. S. W. Chew, P. J. Lucey, S. Lucey, J. Saragih, J. Cohn, and S. Sridharan, "Person-independent facial expression detection using constrained local models," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 915-920.
47. M. Valstar, B. Jiang, M. Méhu, M. Pantic, and K. Scherer, "The first facial expression recognition and analysis challenge," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog.Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 921-926.
48. T. Baltrusaitis, D. McDuff, N. Banda, M. Mahmoud, R. el Kaliouby, P. Robinson, and R. Picard, "Real-time inference of mental states from facial expressions and upper body gestures," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshop Facial Express. Recog. Anal. Challenge, Mar. 2011, pp. 909-914.
49. P. Viola and M. Jones, "Robust real-time face detection," International Journal of Computer Vision, vol. 57, no. 2, pp. 137-154, May 2004.
50. J. Han and B. Bhanu, "Individual recognition using gait energy image," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 2, pp. 316-322, Feb. 2006. (Pubitemid 46395297)
51. C.-C. Chang and C.-J. Lin, LIBSVM: A Library for Support Vector Machines, 2001. [Online]. Available: http://www.csie.ntu.edu.tw/cjlin/libsvm
52. A. Cruz, B. Bhanu, and S. Yang, "A psychologically-inspired match-score fusion model for video-based facial expression recognition," in Proc. Int. Conf. HUMAINE Assoc. Affective Comput. Intell. Interact. Workshop 1st Int. Audio/Visual Emotion Challenge, Oct. 2011, pp. 341-350.