Adaptive 3D facial action intensity estimation and emotion recognition

Expert Systems with Applications

Volumn 42, Issue 3, 2015, Pages 1446-1464

  Zhang, Yang ^a   Zhang, Li ^a   Hossain, M A ^b  

^a NORTHUMBRIA UNIVERSITY   (United Kingdom)

^b ANGLIA RUSKIN UNIVERSITY   (United Kingdom)

Author keywords

Action unit intensity estimation; Adaptive ensemble classifiers; Complementary Neural Networks; Facial emotion recognition; Support vector classification; Support Vector Regression

Indexed keywords

COMPLEMENTARY NEURAL NETWORKS; ENSEMBLE CLASSIFIERS; FACIAL EMOTIONS; INTENSITY ESTIMATION; SUPPORT VECTOR CLASSIFICATION; SUPPORT VECTOR REGRESSION (SVR);

EID: 84908424301     PISSN: 09574174     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.eswa.2014.08.042     Document Type: Article

References (84)

1. Ahlberg, J. (2001). CANDIDE-3 - an updated parameterized face. Report No. LiTH-ISY-R-2326, Department of Electrical Engineering, Linkoping University, Sweden.
2. G. Antonini, M. Sorci, M. Bierlaire, and J. Thiran Discrete choice models for static facial expression recognition 8th international conference on advanced concepts for intelligent vision systems Lecture notes in computer science Vol. 41 2006 Springer Berlin 710 721
3. M.S. Bartlett, G.C. Littlewort, M.G. Frank, C. Lainscsek, I.R. Fasel, and J.R. Movellan Automatic recognition of facial actions in spontaneous expressions Journal of Multimedia 1 6 2006 22 35
4. D. Basak, S. Pal, and D.C. Patranabis Support vector regression Neural Information Processing - Letters and Review 11 10 2007
5. A. Besinger, T. Sztynda, S. Lal, C. Duthoit, J. Agbinya, and B. Jap Optical flow based analyses to detect emotion from human facial image data Expert Systems with Applications 37 2010 8897 8902
6. Bouguet, Y. L. (1999). Pyramidal implementation of the Lucas-Kanade feature tracker. Technical Report, Intel Corporation, Microprocessor Research Labs.
7. C.-C. Chang, and C.-J. Lin LIBSVM: A library for support vector machines ACM Transactions on Intelligent Systems and Technology 2011 2:27: 1-27:27
8. Chang, K. Y.; Liu, T. L.; & Lai, S. H. (2009). Learning partially-observed hidden conditional random fields for facial expression recognition. In IEEE conference on computer vision and pattern recognition, CVPR (pp. 533-540).
9. Chang, Y.; Hu, C.; & Turk, M. (2004). Probabilistic expression analysis on manifolds. In Proceedings of the IEEE international'1 conference on computing. vision and pattern recognition.
10. U.B. Chavan, and D.B. Kulkarni Facial expression recognition - review International Journal of Latest Trends in Engineering and Technology (IJLTET) 3 1 2013 237 243
11. S.C. Cheng, M.Y. Chen, H.Y. Chang, and T.C. Chou Semantic-based facial expression recognition using analytical hierarchy process Expert Systems with Applications 33 1 2007 86 95
12. Cohn, J.; Kruez, T. S.; Matthew, I.; Yang, Y.; Nguyen, M.; Padilla, M. T.; et al. (2009). Detecting depression from facial actions and vocal prosody. In International conference on affective computing and intelligent interaction (ACII2009).
13. S. D'Mello, and A. Graesser Multimodal semi-automated affect detection from conversational cues, gross body language and facial features User Model User-Adapted Interaction 20 2 2010 147 187
14. R.A. Davis, A. Charlton, S. Oehlschlager, and J. Wilson Novel feature selection method for genetic programming using metabolomic 1H NMR data Chemometrics and Intelligent Laboratory Systems 81 1 2006 50 59
15. M.H. DeGroot, and M.J. Schervish Probability and statistics 4th ed. 2011 Pearson
16. P. Ekman, and W.V. Friesen Constants across cultures in the face and emotion Journal of Personality and Social Psychology 17 2 1971 124 129
17. Ekman, P.; & Friesen, W. V. (1983). Emfacs-7: Emotional facial action coding system. University of California at San Francisco.
18. P. Ekman, W.V. Friesen, and J.C. Hager Facial action coding system, the manual 2002 Research Nexus division of Network Information Research Corporation USA
19. P. Ekman, W.V. Friesen, and J.C. Hager Facial action coding system investigator's guide 2002 Consulting Psychologist Press Palo Alto, CA
20. D. Farid, L. Zhang, A.M. Hossain, C.M. Rahman, R. Strachan, and G. Sexton An adaptive ensemble classifier for mining concept-drifting data streams Expert Systems with Applications 40 15 2013 5895 5906
21. A.S. G'Mussel, and J. Hewig The value of a smile: Facial expression affects ultimatum-game responses Judgment and Decision Making 8 3 2013 381 385
22. N. Garćia-Pedrajas, C. Hervás-MartÍInez, and D. Ortiz-Boyer Cooperative coevolution of artificial neural network ensembles for pattern classification IEEE Transactions on Evolutionary Computation 9 3 2005 271 302
23. W. Gu, C. Xiang, Y.V. Venkatesh, D. Huang, and H. Lin Facial expression recognition using radial encoding of local Gabor features and classifier synthesis Pattern Recognition 45 2012 80 91
24. Hecht-Nielsen, R. (1989). Theory of the backpropagation neural network. In IJCNN, international joint conference on neural networks. San Diego, CA, USA.
25. Hsu, C.; Chang, C.; & Lin, C. (2010). A practical guide to support vector classification. Department of Computer Science National, Taiwan University.
26. Jeatrakul, P.; & Wong, K. W. (2009). Comparing the performance of different neural networks for binary classification problems. In Eighth international symposium on natural language processing, SNLP '09 (pp. 111-115).
27. Jong, D. H.; Ziemkiewicz, C.; Ribarsky, W.; & Chang, R. (2009). Understanding principal component analysis using a visual analytics tool. Charlotte Visualization Center, UNC Charlotte.
28. Kaltwang, S.; Rudovic, O.; & Pantic, M. (2012). Continuous pain intensity estimation from facial expressions. In Advances in visual computing. Lecture notes in computer science (Vol. 7432, pp. 368-377). Heidelberg: Springer.
29. A. Kappas Smile when you read this, whether you like it or not: Conceptual challenges to affect detection IEEE Transactions on Affective Computing 1 1 2010 38 42
30. S. Koelstra, M. Pantic, and I. Patras A dynamic texture based approach to recognition of facial actions and their temporal models IEEE Transactions on, Pattern Analysis and Machine Intelligence 32 11 2010 1940 1954
31. I. Kotsia, S. Zafeiriou, and I. Pitas Texture and shape information fusion for facial expression and facial action unit recognition Pattern Recognition 41 2008 822 851
32. Kraipeerapun, P. (2008). Neural network classification based on quantification of uncertainty. Murdoch University.
33. Y. Li, and Q. Ji Data-free Prior Model for Facial Action Unit Recognition IEEE Transactions on Affective Computing 4 2 2013 127 141
34. Lucey, P.; Cohn, J.; Lucey, S.; Matthews, I.; Sridharan, S.; & Prkachin, K. (2009). Automatically detecting pain using facial actions. In Proceedings of the international conference on affective computing and intelligent interaction (pp. 1-8).
35. Lucey, P.; Cohn, J. F.; Kanade, T.; Saragih, J.; Ambadar, Z.; & Matthews, I. (2010). The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In Proceedings of IEEE workshop on CVPR for human communicative behavior analysis. San Francisco, USA.
36. Lucey, S.; Matthews, I.; Hu, C.; Cohn, J.; & Ambadar, Z. (2006). AAM derived face representations for robust facial action recognition. In IEEE international conference on automatic face and gesture recognition (FG).
37. Mahoor, M. H.; Zhou, M.; Veon, S.; & Cohn, J. (2011). Facial action unit recognition with sparse representation. In Proceedings of IEEE automatic face & gesture recognition and workshops (FG 2011).
38. M.M. Masud, J. Gao, L. Khan, J. Han, and B. Thuraisingham Classification and novel class detection in concept-drifting data streams under time constraints IEEE Transactions on Knowledge and Data Engineering 23 2011 859 874
39. A. Metallinou, A. Katsamanis, and S. Narayanan Tracking continuous emotional trends of participants during affective dyadic interactions using body language and speech information Image and Vision Computing 31 2 2013 137 152
40. T. Mitchell, and M. Hill Machine learning 1997 Publisher McGraw-Hill Inc NY, USA
41. Mpiperis, L. (2008). 3D facial expression recognition using swarm intelligence. In IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 23-24). Las Vegas, NV, USA.
42. E. Owusu, Y. Zhan, and Q.R. Mao A neural-AdaBoost based facial expression recognition system Expert Systems with Applications 41 7 2014 3383 3390
43. I. Pandzic, and R. Forchheimer MPEG-4 facial animation: The standard, implementation and applications 2002 Wiley
44. M. Pantic, and I. Patras Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences IEEE Transactions on Systems, Man and Cybernetics, Part B: Cybernetics 36 2 2006 433 449
45. H. Peng, F. Long, and C. Ding Feature selection based on mutual information: Criteria of max-dependency, max-relevance and min-redundancy IEEE Transactions on Pattern Analysis and Machine Intelligence 27 2005 1226 1238
46. K.S. Rao, V.K. Saroj, S. Maity, and S.G. Koolagudi Recognition of emotions from video using neural network models Expert Systems with Applications 38 10 2011 13181 13185
47. L. Rokach Ensemble-based classifiers Artificial Intelligence Review 33 1-2 2010 1 39
48. Ryan, A.; Cohn, J.; Lucey, S.; Saragih, J.; Lucey, P.; & Rossi, A. (2009). Automated facial expression recognition system. In Proceedings of the international Carnahan conference on security technology (pp. 172-177).
49. Salahshoor, S. & Faez, K. (2012). 3D face recognition using an expression insensitive dynamic mask. In Image and signal processing. Lecture notes in computer science (Vol. 7340, pp. 253-260).
50. G. Sandbach, S. Zafeiriou, M. Pantic, and D. Rueckert Recognition of 3D facial expression dynamics Image Vision Computing 30 2012 762 773 issue
51. Savran, A.; Alyuz, N.; Dibeklioglu, H.; Celiktutan, O.; Gokberk, B.; Sankur, B.; et al. (2008). Bosphorus database for 3D face analysis. In: Proceedings of first COST 2101 workshop on biometrics and identity management (pp. 47-56). Denmark.
52. A. Savran, B. Sankur, and M.T. Bilge Comparative evaluation of 3D vs. 2D modality for automatic detection of facial action units Pattern Recognition 45 2012 767 782
53. A. Savran, B. Sankur, and M.T. Bilge Regression-based intensity estimation of facial action units Pattern Recognition 30 2012 774 784
54. C. Shan, S. Gong, and P.W. McOwan Facial expression recognition based on local binary patterns: A comprehensive study Image and Vision Computing 27 2009 803 816
55. R. Sikora, and S. Piramuthu Framework for efficient feature selection in genetic algorithm based data mining European Journal of Operational Research 180 2 2007 723 737
56. M. Sorci, and J.Ph. Thiran Modeling human perception of static facial expressions Image and Vision Computing 28 5 2010 790 806
57. Soyel, H.; & Demirel, H. (2007). Facial expression recognition using 3D facial feature distances. In ICIAR07 (pp. 831-838).
58. Srivastava, R.; & Roy, S. (2009). 3D facial expression recognition using residues. In 2009-2009 IEEE region 10 conference on TENCON (pp. 1-5).
59. D.L. Swets, and J. Weng Using discriminant eigenfeatures for image retrieval IEEE Transactions on Pattern Analysis and Machine Intelligence 18 8 1996 831 836
60. Tang, H.; & Huang, T. S. (2008). 3D facial expression recognition based on properties of line segments connecting facial feature points. In Proceedings of the 8th IEEE international conference on automatic face gesture recognition (pp. 1-6).
61. Tang, H.; & Huang, T. S. (2008). 3D facial expression recognition based on automatically selected features. In IEEE computer society conference on computer vision and pattern recognition workshops (pp. 1-8).
62. Tian, Y. L. (2002). Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity. In Proceedings of the 5th IEEE international conference on automatic face and gesture recognition (FG'02). Washington, DC.
63. Y. Tong, J. Chen, and Q. Ji A unified probabilistic framework for spontaneous facial action modeling and understanding IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 32 2 2010 258 273
64. Y. Tong, E. Liao, and Q. Ji Facial action unit recognition by exploiting their dynamic and semantic relationships IEEE Transactions on Pattern Analysis and Machine Intelligence 29 10 2007 1683 1699
65. Torralba, A.; & Efros, A. (2011). Unbiased look at dataset bias. In IEEE conference on computer vision and pattern recognition.
66. F. Tsalakanidou, and S. Malassiotis Real-time 2D+3d facial action and expression recognition Pattern Recognition 43 5 2010 1763 1775
67. Ujir, H. (2013). 3D facial expression classification using a statistical model of surface normals and a modular approach (Ph.D. thesis): University of Birmingham.
68. Valstar, M.; & Pantic, M. (2006). Biologically vs. logic inspired encoding of facial actions and emotions. In Proceedings of IEEE international conference on multimedia and expo (ICME) (pp. 325-328).
69. M. Valstar, and M. Pantic Combined support vector machines and hidden markov models for modeling facial action temporal dynamics Lecture Notes on Computer Science 4796 2007 118 127
70. M. Valstar, and M. Pantic Fully automatic recognition of the temporal phases of facial actions IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 42 1 2012 28 43
71. V. Vapnik The nature of statistical learning theory 1995 Springer Verlag USA
72. V.N. Vapnik The nature of statistical learning theory 2nd ed. 2001 Springer NY, USA
73. Vukadinovic, D.; & Pantic, M. (2005). Fully automatic facial feature point detection using Gabor feature based boosted features. In Proceedings of IEEE international conference on systems, man, and cybernetics (pp. 1692-1698).
74. Vural, E.; Cetin, M.; Ercil, A.; Littlewort, G.; Bartlett, M.; & Movellan, J. (2008). Automated drowsiness detection for improved driver safety. In Proceedings of the international conference on automotive technologies.
75. Wang, J.; Yin, L.; Wei, X.; & Sun, Y. (2006). 3D facial expression recognition based on primitive surface feature distribution. In IEEE conference on computer vision and pattern recognition (CVPR). Washington, DC, USA.
76. T.H. Wang, and J. Lien Facial expression recognition system based on rigid and non-rigid motion separation and 3D pose estimation Pattern Recognition 42 2009 962 977
77. X. Wang, J. Yang, X. Teng, W. Xia, and R. Jensen Feature selection based on rough sets and particle swarm optimization Pattern Recognition Letters 28 4 2007 459 471
78. J. Webb, and J. Ashley Beginning Kinect programming with the Microsoft Kinect SDK 2012 Apress USA
79. Wen, Z.; & Huang, T. (2003). Capturing subtle facial motions in 3d face tracking. In Proceedings of international conference on computer vision.
80. Whitehill, J.; & Omlin, C.W. (2006). Haar features for FACS AU recognition. In 7th international conference on automatic face and gesture recognition, FGR 2006 (pp. 217-222).
81. Whitehill, J.; Tingfan, W.; Fasel, I.; Frank, M.; Movellan. J.; & Bartlett, M. (2011). The computer expression recognition toolbox (CERT). In IEEE conference on, automatic face & gesture recognition and workshops (FG 2011) (pp. 298-305).
82. T. Zavaschi, L.S. Oliveira, A.B. Souza Jr, and A. Koerich Fusion of feature sets and classifiers for facial expression recognition Expert System with Applications 40 2 2013 646 655
83. L. Zhang Facial expression recognition using facial movement features IEEE Transactions on Affective Computing 2 4 2011 219 230
84. 4C. Zhang, M. Jiang, D. Farid, and M.A. Hossain Intelligent facial emotion recognition and semantic-based topic detection for a humanoid robot Expert Systems with Applications 40 13 2013 5160 5168