Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order

Pattern Recognition

Volumn 61, Issue , 2017, Pages 610-628

  Lopes, André Teixeira ^a   de Aguiar, Edilson ^a   De Souza, Alberto F ^a   Oliveira Santos, Thiago ^a  

^a FEDERAL UNIVERSITY OF ESPÍRITO SANTO   (Brazil)

Author keywords

Computer vision; Convolutional Neural Networks; Expression specific features; Facial expression recognition; Machine learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; BIG DATA; COMPUTER VISION; CONVOLUTION; HUMAN COMPUTER INTERACTION; IMAGE PROCESSING; LEARNING SYSTEMS; NEURAL NETWORKS;

CONVOLUTIONAL NEURAL NETWORK; DEEP ARCHITECTURES; EXPRESSION SPECIFIC FEATURES; FACIAL EXPRESSION RECOGNITION; IMAGE PREPROCESSING; MACHINE LEARNING METHODS; RECOGNITION OF FACIAL EXPRESSIONS; TRAINING AND TESTING;

FACE RECOGNITION;

EID: 84991821737     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2016.07.026     Document Type: Article

References (82)

1. Y. Wu, H. Liu, H. Zha, Modeling facial expression space for recognition, in: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2005 (IROS 2005), 2005, pp. 1968–1973.
2. C. Darwin, The Expression of the Emotions in Man and Animals, CreateSpace Independent Publishing Platform, 2012.
3. S.Z. Li, A.K. Jain, Handbook of Face Recognition, Springer Science & Business Media, Secaucus, NJ, USA, 2011.
4. P. Lucey, J. Cohn, T. Kanade, J. Saragih, Z. Ambadar, I. Matthews, The extended Cohn–Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression, in: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2010, pp. 94–101.
5. Lyons, M., Budynek, J., Akamatsu, S., Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21:12 (1999), 1357–1362.
6. L. Yin, X. Wei, Y. Sun, J. Wang, M. Rosato, A 3d facial expression database for facial behavior research, in: 7th International Conference on Automatic Face and Gesture Recognition (FGR06), Institute of Electrical & Electronics Engineers (IEEE), Southampton, UK, 2006.
7. P. Liu, S. Han, Z. Meng, Y. Tong, Facial expression recognition via a boosted deep belief network, in: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1805–1812.
8. Shan, C., Gong, S., McOwan, P.W., Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27:6 (2009), 803–816.
9. W. Liu, C. Song, Y. Wang, Facial expression recognition based on discriminative dictionary learning, in: 2012 21st International Conference on Pattern Recognition (ICPR), 2012, pp. 1839–1842.
10. I. Song, H.-J. Kim, P.B. Jeon, Deep learning for real-time robust facial expression recognition on a smartphone, in: International Conference on Consumer Electronics (ICCE), Institute of Electrical & Electronics Engineers (IEEE), Las Vegas, NV, USA, 2014.
11. P. Burkert, F. Trier, M.Z. Afzal, A. Dengel, M. Liwicki, Dexpression: Deep Convolutional Neural Network for Expression Recognition, CoRR abs/1509.05371 (URL (http://arxiv.org/abs/1509.05371).
12. Liu, M., Li, S., Shan, S., Chen, X., Au-inspired deep networks for facial expression feature learning. Neurocomputing 159 (2015), 126–136, 10.1016/j.neucom.2015.02.011.
13. Ali, G., Iqbal, M.A., Choi, T.-S., Boosted NNE collections for multicultural facial expression recognition. Pattern Recognit. 55 (2016), 14–27, 10.1016/j.patcog.2016.01.032.
14. Y.-H. Byeon, K.-C. Kwak, Facial expression recognition using 3d convolutional neural network. International Journal of Advanced Computer Science and Applications(IJACSA), 5 (2014).
15. J.-J.J. Lien, T. Kanade, J. Cohn, C. Li, Detection, tracking, and classification of action units in facial expression, J. Robot. Auton. Syst. 31(3), 2000, 131-146.
16. Fan, X., Tjahjadi, T., A spatial–temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences. Pattern Recognit. 48:11 (2015), 3407–3416.
17. Zhang, W., Zhang, Y., Ma, L., Guan, J., Gong, S., Multimodal learning for facial expression recognition. Pattern Recognit. 48:10 (2015), 3191–3202.
18. Chen, C.-R., Wong, W.-S., Chiu, C.-T., A 0.64 mm real-time cascade face detection design based on reduced two-field extraction. IEEE Trans. Very Large Scale Integr. (VLSI) Syst. 19:11 (2011), 1937–1948.
19. Garcia, C., Delakis, M., Convolutional face finder: a neural architecture for fast and robust face detection. IEEE Trans. Pattern Anal. Mach. Intell. 26:11 (2004), 1408–1423, 10.1109/TPAMI.2004.97.
20. Zhang, Z., Yi, D., Lei, Z., Li, S., Regularized transfer boosting for face detection across spectrum. IEEE Signal Process. Lett. 19:3 (2012), 131–134.
21. M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, J. Movellan, Recognizing facial expression: machine learning and application to spontaneous behavior, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005 (CVPR 2005), vol. 2, 2005, pp. 568–573.
22. P. Liu, M. Reale, L. Yin, 3d head pose estimation based on scene flow and generic head model, in: 2012 IEEE International Conference on Multimedia and Expo (ICME), 2012, pp. 794–799.
23. W.W. Kim, S. Park, J. Hwang, S. Lee, Automatic head pose estimation from a single camera using projective geometry, in: 2011 8th International Conference on Information, Communications and Signal Processing (ICICS), 2011, pp. 1–5.
24. M. Demirkus, D. Precup, J. Clark, T. Arbel, Multi-layer temporal graphical model for head pose estimation in real-world videos, in: 2014 IEEE International Conference on Image Processing (ICIP), 2014, pp. 3392–3396.
25. Z. Zhang, M. Lyons, M. Schuster, S. Akamatsu, Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron, in: 1998 Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, pp. 454–459.
26. P. Yang, Q. Liu, D. Metaxas, Boosting coded dynamic features for facial action units and facial expression recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, 2007 (CVPR’07), 2007, pp. 1–6.
27. S. Jain, C. Hu, J. Aggarwal, Facial expression recognition with temporal modeling of shapes, in: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), 2011, pp. 1642–1649, 10.1109/ICCVW.2011.6130446.
28. Lin, Y., Song, M., Quynh, D.T.P., He, Y., Chen, C., Sparse coding for flexible, robust 3d facial-expression synthesis. IEEE Comput. Graph. Appl. 32:2 (2012), 76–88.
29. S. Rifai, Y. Bengio, A. Courville, P. Vincent, M. Mirza, Disentangling factors of variation for facial expression recognition, in: A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, C. Schmid (Eds.), Computer Vision – ECCV 2012, Lecture Notes in Computer Science, vol. 7577, Springer, Berlin Heidelberg, 2012, pp. 808–822.
30. B. Fasel, Robust face analysis using convolutional neural networks, in: Proceedings of the 16th International Conference on Pattern Recognition, 2002, vol. 2, 2002, pp. 40–43.
31. F. Beat, Head-pose invariant facial expression recognition using convolutional neural networks, in: Proceedings of the Fourth IEEE International Conference on Multimodal Interfaces, 2002, 2002, pp. 529–534.
32. Matsugu, M., Mori, K., Mitari, Y., Kaneda, Y., Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw.: Off. J. Int. Neural Netw. Soc. 16:5 (2003), 555–559.
33. Y. Bengio, Y. LeCun, Scaling learning algorithms towards AI, in: L. Bottou, O. Chapelle, D. DeCoste, J. Weston (Eds.), Large-Scale Kernel Machines, MIT Press, Cambridge, Massachusetts, USA, 2007 (URL (http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf).
34. Utgoff, P.E., Stracuzzi, D.J., Many-layered learning. Neural Comput. 14:10 (2002), 2497–2529.
35. Y. Bengio, I.J. Goodfellow, A. Courville, Deep Learning, MIT Press, Cambridge, Massachusetts, USA, 2015.
36. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based Learn. Appl. Doc. Recognit. 86:11 (1998), 2278–2324, 10.1109/5.726791.
37. D.C. Cirean, U. Meier, J. Masci, L.M. Gambardella, J. Schmidhuber, Flexible, high performance convolutional neural networks for image classification, in: Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence (IJCAI'11), vol. 2, AAAI Press, Barcelona, Catalonia, Spain, 2011, pp. 1237–1242.
38. P. Zhao-yi, W. Zhi-qiang, Z. Yu, Application of mean shift algorithm in real-time facial expression recognition, in: International Symposium on Computer Network and Multimedia Technology, 2009 (CNMT 2009), 2009, pp. 1–4.
39. Patil, H.Y., Kothari, A.G., Bhurchandi, K.M., Expression invariant face recognition using local binary patterns and contourlet transform. Opt.-Int. J. Light Electron Opt. 127:5 (2016), 2670–2678, 10.1016/j.ijleo.2015.11.187.
40. J.Y.R. Cornejo, H. Pedrini, F. Florez-Revuelta, Facial expression recognition with occlusions based on geometric representation, in: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: Proceedings of the 20th Iberoamerican Congress (CIARP 2015), Montevideo, Uruguay, November 9–12, 2015, Springer International Publishing, Cham, 2015, pp. 263–270.
41. Wang, Z., Ruan, Q., An, G., Facial expression recognition using sparse local fisher discriminant analysis. Neurocomputing 174:Part B (2016), 756–766, 10.1016/j.neucom.2015.09.083.
42. S. Arivazhagan, R.A. Priyadharshini, S. Sowmiya, Facial expression recognition based on local directional number pattern and anfis classifier, in: 2014 International Conference on Communication and Network Technologies (ICCNT), 2014, pp. 62–67 ( http://dx.doi.org/10.1109/CNT.2014.7062726).
43. Rivera, A.R., Castillo, J.R., Chae, O.O., Local directional number pattern for face analysis: face and expression recognition. IEEE Trans. Image Process. 22:5 (2013), 1740–1752.
44. Zavaschi, T.H., Britto, A.S., Oliveira, L.E., Koerich, A.L., Fusion of feature sets and classifiers for facial expression recognition. Expert Syst. Appl. 40:2 (2013), 646–655, 10.1016/j.eswa.2012.07.074.
45. A.T. Lopes, E. de Aguiar, T.O. Santos, A facial expression recognition system using convolutional networks, in: 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images, Institute of Electrical & Electronics Engineers (IEEE), Salvador, Bahia, Brasil, 2015.
46. S. Demyanov, J. Bailey, R. Kotagiri, C. Leckie, Invariant Backpropagation: How To Train a Transformation-Invariant Neural Network ( arXiv:1502.04434[cs, stat]).
47. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, T. Darrell, Caffe: Convolutional Architecture for Fast Feature Embedding ( arXiv:1408.5093).
48. C.-D. Caleanu, Face expression recognition: a brief overview of the last decade, in: 2013 IEEE 8th International Symposium on Applied Computational Intelligence and Informatics (SACI), 2013, pp. 157–161.
49. Meguid, M.K.A.E., Levine, M.D., Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers. IEEE Trans. Affect. Comput. 5:2 (2014), 141–154, 10.1109/TAFFC.2014.2317711.
50. C. Turan, K. M. Lam, Region-based feature fusion for facial-expression recognition, in: 2014 IEEE International Conference on Image Processing (ICIP), 2014, pp. 5966–5970 ( http://dx.doi.org/10.1109/ICIP.2014.7026204).
51. M. Lyons, S. Akamatsu, M. Kamachi, J. Gyoba, Coding facial expressions with gabor wavelets, in: Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, 1998, pp. 200–205.
52. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15:1 (2014), 1929–1958.
53. J.M. Girard, J.F. Cohn, L.A. Jeni, S. Lucey, F.D. la Torre, How much training data for facial action unit detection?, in: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, 2015, pp. 1–8 ( http://dx.doi.org/10.1109/FG.2015.7163106).
54. M. Valstar, M. Pantic, Induced disgust, happiness and surprise: an addition to the mmi facial expression database, in: Proceedings of the 3rd International Workshop on EMOTION (Satellite of LREC): Corpora for Research on Emotion and Affect, 2010, p. 65.
55. A. Dhall, R. Goecke, S. Lucey, T. Gedeon, Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark, in: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), IEEE, Barcelona, Catalonia, Spain, 2011, pp. 2106–2112.
56. P. Simard, D. Steinkraus, J.C. Platt, Best practices for convolutional neural networks applied to visual document analysis, in: 2003 Proceedings of the Seventh International Conference on Document Analysis and Recognition, 2003, pp. 958–963.
57. J.-I. Choi, C.-W. La, P.-K. Rhee, Y.-L. Bae, Face and eye location algorithms for visual user interface, in: Proceedings of First Signal Processing Society Workshop on Multimedia Signal Processing, Institute of Electrical & Electronics Engineers (IEEE), Princeton, NJ, USA, 1997.
58. G. Li, X. Cai, X. Li, Y. Liu, An efficient face normalization algorithm based on eyes detection, in: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, Institute of Electrical & Electronics Engineers (IEEE), Beijing, China, 2006.
59. Saragih, J.M., Lucey, S., Cohn, J.F., Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vision. 91:2 (2010), 200–215.
60. A. Asthana, S. Zafeiriou, S. Cheng, M. Pantic, Robust discriminative response map fitting with constrained local models, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 3444–3451.
61. S. Cheng, A. Asthana, S. Zafeiriou, J. Shen, M. Pantic, Real-time generic face tracking in the wild with cuda, in: Proceedings of the 5th ACM Multimedia Systems Conference, ACM, Singapore, Singapore 2014, pp. 148–151.
62. B.A. Wandell, Foundations of Vision, 1st ed., Sinauer Associates Inc, Sunderland, Mass, 1995.
63. L. Bottou, Stochastic Gradient Descent Tricks, Springer, New York, NY, USA. 2012.
64. X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS10), Society for Artificial Intelligence and Statistics, Sardinia, Italy, 2010.
65. X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, in: G.J. Gordon, D.B. Dunson (Eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS-11), vol. 15, 2011, pp. 315–323.
66. Ekman, P., Friesen, W., Facial Action Coding System: A Technique for the Measurement of Facial Movement. 1978, Consulting Psychologists Press, Palo Alto.
67. Gu, W., Xiang, C., Venkatesh, Y., Huang, D., Lin, H., Facial expression recognition using radial encoding of local gabor features and classifier synthesis. Pattern Recognit. 45:1 (2012), 80–91, 10.1016/j.patcog.2011.05.006.
68. L. Zhong, Q. Liu, P. Yang, B. Liu, J. Huang, D. Metaxas, Learning active facial patches for expression analysis, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 2562–2569.
69. Lee, S.H., Baddar, W.J., Ro, Y.M., Collaborative expression representation using peak expression and intra class variation face images for practical subject-independent emotion recognition in videos. Pattern Recognit. 54 (2016), 52–67, 10.1016/j.patcog.2015.12.016.
70. F.D. la Torre, W.S. Chu, X. Xiong, F. Vicente, X. Ding, J. Cohn, Intraface, in: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, 2015, pp. 1–8 ( http://dx.doi.org/10.1109/FG.2015.7163082).
71. J. Cohn A. Zlochower, A Computerized Analysis of Facial Expression: Feasibility of Automated Discrimination, vol. 2. American Psychological Society, 1995, p. 6.
72. M. Xue, A. Mian, W. Liu, L. Li, Fully automatic 3d facial expression recognition using local depth features, in: IEEE Winter Conference on Applications of Computer Vision, 2014, pp. 1096–1103 (http://dx.doi.org/10.1109/WACV.2014.6835736).
73. Sha, T., Song, M., Bu, J., Chen, C., Tao, D., Feature level analysis for 3d facial expression recognition. Neurocomputing 74:12–13 (2011), 2135–2141, 10.1016/j.neucom.2011.01.008.
74. Maalej, A., Amor, B.B., Daoudi, M., Srivastava, A., Berretti, S., Shape analysis of local facial patches for 3d facial expression recognition. Pattern Recognit. 44:8 (2011), 1581–1589, 10.1016/j.patcog.2011.02.012.
75. M. Liu, S. Shan, R. Wang, X. Chen, Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1749–1756.
76. Mery, D., Bowyer, K., Automatic facial attribute analysis via adaptive sparse representation of random patches. Pattern Recognit. Lett. 68:Part 2 (2015), 260–269 (Special Issue on “Soft Biometrics”, http://dx.doi.org/10.1016/j.patrec.2015.05.005).
77. Siddiqi, M.H., Ali, R., Khan, A.M., Park, Y.T., Lee, S., Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans. Image Process. 24:4 (2015), 1386–1398, 10.1109/TIP.2015.2405346.
78. A. Zafer, R. Nawaz, J. Iqbal, Face recognition with expression variation via robust ncc, in: 2013 IEEE 9th International Conference on Emerging Technologies (ICET), 2013, pp. 1–5 (http://dx.doi.org/10.1109/ICET.2013.6743520).
79. Siddiqi, M.H., Ali, R., Khan, A.M., Kim, E.S., Kim, G.J., Lee, S., Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection. Multimed. Syst. 21:6 (2014), 541–555, 10.1007/s00530-014-0400-2.
80. Siddiqi, M.H., Ali, R., Idris, M., Khan, A.M., Kim, E.S., Whang, M.C., Lee, S., Human facial expression recognition using curvelet feature extraction and normalized mutual information feature selection. Multimed. Tools Appl. 75:2 (2014), 935–959, 10.1007/s11042-014-2333-3.
81. T. Kanade, Y. Tian, J.F. Cohn, Comprehensive database for facial expression analysis, in: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000 (FG’00), IEEE Computer Society, Washington, DC, USA, 2000, p. 46.
82. O.M. Parkhi, A. Vedaldi, A. Zisserman, Deep face recognition, in: British Machine Vision Conference, 2015, 46-53.