Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 38, Issue 8, 2016, Pages 1548-1568

  Corneanu, Ciprian Adrian ^a,b   Simón, Marc Oliu ^a,b   Cohn, Jeffrey F ^c,d   Guerrero, Sergio Escalera ^a,b  

^a UNIVERSITAT AUTÒNOMA DE BARCELONA   (Spain)

^b UNIVERSITY OF BARCELONA   (Spain)

^c CARNEGIE MELLON UNIVERSITY   (United States)

^d UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

Author keywords

3D; affect; emotion recognition; Facial expression; multimodal; RGB; thermal

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER VISION;

AFFECT; EMOTION RECOGNITION; FACIAL EXPRESSIONS; MULTI-MODAL; THERMAL;

FACE RECOGNITION;

ALGORITHM; AUTOMATED PATTERN RECOGNITION; FACE; FACIAL EXPRESSION; HUMAN;

ALGORITHMS; FACE; FACIAL EXPRESSION; HUMANS; PATTERN RECOGNITION, AUTOMATED;

EID: 84978370387     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2515606     Document Type: Article

References (274)

1. R. W. Roger Highfield and R. Jenkins, "How your looks betray your personality," New Scientist, 2009.
2. A. Chastel, Leonardo on Art and the Artist. Mineola, NY, USA: Courier Corp., 2002.
3. S. Greenblatt, "Toward a universal language of motion: Reflections on a seventeenth centurymuscleman," vol. 21, pp. 56-62, 1994.
4. G.-B. D. de Boulogne and R. A. Cuthbertson, The Mechanism of Human Facial Expression. Cambridge, U.K.: Cambridge Univ. Press, 1990.
5. C. Darwin, The Expression of Emotion in Man and Animals. Oxford, U.K.: Oxford Univ. Press, 1872.
6. C. E. Izard, The Face of Emotion. New York, NY, USA: St. Martin's Griffin, 1971.
7. P. Ekman, "Universal and cultural differences in facial expression of emotion," Nebr. Sym. Motiv., vol. 19, pp. 207-283, 1971.
8. P. Ekman and H. Oster, "Facial expressions of emotion," Annu. Rev. Psychol., no. 30, pp. 527-554, 1979.
9. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
10. A. A. Salah, N. Sebe, and T. Gevers, "Communication and automatic interpretation of affect from facial expressions," in Affective Computing and Interaction: Psychological, Cognitive and Neuroscientific Perspectives. Hershey, PA, USA: IGI Global, 2010, p. 157.
11. G. Sandbach, S. Zafeiriou, M. Pantic, and L. Yin, "Static and dynamic 3D facial expression recognition: A comprehensive survey," Image Vision Comput., vol. 30, pp. 683-697, 2012.
12. E. Sariyanidi, H. Gunes, and A. Cavallaro, "Automatic analysis of facial affect: A survey of registration, representation and recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 6, pp. 1113-1133, Jun. 2015.
13. P. Ekman, "Strong evidence for universals in facial expressions: A reply to Russell's mistaken critique," Psychol. Bull., vol. 115, no. 2, pp. 268-287, 1994.
14. (2016). [Online]. Available: http://what-when-how.com/facerecognition/ facial-expression-recognition-face-recognition-techniques- part1/
15. M. Greenwald, E. Cook, and P. Lang, "Affective judgment and psychophysiological response: Dimensional covariation in the evaluation of pictorial stimuli," J. Psychophysiol., vol. 3, pp. 51-64, 1989.
16. J. Russell and A. Mehrabian, "Evidence for a three-factor theory of emotions," J. Res. Personality, vol. 11, pp. 273-294, 1977.
17. D. Watson, L. A. Clark, and A. Tellegen, "Development and validation of brief measures of positive and negative affect: The PANAS scales," J. Pers. Soc. Psychol., vol. 54, pp. 1063-1070, 1988.
18. A. J. Fridlund, "The behavioral ecology and sociality of human faces," in Emotion. Newbury, Park, CA, USA: Sage, 1997, pp. 90-121.
19. A. F. Shariff and J. L. Tracy, "What are emotion expressions for?" Current Directions Psychol. Sci., vol. 20, no. 6, pp. 395-399, 2011.
20. L. F. Barrett, "Was Darwin wrong about emotional expressions?" Current Directions Psychol. Sci., vol. 20, no. 6, pp. 400-406, 2011.
21. I. Eibl-Eibesfeldt, "An argument for basic emotions," Cogn. Emotion, vol. 6, pp. 169-200, 1992.
22. D. Keltner and P. Ekman, "Facial expression of emotion," in Handbook of Emotions, 2nd ed. New York, NY, USA: Guilford Press, 2000, pp. 236-249.
23. D. Matsumoto, D. Keltner, M. N. Shiota, M. O'Sullivan, and M. Frank, "Facial expressions of emotion," in Handbook of Emotions, 2008, ch. 13, pp. 211-234.
24. K. L. Schmidt and J. F. Cohn, "Human facial expressions as adaptations: Evolutionary perspectives in facial expression research," Yearbook Phys. Anthropol., vol. 116, pp. 8-24, 2001.
25. H. Gray and C. M. Goss, Anatomy of the Human Body, 28th ed. Philadelphia, PA, USA: Lea & Febiger, 1966.
26. A. Burrows and J. F. Cohn, "Comparative anatomy of the face." in Handbook of Biometrics, 2nd ed. New York, NY, USA: Springer, 2014, pp. 1-10.
27. B. M. Waller, J. J. Cray, and A. M. Burrows, "Selection for universal facial emotion," Emotion, vol. 8, no. 3, pp. 435-439, 2008.
28. B. M. Waller, M. Lembeck, P. Kuchenbuch, A. M. Burrows, and K. Liebal, "GibbonFACS: A muscle-based facial movement coding system for hylobatids," J. Primatol., vol. 33, pp. 809-821, 2012.
29. B. M. Waller, L. A. Parr, K. M. Gothard, A. M. Burrows, and A. J. Fuglevand, "Mapping the contribution of single muscles to facial movements in the rhesus macaque," Physiol. Behav., vol. 95, pp. 93-100, 2008.
30. I. Eibl-Eibesfeldt, Human Ethology. Piscataway, NJ, USA: Transaction Publishers, 1989.
31. J. A. Russell, "Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies," Psychol. Bull., vol. 115, no. 1, pp. 102-141, 1994.
32. R. E. Jack, C. Blais, C. Scheepers, P. G. Schyns, and R. Caldara, "Cultural confusions show that facial expressions are not universal," Current Biol., vol. 19, pp. 1-6, 2009.
33. R. W. Levenson, P. Ekman, and W. V. Friesen, "Voluntary facial action generates emotion-specific autonomic nervous system activity," Psychophysiology, vol. 27, no. 4, pp. 363-384, 1990.
34. P. Ekman, R. J. Davidson, and W. V. Friesen, "The Duchenne smile: Emotional expression and brain psychology ii," J. Pers. Social Psychol., vol. 58, no. 2, pp. 342-353, 1990.
35. N. H. Frijda and A. Tcherkassof, "Facial expressions as modes of action readiness," in The Psychology of Facial Expression, 2nd ed. Cambridge, U.K.: Cambridge Univ. Press, 1997, pp. 78-102.
36. P. M. Niedenthal, "Embodying emotion," Science, vol. 116, pp. 1002-1005, 2007.
37. P. Ekman and E. Rosenberg, What the Face Reveals, 2nd ed. New York, NY, USA: Oxford Univ. Press, 2005.
38. X. Zhang, L. Yin, J. F. Cohn, S. Canavan, M. Reale, A. Horowitz, P. Liu, and J. M. Girard, "BP4D-spontaneous: A high-resolution spontaneous 3D dynamic facial expression database," Image Vision Comput., vol. 32, no. 10, pp. 692-706, 2014.
39. Z. Duric, W. D. Gray, R. Heishman, F. Li, A. Rosenfeld, M. J. Schoelles, C. Schunn, and H. Wechsler, "Integrating perceptual and cognitive modeling for adaptive and intelligent humancomputer interaction," Proc. IEEE, vol. 90, no. 7, pp. 1272-1289, Jul. 2002.
40. L. Maat and M. Pantic, "Gaze-X: Adaptive, affective, multimodal interface for single-user office scenarios," in Artifical Intelligence for Human Computing. New York, NY, USA: Springer, 2007, pp. 251-271.
41. A. Vinciarelli, M. Pantic, and H. Bourlard, "Social signal processing: Survey of an emerging domain," Image Vision Comput., vol. 27, no. 12, pp. 1743-1759, 2009.
42. D. DeVault, R. Artstein, G. Benn, T. Dey, E. Fast, A. Gainer, and L.-P. Morency, "SimSensei kiosk: A virtual human interviewer for healthcare decision support," in Proc. Int. Conf. Auton. Agents Multi-Agent Systems, 2014, pp. 1061-1068.
43. H. Ishiguro, T. Ono, M. Imai, T. Maeda, T. Kanda, and R. Nakatsu, "Robovie: An interactive humanoid robot," Ind. Robot: Int. J., vol. 28, no. 6, pp. 498-504, 2001.
44. A. Kapoor, W. Burleson, and R. W. Picard, "Automatic prediction of frustration," Int. J. Human-Comput. Stud., vol. 65, no. 8, pp. 724-736, 2007.
45. S. Bakkes, C. T. Tan, and Y. Pisan, "Personalised gaming," JCT, vol. 3, p. 4, 2012.
46. C. T. Tan, D. Rosser, S. Bakkes, and Y. Pisan, "A feasibility study in using facial expressions analysis to evaluate player experiences," in Proc. 8th Australasian Conf. Interactive Entertainment: Playing Syst., 2012, p. 5.
47. P. M. Blom, S. Bakkes, C. T. Tan, S. Whiteson, D. Roijers, R. Valenti, and T. Gevers, "Towards personalised gaming via facial expression recognition," in Proc. 10th Annu. AAAI Conf. Artif. Intell. Interactive Digital Entertainment, 2014, pp. 30-36.
48. P. Lucey, J. F. Cohn, I. Matthews, S. Lucey, S. Sridharan, J. Howlett, and K. M. Prkachin, "Automatically detecting pain in video through facial action units," IEEE Trans. Syst., Man., Cybern. B., Cybern., vol. 41, no. 3, pp. 664-674, Jun. 2011.
49. S. Kaltwang, O. Rudovic, and M. Pantic, "Continuous pain intensity estimation from facial expressions," in Proc. Int. Adv. Visual Comput., 2012, pp. 368-377.
50. R. Irani, K. Nasrollahi, M. O. Simon, C. A. Corneanu, S. Escalera, C. Bahnsen, D. H. Lundtoft, T. B. Moeslund, T. L. Pedersen, M.- L. Klitgaard, and L. Petrini, "Spatiotemporal analysis of RGB-DT facial images for multimodal pain level recognition," in Proc. IEEE Conf. Comput. Vision Pattern Recog. Workshop, 2015, pp. 88-95.
51. A. Ryan, J. F. Cohn, S. Lucey, J. Saragih, P. Lucey, F. De la Torre, and A. Ross, "Automated facial expression recognition system," in Proc. 43rd Ann. Int. Carnahan Conf. Security Technol., 2009, pp. 172-177.
52. E. Vural, M. Cetin, A. Ercil, G. Littlewort, M. Bartlett, and J. Movellan, "Drowsy driver detection through facial movement analysis," in Proc. IEEE Int. Conf. Human-Comput. Interaction, 2007, pp. 6-18.
53. J. M. Girard, J. F. Cohn, M. H. Mahoor, S. M. Mavadati, Z. Hammal, and D. P. Rosenwald, "Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses," Image Vision Comput., vol. 32, no. 10, pp. 641-647, 2014.
54. S. Scherer, G. Stratou, M. Mahmoud, J. Boberg, J. Gratch, A. Rizzo, and L.-P. Morency, "Automatic behavior descriptors for psychological disorder analysis," in Proc. 10th IEEE Int. Conf. Workshops Autom. Face Gesture Recog., 2013, pp. 1-8.
55. J. Joshi, A. Dhall, R. Goecke, M. Breakspear, and G. Parker, "Neural-net classification for spatio-temporal descriptor based depression analysis," in Proc. 21st Int. Conf. Pattern Recog., 2012, pp. 2634-2638.
56. (2016). [Online]. Available: www.emotient.com
57. (2016). [Online]. Available: www.affectiva.com
58. (2016). [Online]. Available: www.realeyesit.com
59. (2016). [Online]. Available: www.kairos.com
60. P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2001, pp. I-511.
61. M. Jones and P. Viola, "Fast multi-view face detection," Mitsubishi Electric Res. Lab., Cambridge, MA, USA, Tech. Rep. TR2003-96, 2003.
62. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2005, pp. 886-893.
63. M. Osadchy, Y. L. Cun, and M. L. Miller, "Synergistic face detection and pose estimation with energy-based models," J. Mach. Learning Res., vol. 8, pp. 1197-1215, 2007.
64. A. Colombo, C. Cusano, and R. Schettini, "3D face detection using curvature analysis," Pattern Recog., vol. 39, no. 3, pp. 444-455, 2006.
65. P. Nair and A. Cavallaro, "3-D face detection, landmark localization, and registration using a point distribution model," IEEE Trans. Multimedia, vol. 11, no. 4, pp. 611-623, 2009.
66. K. Sobottka and I. Pitas, "Segmentation and tracking of faces in color images," in Proc. 2nd Int. Conf. Autom. Face Gesture Recog., 1996, pp. 236-241.
67. S. A. Sirohey, "Human face segmentation and identification, "Univ. Maryland, College Park, MD, USA, Tech. Rep. CS-TR-3176, 1998.
68. K. Sobottka and I. Pitas, "A novel method for automatic face segmentation, facial feature extraction and tracking," Signal Process.: Image Commun., vol. 12, no. 3, pp. 263-281, 1998.
69. D. Chai and K. N. Ngan, "Face segmentation using skin-color map in videophone applications," IEEE Trans. Circuits Syst. Video Technol., vol. 9, no. 4, pp. 551-564, Jun. 1999.
70. H. Li and K. N. Ngan, "Saliency model-based face segmentation and tracking in head-and-shoulder video sequences," J. Visual Commun. Image Representation, vol. 19, no. 5, pp. 320-333, 2008.
71. J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, M. Finocchio, A. Blake, M. Cook, and R. Moore, "Real-time human pose recognition in parts from single depth images," Commun. ACM, vol. 56, no. 1, pp. 116-124, 2013.
72. A. Hernández, N. Zlateva, A. Marinov, M. Reyes, P. Radeva, D. Dimov, and S. Escalera, "Graph cuts optimization for multi-limb human segmentation in depth maps," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2012, pp. 726-732.
73. M. Pamplona Segundo, L. Silva, O. R. P. Bellon, and C. C. Queirolo, "Automatic face segmentation and facial landmark detection in range images," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 40, no. 5, pp. 1319-1330, Oct. 2010.
74. Y. Koda, Y. Yoshitomi, M. Nakano, and M. Tabuse, "A facial expression recognition for a speaker of a phoneme of vowel using thermal image processing and a speech recognition system," in Proc. 18th IEEE Int. Symp. Robot Human Interactive Commun., 2009, pp. 955-960.
75. L. Trujillo, G. Olague, R. Hammoud, and B. Hernandez, "Automatic feature localization in thermal images for facial expression recognition," in Proc. IEEE Conf. Comput. Vision Pattern Recog. Workshops, 2005, pp. 14-14.
76. T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, "Active shape models - Their training and application," Comput. Vision Image Understanding, vol. 61, no. 1, pp. 38-59, 1995.
77. T. F. Cootes, G. J. Edwards, and C. J. Taylor, "Active appearance models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 6, pp. 681-685, Jun. 2001.
78. S. Romdhani and T. Vetter, "Efficient, robust and accurate fitting of a 3D morphable model," in Proc. IEEE Int. Conf. Comput. Vision, 2003, pp. 59-66.
79. S. Baker and I. Matthews, "Lucas-Kanade 20 years on: A unifying framework," Int. J. Comput. Vision, vol. 56, no. 3, pp. 221-255, 2004.
80. M. Dantone, J. Gall, G. Fanelli, and L. Van Gool, "Real-time facial feature detection using conditional regression forests," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2012, pp. 2578-2585.
81. X. Xiong and F. De la Torre, "Supervised descent method and its applications to face alignment," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2013, pp. 532-539.
82. P. J. Besl and N. D. McKay, "Method for registration of 3-d shapes," in Proc. Robotics-DL tentative, 1992, pp. 586-606.
83. N. Alyuz, B. Gokberk, and L. Akarun, "Adaptive registration for occlusion robust 3D face recognition," in Proc. Eur. Conf. Comput. Vision, 2012, pp. 557-566.
84. Z. Mao, J. P. Siebert, W. P. Cockshott, and A. F. Ayoub, "Constructing dense correspondences to analyze 3D facial change," in Proc. Int. Conf. Pattern Recog., 2004, pp. 144-148.
85. J. R. Tena, M. Hamouz, A. Hilton, and J. Illingworth, "A validated method for dense non-rigid 3D face registration," in Proc. IEEE Int. Conf. Video Signal Based Surveillance, 2006, pp. 81-81.
86. P. Szeptycki, M. Ardabilian, and L. Chen, "A coarse-to-fine curvature analysis-based rotation invariant 3D face landmarking," in Proc. 3rd IEEE Int. Conf. Biometrics: Theory, Appl. Syst., 2009, pp. 1-6.
87. N. Alyuz, B. Gokberk, and L. Akarun, "Regional registration for expression resistant 3-d face recognition," IEEE Trans. Inf. Forensics Security, vol. 5, no. 3, pp. 425-440, Sep. 2010.
88. V. Blanz and T. Vetter, "A morphable model for the synthesis of 3D faces," in Proc. SIGGRAPH, 1999, pp. 187-194.
89. G. Fanelli, M. Dantone, and L. Van Gool, "Real time 3D face alignment with random forests-based active appearance models," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognition, 2013, pp. 1-8.
90. A. Savran and B. Sankur, "Non-rigid registration of 3D surfaces by deformable 2D triangular meshes," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2008, pp. 1-6.
91. C. C. Queirolo, L. Silva, O. R. Bellon, and M. Pamplona Segundo, "3D face recognition using simulated annealing and the surface interpenetration measure," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 2, pp. 206-219, Feb. 2010.
92. I. Mpiperis, S. Malassiotis, V. Petridis, and M. G. Strintzis, "3D facial expression recognition using swarm intelligence," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2008, pp. 2133-2136.
93. A. Dhall, A. Asthana, R. Goecke, and T. Gedeon, "Emotion recognition using PHOG and LPQ features," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. Workshops, 2011, pp. 878-883.
94. B. Sun, L. Li, T. Zuo, Y. Chen, G. Zhou, and X. Wu, "Combining multimodal features with hierarchical classifier fusion for emotion recognition in the wild," in Proc. 16th Int. Conf. Multimodal Interaction, 2014, pp. 481-486.
95. R. Zhi, M. Flierl, Q. Ruan, and W. B. Kleijn, "Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 1, pp. 38-52, Feb. 2011.
96. S. Zafeiriou and M. Petrou, "Nonlinear nonnegative component analysis," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2009, pp. 2860-2865.
97. C. Shan, S. Gong, and P. W. McOwan, "Facial expression recognition based on local binary patterns: A comprehensive study," Image Vision Comput., vol. 27, no. 6, pp. 803-816, 2009.
98. A. Savran, H. Cao, A. Nenkova, and R. Verma, "Temporal Bayesian fusion for affect sensing: Combining video, audio, and lexical modalities," IEEE Trans. Cybern., vol. 45, no. 9, pp. 1927-1941, Sep. 2015.
99. G. Littlewort, M. S. Bartlett, I. Fasel, J. Susskind, and J. Movellan, "Dynamics of facial expression extracted automatically from video," in Proc. IEEE Conf. Comput. Vision Pattern Recog. Workshops, 2004, p. 80.
100. G. Littlewort, J. Whitehill, T. Wu, I. Fasel, M. Frank, J. Movellan, and M. Bartlett, "The computer expression recognition toolbox (CERT)," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2011, pp. 298-305.
101. W. Gu, C. Xiang, Y. Venkatesh, D. Huang, and H. Lin, "Facial expression recognition using radial encoding of local Gabor features and classifier synthesis," Pattern Recog., vol. 45, no. 1, pp. 80-91, 2012.
102. M. J. Lyons, J. Budynek, and S. Akamatsu, "Automatic classification of single facial images," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 12, pp. 1357-1362, Dec. 1999.
103. Y. Yoshitomi, N. Miyawaki, S. Tomita, and S. Kimura, "Facial expression recognition using thermal image processing and neural network," in Proc. 6th IEEE Int. Workshop Robot Human Commun., 1997, pp. 380-385.
104. S. Wang, M. He, Z. Gao, S. He, and Q. Ji, "Emotion recognition from thermal infrared images using deep Boltzmann machine," Frontiers Comput. Sci., vol. 8, no. 4, pp. 609-618, 2014.
105. Y. Yoshitomi, T. Asada, K. Shimada, and M. Tabuse, "Facial expression recognition for speaker using thermal image processing and speech recognition system," in Proc. 10th WSEAS Int. Conf. Appl. Comput. Sci., 2010, pp. 182-186.
106. G. Zhao and M. Pietikainen, "Dynamic texture recognition using local binary patterns with an application to facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 915-928, Jun. 2007.
107. L. He, D. Jiang, L. Yang, E. Pei, P. Wu, and H. Sahli, "Multimodal affective dimension prediction using deep bidirectional long short-term memory recurrent neural networks," in Proc. 5th Int. Workshop Audio/Visual Emotion Challenge, 2015, pp. 73-80.
108. M. Liu, R. Wang, S. Li, S. Shan, Z. Huang, and X. Chen, "Combining multiple kernel methods on Riemannian manifold for emotion recognition in the wild," in Proc. Int. Conf. Multimodal Interaction, 2014, pp. 494-501.
109. Z. Liu and S. Wang, "Emotion recognition using hidden Markov models from facial temperature sequence," in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 240-247.
110. A. Geetha, V. Ramalingam, S. Palanivel, and B. Palaniappan, "Facial expression recognition - A real time approach," Expert Syst. Appl., vol. 36, no. 1, pp. 303-308, 2009.
111. B. Hernández, G. Olague, R. Hammoud, L. Trujillo, and E. Romero, "Visual learning of texture descriptors for facial expression recognition in thermal imagery," Comput. Vision Image Understanding, vol. 106, no. 2, pp. 258-269, 2007.
112. P. Liu and L. Yin, "Spontaneous facial expression analysis based on temperature changes and head motions," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2015, pp. 1-6.
113. R. Walecki, O. Rudovic, V. Pavlovic, and M. Pantic, "Variablestate latent conditional random fields for facial expression recognition and action unit detection," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2015, pp. 1-8.
114. M. Pantic and I. Patras, "Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences," IEEE Trans. Syst., Man., Cybern. B., Cybern., vol. 36, no. 2, pp. 433-449, Apr. 2006.
115. N. Sebe, M. S. Lew, Y. Sun, I. Cohen, T. Gevers, and T. S. Huang, "Authentic facial expression analysis," Image Vision Comput., no. 12, pp. 1856-1863, 2007.
116. I. Kotsia and I. Pitas, "Facial expression recognition in image sequences using geometric deformation features and support vector machines," IEEE Trans. Image Process., vol. 16, no. 1, pp. 172-187, Jan. 2007.
117. H. Tang and T. S. Huang, "3D facial expression recognition based on properties of line segments connecting facial feature points," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2008, pp. 1-6.
118. H. Tang and T. Huang, "3D facial expression recognition based on automatically selected features," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2008, pp. 1-8.
119. Y. Chang, M. Vieira, M. Turk, and L. Velho, "Automatic 3D facial expression analysis in videos," in Proc. 2nd Int. Conf. Anal. Model. Faces Gestures, 2005, pp. 293-307.
120. S. Berretti, B. B. Amor, M. Daoudi, and A. Del Bimbo, "3D facial expression recognition using sift descriptors of automatically detected keypoints," Visual Comput.: Int. J. Comput. Graphics, vol. 27, no. 11, pp. 1021-1036, 2011.
121. N. Vretos, N. Nikolaidis, and I. Pitas, "3D facial expression recognition using Zernike moments on depth images," in Proc. 18th IEEE Int. Conf. Image Process., 2011, pp. 773-776.
122. G. Sandbach, S. Zafeiriou, and M. Pantic, "Local normal binary patterns for 3D facial action unit detection," in Proc. IEEE Int. Conf. Image Process., 2012, pp. 1813-1816.
123. M. Hayat, M. Bennamoun, and A. A. El-Sallam, "Clustering of video-patches on Grassmannian manifold for facial expression recognition from 3D videos," in Proc. IEEE Workshop Appl. Comput. Vision, 2013, pp. 83-88.
124. W. Zeng, H. Li, L. Chen, J. M. Morvan, and X. D. Gu, "An automatic 3D expression recognition framework based on sparse representation of conformal images," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2013, pp. 1-8.
125. P. Lemaire, M. Ardabilian, L. Chen, and M. Daoudi, "Fully automatic 3D facial expression recognition using differential mean curvature maps and histograms of oriented gradients," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2013, pp. 1-7.
126. M. Wöllmer, M. Kaiser, F. Eyben, B. Schuller, and G. Rigoll, "LSTM-modeling of continuous emotions in an audiovisual affect recognition framework," Image Vision Comput., vol. 31, no. 2, pp. 153-163, 2013.
127. S. Koelstra, M. Pantic, and I. Patras, "A dynamic texture-based approach to recognition of facial actions and their temporal models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 1940-1954, Nov. 2010.
128. V. Le, H. Tang, and T. S. Huang, "Expression recognition from 3D dynamic faces using robust spatio-temporal shape features," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2011, pp. 414-421.
129. G. Sandbach, S. Zafeiriou, M. Pantic, and D. Rueckert, "A dynamic approach to the recognition of 3D facial expressions and their temporal models," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2011, pp. 406-413.
130. T. Fang, X. Zhao, S. K. Shah, and I. A. Kakadiaris, "4D facial expression recognition," in Proc. IEEE Int. Conf. Comput. Vision, 2011, pp. 1594-1601.
131. J. Wang, L. Yin, X. Wei, and Y. Sun, "3D facial expression recognition based on primitive surface feature distribution," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2006, pp. 1399-1406.
132. A. Maalej, B. Amor, M. Daoudi, A. Srivastava, and S. Berretti, "Shape analysis of local facial patches for 3D facial expression recognition," Pattern Recog., vol. 44, no. 8, pp. 1581-1589, 2011.
133. B. Gong, Y. Wang, J. Liu, and X. Tang, "Automatic facial expression recognition on a single 3D face by exploring shape deformation," in Proc. 17th ACMInt. Conf.Multimedia, 2009, pp. 569-572.
134. I. Cohen, N. Sebe, L. Chen, A. Garg, and T. S. Huang, "Facial expression recognition from video sequences: Temporal and static modelling," Comput. Vision Image Understanding, vol. 91, pp. 160-187, 2003.
135. I. Cohen, N. Sebe, F. G. Gozman, M. C. Cirelo, and T. S. Huang, "Learning Bayesian network classifiers for facial expression recognition both labeled and unlabeled data," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2003, pp. I-595-I-601.
136. M. Pardás and A. Bonafonte, "Facial animation parameters extraction and expression detection using HMM," Imaging Signal Process.: Image Commun. J., vol. 17, no. 9, pp. 675-688, 2002.
137. P. S. Aleksic and A. K. Katsaggelos, "Automatic facial expression recognition using facial animation parameters and multistream HMMs," IEEE Trans. Inf. Forensics Security, vol. 1, no. 1, pp. 3-11, Mar. 2006.
138. L. Yin, X. Wei, P. Longo, and A. Bhuvanesh, "Analyzing facial expressions using intensity-variant 3D data for human computer interaction," in Proc. Int. Conf. Pattern Recog., 2006, pp. 1248-1251.
139. Y.-L. Tian, T. Kanade, and J. F. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 97-115, Feb. 2001.
140. A. Dapogny, K. Bailly, and S. Dubuisson, "Dynamic facial expression recognition by joint static and multi-time gap transition classification," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2015, pp. 1-6.
141. S. Ramanathan, A. Kassim, Y. Venkatesh, and W. S. Wah, "Human facial expression recognition using a 3D morphable model," in Proc. IEEE Int. Conf. Image Process., 2006, pp. 661-664.
142. X. Zhao, D. Huang, E. Dellandréa, and L. Chen, "Automatic 3D facial expression recognition based on a Bayesian belief net and a statistical facial feature model," in Proc. Int. Conf. Pattern Recog., 2010, pp. 3724-3727.
143. X. Zhao, E. Dellandréa, J. Zou, and L. Chen, "A unified probabilistic framework for automatic 3D facial expression analysis based on a Bayesian belief inference and statistical feature models," Image Vision Comput., vol. 31, no. 3, pp. 231-245, 2013.
144. M. Ranzato, J. Susskind, V. Mnih, and G. Hinton, "On deep generative models with applications to recognition," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2011, pp. 2857-2864.
145. S. Rifai, Y. Bengio, A. Courville, P. Vincent, and M. Mirza, "Disentangling factors of variation for facial expression recognition," in Proc. Eur. Conf. Comput. Vision, 2012, vol. 7577, pp. 808-822.
146. M. Liu, S. Shan, R. Wang, and X. Chen, "Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2014, pp. 1749-1756.
147. S. E. Kahou, C. Pal, X. Bouthillier, P. Froumenty, Ç. Gülçehre, R. Memisevic, P. Vincent, A. Courville, Y. Bengio, R. C. Ferrari, M. Mirza, S. Jean, P.-L. Carrier, Y. Dauphin, N. Boulanger- Lewandowski, A. Aggarwal, J. Zumer, P. Lamblin, J.-P. Raymond, G. Desjardins, R. Pascanu, D. Warde-Farley, A. Torabi, A. Sharma, E. Bengio, M. C^oteé, and K. R. Konda, and Z. Wu, "Combining modality specific deep neural networks for emotion recognition in video," in Proc. Int. Conf.Multimodal Interaction, 2013, pp. 543-550.
148. I. Song, H.-J. Kim, and P. B. Jeon, "Deep learning for real-time robust facial expression recognition on a smartphone," in Proc. IEEE Int. Conf. Consumer Electron., 2014, pp. 564-567.
149. M. Liu, S. Li, S. Shan, and X. Chen, "AU-aware deep networks for facial expression recognition," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2013, pp. 1-6.
150. E. P. Ijjina and C. K. Mohan, "Facial expression recognition using Kinect depth sensor and convolutional neural networks," in Proc. 13th Int. Conf. Mach. Learning Appl., 2014, pp. 392-396.
151. S. He, S. Wang, W. Lan, H. Fu, and Q. Ji, "Facial expression recognition using deep Boltzmann machine from thermal infrared images," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2013, pp. 239-244.
152. P. Liu, S. Han, Z. Meng, and Y. Tong, "Facial expression recognition via a boosted deep belief network," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2014, pp. 1805-1812.
153. C. Wu, S. Wang, and Q. Ji, "Multi-instance hidden Markov model for facial expression recognition," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2015, pp. 1-6.
154. F. Tsalakanidou and S. Malassiotis, "Real-time 2d+ 3D facial action and expression recognition," Pattern Recog., vol. 43, no. 5, pp. 1763-1775, 2010.
155. F. Tsalakanidou and S. Malassiotis, "Robust facial action recognition from real-time 3D streams," in Proc. IEEE Conf. Comput. Vision Pattern Recog. Workshops, 2009, pp. 4-11.
156. N. Fragopanagos and J. G. Taylor, "Emotion recognition in human-computer interaction," Neural Netw., vol. 18, no. 4, pp. 389-405, 2005.
157. G. Caridakis, L. Malatesta, L. Kessous, N. Amir, A. Raouzaiou, and K. Karpouzis, "Modeling naturalistic affective states via facial and vocal expressions recognition," in Proc. Int. Conf. Multimodal Interaction, 2006, pp. 146-154.
158. J. Nicolle, V. Rapp, K. Bailly, L. Prevost, and M. Chetouani, "Robust continuous prediction of human emotions using multiscale dynamic cues," in Proc. Int. Conf. Multimodal Interaction, 2012, pp. 501-508.
159. A. Savran, H. Cao, M. Shah, A. Nenkova, and R. Verma, "Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering," in Proc. 14th ACM Int. Conf. Multimodal Interaction, 2012, pp. 485-492.
160. T. S. Huang, L. S. Chen, H. Tao, T. Miyasato, and R. Nakatsu, "Bimodal emotion recognition by man and machine," in Proc. ATR Workshop Virtual Commun. Environ., 1998, pp. 1-7.
161. H. Gunes and M. Piccardi, "Affect recognition from face and body: Early fusion vs. late fusion," in Proc. IEEE Conf. Syst., Man Cybern., 2005, pp. 3437-3443.
162. C. Busso, Z. Deng, S. Yildirim, M. Bulut, C.M. Lee, A. Kazemzadeh, S. Lee, U. Neumann, and S. Narayanan, "Analysis of emotion recognition using facial expressions, speech and multimodal information," in Proc. Int. Conf. Multimodal Interaction, 2004, pp. 205-211.
163. N. Sebe, I. Cohen, T. Gevers, and T. S. Huang, "Emotion recognition based on joint visual and audio cues," in Proc. Int. Conf. Pattern Recog., 2006, pp. 1136-1139.
164. Z. Zeng, J. Tu, B. M. Pianfetti, and T. S. Huang, "Audio-visual affective expression recognition through multistream fused HMM," IEEE Trans.Multimedia, vol. 10, no. 4, pp. 570-577, Jun. 2008.
165. L. Kessous, G. Castellano, and G. Caridakis, "Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis," J. Multimodal User Interfaces, vol. 3, nos. 1/2, pp. 33-48, 2010.
166. S. K. D'Mello and A. Graesser, "Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features," User Model. User-Adapted Interaction, vol. 20, no. 2, pp. 147-187, 2010.
167. Y. Yoshitomi, S.-I. Kim, T. Kawano, and T. Kilazoe, "Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face," in Proc. 9th IEEE Int. Workshop Robot Human Interactive Commun., 2000, pp. 178-183.
168. L. C. De Silva and P. C. Ng, "Bimodal emotion recognition," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2000, pp. 332-335.
169. C. Soladié, H. Salam, C. Pelachaud, N. Stoiber, and R. Séguier, "A multimodal fuzzy inference system using a continuous facial expression representation for emotion detection," in Proc. Int. Conf. Multimodal Interaction, 2012, pp. 493-500.
170. L. S. Chen, T. S. Huang, T. Miyasato, and R. Nakatsu, "Multimodal human emotion/expression recognition," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 1998, pp. 366-371.
171. P. Ekman and W. V. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Mountain View, CA, USA: Consulting Psychologists Press, 1978.
172. P. Ekman, W. V. Friesen, and J. C. Hager, Facial Action Coding System: The Manual on CD ROM. Salt Lake City, UT, USA: A Human Face, 2002.
173. W. V. Friesen and P. Ekman, "Emfacs-7: Emotional facial action coding system," unpublished manuscript, Univ. California, vol. 2, p. 36, 1983.
174. C. E. Izard, Maximally Discriminative Facial Movement Coding System (MAX). Newark, DE, USA: Instructional Resources Center, Univ. Delaware, 1983.
175. C. E. Izard, L. M. Dougherty, and E. A. Hembree, A System for Identifying Affect Expressions by Holistic Judgments. Newark, DE, USA: Instructional Resources Center, Univ. Delaware, 1983.
176. C. Zhang and Z. Zhang, "A survey of recent advances in face detection," Microsoft Res., Mountain View, CA, USA, Tech. Rep. MSR-TR-2010-66, 2010.
177. F. De la Torre and J. Cohn, "Facial expression analysis," in Visual Analysis of Humans. New York, NY, USA: Springer, 2011, pp. 377-409.
178. B. Wu, H. Ai, C. Huang, and S. Lao, "Fast rotation invariant multi-view face detection based on real adaboost," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2004, pp. 79-84.
179. H. V. Lakshmi and S. PatilKulakarni, "Segmentation algorithm for multiple face detection in color images with skin tone regions using color spaces and edge detection techniques," J. Comput. Theory Eng., vol. 2, no. 4, pp. 1793-8201, 2010.
180. N. Wang, X. Gao, D. Tao, and X. Li, "Facial feature point detection: A comprehensive survey," arXiv, 2014.
181. G. K. Tam, Z.-Q. Cheng, Y.-K. Lai, F. C. Langbein, Y. Liu, D. Marshall, R. R. Martin, X.-F. Sun, and P. L. Rosin, "Registration of 3D point clouds and meshes: A survey from rigid to nonrigid," IEEE Trans. Visual. Comput. Graphics, vol. 19, no. 7, pp. 1199-1217, Jul. 2013.
182. L. Igual, X. Perez-Sala, S. Escalera, C. Angulo, and F. De la Torre, "Continuous generalized procrustes analysis," Pattern Recog., vol. 47, no. 2, pp. 659-671, 2014.
183. M. Pantic and M. Bartlett, "Machine analysis of facial expressions," in Face Recognition. Vienna, Austria: I-Tech Educ. Publishing, 2007, pp. 377-416.
184. A. Martinez and S. Du, "A model of the perception of facial expressions of emotion by humans: Research overview and perspectives," J. Mach. Learning Res., vol. 13, no. 1, pp. 1589-1608, 2012.
185. A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, Z. Ambadar, K. M. Prkachin, and P. E. Solomon, "The painful face-pain expression recognition using active appearance models," Image Vision Comput., vol. 27, no. 12, pp. 1788-1796, 2009.
186. G. C. Littlewort, M. S. Bartlett, and K. Lee, "Automatic coding of facial expressions displayed during posed and genuine pain," Image Vision Comput., vol. 27, no. 12, pp. 1797-1803, 2009.
187. J. F. Cohn, T. S. Kruez, I. Matthews, Y. Yang, M. H. Nguyen, M. T. Padilla, F. Zhou, and F. De la Torre, "Detecting depression from facial actions and vocal prosody," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2009, pp. 1-7.
188. C. G. Kohler, E. A. Martin, N. Stolar, F. S. Barrett, R. Verma, C. Brensinger, W. Bilker, R. E. Gur, and R. C. Gur, "Static posed and evoked facial expressions of emotions in schizophrenia," Schizophr. Res., vol. 105, nos. 1-3, pp. 49-60, 2008.
189. G. E. Hinton, S. Osindero, and Y. W. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, 2006.
190. T. Kanade, J. F. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2000, pp. 46-53.
191. P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, "The extended Cohn-Kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression," in Proc. IEEE Conf. Comput. Vision Pattern Recog. Workshops, 2010, pp. 94-101.
192. M. Pantic, M. F. Valstar, R. Rademaker, and L. Maat, "Web-based database for facial expression analysis," in Proc. IEEE Int. Conf. Multimedia Expo, 2005, pp. 317-321.
193. I. R. Gross, R. Matthews, J. Cohn, T. Kanade, and S. Baker, "Multi-pie," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2008, pp. 1-8.
194. A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, "Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark," in Proc. IEEE Int. Conf. Comput. Vision Workshops, 2011, pp. 2106-2112.
195. A. Savran, N. Alyüz, H. Dibeklioǧlu, O. Çeliktutan, B. Gökberk, B. Sankur, and L. Akarun, "Bosphorus database for 3D face analysis," in Biometrics and Identity Management, vol. 5372. Berlin, Germany: Springer, 2008, pp. 47-56.
196. H. Nguyen, K. Kotani, F. Chen, and B. Le, "A thermal facial emotion database and its analysis," in Proc. 6th Pacific-Rim Symp. Image Video Technol., 2014, pp. 397-408.
197. I. Pavlidis, J. Dowdall, N. Sun, C. Puri, J. Fei, and M. Garbey, "Interacting with human physiology," Comput. Vision Image Understanding, vol. 108, no. 1, pp. 150-170, 2007.
198. S. Ioannou, V. Gallese, and A. Merla, "Thermal infrared imaging in psychophysiology: Potentialities and limits," Psychophysiology, vol. 51, no. 10, pp. 951-963, 2014.
199. L. Wu, S. L. Oviatt, and P. R. Cohen, "Multimodal integration - A statistical view," IEEE Trans. Multimedia, vol. 1, no. 4, pp. 334-341, Dec. 1999.
200. L. C. De Silva, T. Miyasato, and R. Nakatsu, "Facial emotion recognition using multi-modal information," in Proc. Int. Conf. Inf., Commun. Signal Process., 1997, pp. 397-401.
201. W.-J. Yan, Q. Wu, Y.-J. Liu, S.-J. Wang, and X. Fu, "CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2013, pp. 1-7.
202. S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, and J. F. Cohn, "DISFA: A spontaneous facial action intensity database," IEEE Trans. Affective Comput., vol. 4, no. 2, pp. 151-160, Apr.-Jun. 2013.
203. A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, "Acted facial expressions in the wild database," Australian Nat. Univ., Canberra, Australia, Tech. Rep. TR-CS-11-02, 2011.
204. D. McDuff, R. El Kaliouby, T. Senechal, M. Amr, J. F. Cohn, and R. Picard, "AMFED facial expression dataset: Naturalistic and spontaneous facial expressions collected "in-the-wild"," in Proc. IEEE Conf. Comput. Vision Pattern Recog. Workshops, 2013, pp. 881-888.
205. G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schröder, "The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent," IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 5-17, Jan.- Mar. 2012.
206. L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rosato, "A 3D facial expression database for facial behavior research," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2006, pp. 211-216.
207. L. Yin, X. Chen, Y. Sun, T. Worm, and M. Reale, "A highresolution 3D dynamic facial expression database," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2008, pp. 1-6.
208. (2016). Dataset 02: IRIS Thermal/Visible Face Database. [Online]. Available: http://vcipl-okstate.org/pbvs/bench/
209. (2016). [Online]. Available: http://www.equinoxsensors.com/
210. S. Wang, Z. Liu, S. Lv, Y. Lv, G. Wu, P. Peng, F. Chen, and X. Wang, "A natural visible and infrared facial expression database for expression recognition and emotion inference," IEEE Trans. Multimedia, vol. 12, no. 7, pp. 682-691, Nov. 2010.
211. M. Suwa, N. Sugie, and K. Fujimora, "A preliminary note on pattern recognition of human emotional expression," in Proc. 4th Int. Joint Conf. Pattern Recog., 1978, pp. 408-410.
212. K. Mase and A. Pentland, "Automatic lipreading by optical-flow analysis," Syst. Comput. Japan, vol. 22, pp. 67-76, 1991.
213. P. Ekman, T. S. Huang, T. J. Sejnowski, and J. C. Hager, "Final report to NSF of the planning workshop on facial expression understanding," Human Interaction Lab., Univ. California, San Francisco, CA, USA, Tech. Rep. HIL-0984, 1993.
214. A. Samal and P. A. Iyengar, "Automatic recognition and analysis of human faces and facial expressions: A survey," Pattern Recog., vol. 25, no. 1, pp. 65-77, 1992.
215. M. Pantic and L. J. M. Rothkrantz, "Automatic analysis of facial expressions: The state of the art," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 12, pp. 1424-1445, Dec. 2000.
216. B. Fasel and J. Luettin, "Automatic facial expression analysis: A survey," Pattern Recog., vol. 36, no. 1, pp. 259-275, 2003.
217. H. Soyel and H. Demirel, "Facial expression recognition using 3D facial feature distances," in Proc. 4th Int. Conf. Image Anal. Recog., 2007, pp. 831-838.
218. A. Savran, B. Sankur, and M. T. Bilge, "Comparative evaluation of 3D vs. 2d modality for automatic detection of facial action units," Pattern Recog., vol. 45, no. 2, pp. 767-782, 2012.
219. J. Wang and L. Yin, "Facial expression representation and recognition from static images using topographic context," Dept. Comput. Sci., State Univ. New York, Binghamton, NY, USA, 2005.
220. S. Lucey, A. B. Ashraf, and J. F. Cohn, Investigating Spontaneous Facial Action Recognition Through AAM Representations of the Face. Rijeka, Croatia: INTECH, 2007.
221. M. F. Valstar, M. Pantic, Z. Ambadar, and J. F. Cohn, "Spontaneous vs. posed facial behavior: Automatic analysis of brow actions," in Proc. Int. Conf. Multimodal Interaction, 2006, pp. 162-170.
222. Z. Zeng, Y. Fu, G. I. Roisman, Z. Wen, Y. Hu, and T. S. Huang, "Spontaneous emotional facial expression detection," J. Multimedia, vol. 1, no. 5, pp. 1-8, 2006.
223. R. El Kaliouby and P. Robinson, "Real-time inference of complex mental states from facial expressions and head gestures," in Real-Time Vision for Human-Computer Interaction. New York, NY, USA: Springer, 2005, pp. 181-200.
224. Q. Ji, P. Lan, and C. Looney, "A probabilistic framework formodeling and real-time monitoring human fatigue," IEEE Trans. Syst., Man, Cybern.A, Syst. Humans, vol. 36, no. 5, pp. 862-875, Sep. 2006.
225. G. C. Littlewort, M. S. Bartlett, and K. Lee, "Faces of pain: Automated measurement of spontaneousallfacial expressions of genuine and posed pain," in Proc. 9th Int. Conf. Multimodal Interaction, 2007, pp. 15-21.
226. G. M. Lucas, J. Gratch, S. Scherer, J. Boberg, and G. Stratou, "Towards an affective interface for assessment of psychological distress," Int. Conf. Affective Comput. and Intell. Interaction (ACII), Xi'an, 2015, pp. 539-545.
227. D. DeVault, R. Artstein, G. Benn, T. Dey, E. Fast, A. Gainer, K. Georgila, J. Gratch, A. Hartholt, M. Lhommet et al., "Simsensei kiosk: A virtual human interviewer for healthcare decision support," in Proc. Int. Conf. Auton. Agents Multi-Agent Syst., 2014, pp. 1061-1068.
228. J. F. Cohn and K. L. Schmidt, "The timing of facial motion in posed and spontaneous smiles," in Int. J. Wavelets Multiresolut Inf. Process., vol. 2, p. 121, 2004.
229. Z. Ambadar, J. F. Cohn, and L. I. Reed, "All smiles are not created equal: Morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous," J. Nonverbal Behav., vol. 33, no. 1, pp. 17-34, 2009.
230. K. M. Prkachin and P. E. Solomon, "The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain," Pain, vol. 139, no. 2, pp. 267-274, 2008.
231. Z. Hammal and J. F. Cohn, "Automatic detection of pain intensity," in Proc. Int. Conf. Multimodal Interaction, 2012, pp. 47-52.
232. M. F. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, "The first facial expression recognition and analysis challenge," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2011, pp. 921-926.
233. M. Valstar, J. Girard, T. Almaev, G. McKeown, M. Mehu, L. Yin, M. Pantic, and J. Cohn, "FERA 2015 - Second facial expression recognition and analysis challenge," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2015, pp. 1-8.
234. M. Pantic and L. J. Rothkrantz, "An expert system for recognition of facial actions and their intensity," in Proc. AAAI Int. Conf. Innovative Appl. Artif. Intell., 2000, pp. 1026-1033.
235. M. S. Bartlett, G. C. Littlewort, M. G. Frank, C. Lainscsek, I. R. Fasel, and J. R. Movellan, "Automatic recognition of facial actions in spontaneous expressions," J. Multimedia, vol. 1, no. 6, pp. 22-35, 2006.
236. A. Savran, B. Sankur, and M. Taha Bilge, "Estimation of facial action intensities on 2D and 3D data," in Proc. 19th Eur. Signal Process. Conf., 2011, pp. 1969-1973.
237. A. Savran, B. Sankur, and M. T. Bilge, "Regression-based intensity estimation of facial action units," Image Vision Comput., vol. 30, pp. 774-784, 2012.
238. J. M. Girard, J. F. Cohn, and F. De la Torre, "Estimating smile intensity: A better way," in Pattern Recog. Lett., vol. 66, pp. 13-21, 2015.
239. N. Zaker, M. H. Mahoor, W. Mattson, D. S. Messinger, and J. F. Cohn, "Intensity measurement of spontaneous facial actions: Evaluation of different image representations," in Proc. IEEE Int. Conf. Develop. Learning Epigenetic Robot., 2012, pp. 1-2.
240. G. Sandbach, S. Zafeiriou, and M. Pantic, "Markov random field structures for facial action unit intensity estimation," in Proc. IEEE Int. Conf. Comput. Vision Workshops, 2013, pp. 738-745.
241. Y. Li, S. M. Mavadati, M. H. Mahoor, and Q. Ji, "A unified probabilistic framework for measuring the intensity of spontaneous facial action units," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2013, pp. 1-7.
242. L. Jeni, J. M. Girard, J. F. Cohn, F. De La Torre, et al., "Continuous AU intensity estimation using localized, sparse facial feature space," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2013, pp. 1-7.
243. K. Shimada, Y. Noguchi, and T. Kurita, "Fast and robust smile intensity estimation by cascaded support vector machines," J. Comput. Theory Eng., vol. 5, no. 1, pp. 24-30, 2013.
244. A. Dhall and R. Goecke, "Group expression intensity estimation in videos via gaussian processes," in Proc. Int. Conf. Pattern Recog., 2012, pp. 3525-3528.
245. E. A. Haggard and K. S. Isaacs, "Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy," in Methods of Research in Psychotherapy. New York, NY, USA: Springer, 1966, pp. 154-165.
246. M. Shreve, S. Godavarthy, V. Manohar, D. Goldgof, and S. Sarkar, "Towards macro-and micro-expression spotting in video using strain patterns," in Proc. IEEE Workshop Appl. Comput. Vision, 2009, pp. 1-6.
247. M. Shreve, S. Godavarthy, D. Goldgof, and S. Sarkar, "Macroand micro-expression spotting in long videos using spatiotemporal strain," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2011, pp. 51-56.
248. T. Pfister, X. Li, G. Zhao, and M. Pietikäinen, "Recognising spontaneous facial micro-expressions," in Proc. IEEE Int. Conf. Comput. Vision, 2011, pp. 1449-1456.
249. S. Polikovsky, Y. Kameda, and Y. Ohta, "Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor," in Proc. 3rd Int. Conf. Crime Detection Prevention, 2009, pp. 1-6.
250. Q. Wu, X. Shen, and X. Fu, "The machine knows what you are hiding: An automatic micro-expression recognition system," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 152-162.
251. G. M. Lucas, J. Gratch, S. Scherer, J. Boberg, and G. Stratou, "Towards an affective interface for assessment of psychological distress," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2015, pp. 539-545.
252. J.-I. Biel, L. Teijeiro-Mosquera, and D. Gatica-Perez, "Facetube: Predicting personality from facial expressions of emotion in online conversational video," in Proc. Int. Conf. Multimodal Interaction, 2012, pp. 53-56.
253. D. Sanchez-Cortes, J.-I. Biel, S. Kumano, J. Yamato, K. Otsuka, and D. Gatica-Perez, "Inferring mood in ubiquitous conversational video," in Proc. 12th Int. Conf.Mobile UbiquitousMultimedia, 2013, p. 22.
254. J. R. Williamson, T. F. Quatieri, B. S. Helfer, G. Ciccarelli, and D. D. Mehta, "Vocal and facial biomarkers of depression based on motor incoordination and timing," in Proc. Int. Workshop Audio/ Visual Emotion Challenge, 2014, pp. 65-72.
255. M. Sidorov and W. Minker, "Emotion recognition and depression diagnosis by acoustic and visual features: A multimodal approach," in Proc. Int. Workshop Audio/Visual Emotion Challenge, 2014, pp. 81-86.
256. M. Senoussaoui, M. Sarria-Paja, J. F. Santos, and T. H. Falk, "Model fusion for multimodal depression classification and level detection," in Proc. Int. Workshop Audio/Visual Emotion Challenge, 2014, pp. 57-63.
257. V. Jain, J. L. Crowley, A. K. Dey, and A. Lux, "Depression estimation using audiovisual features and Fisher vector encoding," in Proc. Int. Workshop Audio/Visual Emotion Challenge, 2014, pp. 87-91.
258. J.-I. Biel, V. Tsiminaki, J. Dines, and D. Gatica-Perez, "Hi You- Tube!: Personality impressions and verbal content in social video," in Proc. Int. Conf. Multimodal Interaction, 2013, pp. 119-126.
259. M. K. Abadi, J. A. M. Correa, J. Wache, H. Yang, I. Patras, and N. Sebe, "Inference of personality traits and affect schedule by analysis of spontaneous reactions to affective videos," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2015, pp. 1-8.
260. L. M. Batrinca, N. Mana, B. Lepri, F. Pianesi, and N. Sebe, "Please, tell me about yourself: Automatic personality assessment using short self-presentations," in Proc. Int. Conf. Multimodal Interaction, 2011, pp. 255-262.
261. J.-I. Biel and D. Gatica-Perez, "The YouTube lens: Crowdsourced personality impressions and audiovisual analysis of vlogs," Multimedia, vol. 15, no. 1, pp. 41-55, 2013.
262. J. M. Girard, J. F. Cohn, M. A. Sayette, L. A. Jeni, and F. De la Torre, "Spontaneous facial expression can be measured automatically," Beh. Res. Methods, vol. 47, pp. 1136-1147, 2015.
263. T. Gehrig and H. K. Ekenel, "Why is facial expression analysis in the wild challenging?" in Proc. Int. Conf. Multimodal Interaction Workshops, 2013, pp. 9-16.
264. K. Sikka, K. Dykstra, S. Sathyanarayana, G. Littlewort, and M. Bartlett, "Multiple kernel learning for emotion recognition in the wild," in Proc. Int. Conf. Multimodal Interaction, 2013, pp. 517-524.
265. M. Liu, R. Wang, Z. Huang, S. Shan, and X. Chen, "Partial least squares regression on Grassmannian manifold for emotion recognition," in Proc. Int. Conf. Multimodal Interaction, 2013, pp. 525-530.
266. A. Dhall, R. Goecke, J. Joshi, K. Sikka, and T. Gedeon, "Emotion recognition in the wild challenge 2014: Baseline, data and protocol," in Proc. Int. Conf. Multimodal Interaction, 2014, pp. 461-466.
267. N. Bosch, S. D'Mello, R. Baker, J. Ocumpaugh, V. Shute, M. Ventura, L. Wang, and W. Zhao, "Automatic detection of learningcentered affective states in the wild," in Proc. 20th Int. Conf. Intell. User Interfaces, 2015, pp. 379-388.
268. A. Dhall, J. Joshi, K. Sikka, R. Goecke, and N. Sebe, "The more the merrier: Analysing the affect of a group of people in images," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2015.
269. D. McDuff, R. El Kaliouby, T. Senechal, D. Demirdjian, and R. Picard, "Automatic measurement of ad preferences from facial responses gathered over the internet," Image Vision Comput., vol. 32, no. 10, pp. 630-640, 2014.
270. L. I. Kuncheva, Combining Pattern Classifier: Methods and Algorithms. Hoboken, NJ, USA: Wiley, 2004.
271. J. A. Russell, J. Bachorowski, and J. Fernandez-Dols, "Facial and vocal expressions of emotion," Annu. Rev. Psychol., vol. 54, pp. 329-349, 2003.
272. G. Castellano, I. Leite, A. Pereira, C. Martinho, A. Paiva, and P. McOwan, "Multimodal affect modelling and recognition for empathic robot companions," Int. J. Humanoid Robot., vol. 10, no. 1, pp. 1350010, 2013.
273. H. P. Martínez and G. N. Yannakakis, "Mining multimodal sequential patterns: A case study on affect detection," in Proc. Int. Conf. Multimodal Interaction, 2011, pp. 3-10.
274. J. Van den Stock, R. Righart, and B. De Gelder, "Body expressions influence recognition of emotions in the face and voice," Emotion, vol. 7, no. 3, pp. 487-494, 2007.