Estimating smile intensity: A better way

Pattern Recognition Letters

Volumn 66, Issue , 2015, Pages 13-21

  Girard, Jeffrey M ^a   Cohn, Jeffrey F ^a,b   De La Torre, Fernando ^b  

^a UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Facial Action Coding System; Facial expression; Nonverbal behavior; Smile intensity; Support vector machines

Indexed keywords

BINS; CLASSIFICATION (OF INFORMATION); FEATURE EXTRACTION; GESTURE RECOGNITION; REGRESSION ANALYSIS; SUPPORT VECTOR MACHINES;

CLASSIFIER DECISIONS; DIMENSIONALITY REDUCTION; EXPRESSION INTENSITIES; FACIAL ACTION CODING SYSTEM; FACIAL EXPRESSIONS; INTENSITY ESTIMATION; NONVERBAL BEHAVIOR; SMILE INTENSITY;

FACE RECOGNITION;

EID: 84943187292     PISSN: 01678655     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patrec.2014.10.004     Document Type: Article

References (90)

1. C. Darwin The Expression of Emotions in Man and Animals 3rd ed. 1872 Oxford University New York
2. P. Ekman, W.V. Friesen, and S. Ancoli Facial signs of emotional experience J. Pers. Soc. Psychol. 39 6 1980 1125 1134
3. Z. Zeng, M. Pantic, G.I. Roisman, and T.S. Huang A survey of affect recognition methods: audio, visual, and spontaneous expressions IEEE Trans. Pattern Anal. Mach. Intell. 31 1 2009 39 58
4. A.J. Fridlund The behavioral ecology and sociality of human faces M.S. Clark, Review of Personality Social Psychology 1992 Sage Publications 90 121
5. D. Keltner Evidence for the distinctness of embarrassment, shame, and guilt: a study of recalled antecedents and facial expressions of emotion Cogn. Emotion 10 2 1996 155 172
6. G.C. Littlewort, M.S. Bartlett, and K. Lee Automatic coding of facial expressions displayed during posed and genuine pain Image Vis. Comput. 27 12 2009 1797 1803
7. K.M. Prkachin, and P.E. Solomon The structure, reliability and validity of pain expression: evidence from patients with shoulder pain Pain 139 2 2008 267 274
8. J.F. Cohn, T.S. Kruez, I. Matthews, Y. Ying, N. Minh Hoai, M.T. Padilla, Z. Feng, and F. De la Torre Detecting depression from facial actions and vocal prosody Proceedings of the IEEE International Conference on Affective Computing and Intelligent Interaction 2009 1 7 Amsterdam
9. J.M. Girard, J.F. Cohn, M.H. Mahoor, S. Mavadati, and D.P. Rosenwald Social risk and depression: evidence from manual and automatic facial expression analysis IEEE International Conference on Automatic Face & Gesture Recognition 2013
10. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J.G. Taylor Emotion recognition in human-computer interaction IEEE Signal Process. Mag. 18 1 2001 32 80
11. M. Pantic, and L.J.M. Rothkrantz Toward an affect-sensitive multimodal human-computer interaction Proc. IEEE 91 9 2003 1370 1390
12. I.S. Pandzic, and R. Forchheimer The Origins of the MPEG-4 Facial Animation Standard Wiley Online Library 2002
13. J.F. Cohn, and P. Ekman Measuring facial action by manual coding, facial EMG, and automatic facial image analysis J.A. Harrigan, R. Rosenthal, K.R. Scherer, The New Handbook of Nonverbal Behavior Research 2005 Oxford University Press New York, NY 9 64
14. P. Ekman Basic emotions T. Dalagleish, M. Power, Handbook of Cognition and Emotion 1999 John Wiley & Sons UK 45 60
15. C.E. Izard The Face of Emotion 1971 Appleton-Century-Crofts New York, NY
16. P. Ekman Darwin, deception, and facial expression Ann. N. Y. Acad. Sci. 1000 1 2003 205 221
17. L.F. Barrett, B. Mesquita, and M. Gendron Context in emotion perception Curr. Dir. Psychol. Sci. 20 5 2011 286 290
18. P. Ekman, W.V. Friesen, and J. Hager Facial Action Coding System: A Technique for the Measurement of Facial Movement 2002 Research Nexus Salt Lake City, UT
19. P. Ekman, and E.L. Rosenberg What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS) 2nd ed. 2005 Oxford University Press New York, NY
20. P. Ekman, R.J. Davidson, and W.V. Friesen The Duchenne smile: emotional expression and brain physiology: II J. Pers. Soc. Psychol. 58 2 1990 342 353
21. M.G. Frank, P. Ekman, and W.V. Friesen Behavioral markers and recognizability of the smile of enjoyment J. Pers. Soc. Psychol. 64 1 1993 83 93
22. L.I. Reed, M.A. Sayette, and J.F. Cohn Impact of depression on response to comedy: a dynamic facial coding analysis J. Abnorm. Psychol. 116 4 2007 804 809
23. A.J. Fridlund Sociality of solitary smiling: potentiation by an implicit audience J. Pers. Soc. Psychol. 60 2 1991 12
24. U. Hess, R. Banse, and A. Kappas The intensity of facial expression is determined by underlying affective state and social situation J. Pers. Soc. Psychol. 69 2 1995 280 288
25. E.L. Abel, and M.L. Kruger Smile intensity in photographs predicts longevity Psychol. Sci. 21 4 2010 542 544
26. L. Harker, and D. Keltner Expressions of positive emotion in Women's College yearbook pictures and their relationship to personality and life outcomes across adulthood J. Pers. Soc. Psychol. 80 1 2001 112 124
27. M. Hertenstein, C. Hansel, A. Butts, and S. Hile Smile intensity in photographs predicts divorce later in life Motiv. Emotion 33 2 2009 99 105
28. C. Oveis, J. Gruber, D. Keltner, J.L. Stamper, and W.T. Boyce Smile intensity and warm touch as thin slices of child and family affective style Emotion 9 4 2009 544 548
29. J.P. Seder, and S. Oishi Intensity of smiling in Facebook photos predicts future life satisfaction Soc. Psychol. Pers. Sci. 3 4 2012 407 413
30. Z. Ambadar, J.F. Cohn, and L.I. Reed All smiles are not created equal: Morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous J. Nonverbal Behav. 33 1 2009 17 34
31. E. Krumhuber, A.S. Manstead, and A. Kappas Temporal aspects of facial displays in person and expression perception: the effects of smile dynamics, head-tilt, and gender J. Nonverbal Behav. 31 1 2007 39 56
32. R. Mergl, M. Vogel, P. Mavrogiorgou, C. Gobel, M. Zaudig, U. Hegerl, and G. Juckel Kinematical analysis of emotionally induced facial expressions in patients with obsessive-compulsive disorder Psychol. Med. 33 8 2003 1453 1462
33. R. Mergl, P. Mavrogiorgou, U. Hegerl, and G. Juckel Kinematical analysis of emotionally induced facial expressions: a novel tool to investigate hypomimia in patients suffering from depression J. Neurol. Neurosurg. Psychiatry 76 1 2005 138 140
34. G. Juckel, R. Mergl, A. Prassl, P. Mavrogiorgou, H. Witthaus, H.J. Moller, and U. Hegerl Kinematic analysis of facial behaviour in patients with schizophrenia under emotional stimulation by films with "Mr. Bean" Eur. Arch. Psychiatry Clin. Neurosci. 258 3 2008 186 191
35. M.F. Valstar, and M. Pantic Fully Automatic Facial Action Unit Detection and Temporal Analysis Conference on Computer Vision and Pattern Recognition Workshops 2006
36. M.F. Valstar, and M. Pantic Fully automatic recognition of the temporal phases of facial actions IEEE Trans. Syst. Man Cybern. 42 1 2012 28 43
37. S.W. Chew, P. Lucey, S. Lucey, J. Saragih, J.F. Cohn, I. Matthews, and S. Sridharan In the pursuit of effective affective computing: the relationship between features and registration IEEE Trans. Syst. Man Cybern. 42 4 2012 1006 1016
38. M.S. Bartlett, G. Littlewort, B. Braathen, T.J. Sejnowski, and J.R. Movellan A prototype for automatic recognition of spontaneous facial actions S. Becker, K. Obermayer, Advances in Neural Information Processing Systems 2003 MIT Press
39. M.S. Bartlett, G. Littlewort, M.G. Frank, C. Lainscsek, I.R. Fasel, and J.R. Movellan Automatic recognition of facial actions in spontaneous expressions J. Multimedia 1 6 2006 22 35
40. M.S. Bartlett, G. Littlewort, M.G. Frank, C. Lainscsek, I.R. Fasel, and J.R. Movellan Fully automatic facial action recognition in spontaneous behavior IEEE International Conference on Automatic Face & Gesture Recognition 2006 223 230 Southampton
41. P. Yang, L. Qingshan, and D.N. Metaxas RankBoost with l1 regularization for facial expression recognition and intensity estimation IEEE International Conference on Computer Vision 2009 1018 1025
42. G. Littlewort, M.S. Bartlett, I.R. Fasel, J. Susskind, and J.R. Movellan Dynamics of facial expression extracted automatically from video Image Vis. Comput. 24 6 2006 615 625
43. J. Reilly, J. Ghent, and J. McDonald Investigating the dynamics of facial expression Adv. Vis. Comput. 2006 334 343
44. S. Koelstra, and M. Pantic Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics IEEE International Conference on Automatic Face & Gesture Recognition 2008 1 8
45. J. Whitehill, G. Littlewort, I.R. Fasel, M.S. Bartlett, and J.R. Movellan Toward practical smile detection IEEE Trans. Pattern Anal. Mach. Intell. 31 11 2009 2106 2111
46. A. Savran, B. Sankur, and M. Taha Bilge Regression-based intensity estimation of facial action units Image Vis. Comput. 2011
47. K. Shimada, T. Matsukawa, Y. Noguchi, and T. Kurita Appearance-based smile intensity estimation by cascaded support vector machines ACCV 2010 Workshops 2011 277 286
48. K. Shimada, Y. Noguchi, and T. Kurita Fast and robust smile intensity estimation by cascaded support vector machines Int. J. Comput. Theory Eng. 5 1 2013 24 30
49. L. Ka Keung, and X. Yangsheng Real-time estimation of facial expression intensity IEEE International Conference on Robotics and Automation Vol. 2 2003 2567 2572
50. A. Dhall, and R. Goecke Group expression intensity estimation in videos via Gaussian processes International Conference on Pattern Recognition 2012 3525 3528
51. S. Kaltwang, O. Rudovic, and M. Pantic Continuous pain intensity estimation from facial expressions G. Bebis, R. Boyle, B. Parvin, D. Koracin, C. Fowlkes, S. Wang, M.H. Choi, S. Mantler, J. Schulze, D. Acevedo, K. Mueller, M. Papka, Advances in Visual Computing Lecture Notes in Computer Science Volume 7432 2012 Springer Berlin, Heidelberg 368 377
52. L.A. Jeni, J.M. Girard, J.F. Cohn, and F. De la Torre Continuous AU intensity estimation using localized, sparse facial feature space Proceedings of the IEEE International Conference on Automated Face & Gesture Recognition Workshops 2013 IEEE 1 7
53. M.H. Mahoor, S. Cadavid, D.S. Messinger, and J.F. Cohn A framework for automated measurement of the intensity of non-posed Facial Action Units Computer Vision and Pattern Recognition Workshops 2009 74 80 Miami
54. D.S. Messinger, M.H. Mahoor, S.M. Chow, and J.F. Cohn Automated measurement of facial expression in infant-mother interaction: a pilot study Infancy 14 3 2009 285 305
55. S.M. Mavadati, M.H. Mahoor, K. Bartlett, P. Trinh, and J.F. Cohn DISFA: a spontaneous facial action intensity database IEEE Trans. Affect. Comput. 4 2013 151 160
56. J.F. Cohn, and K.L. Schmidt The timing of facial motion in posed and spontaneous smiles Int. J. Wavelets Multiresolution Inform. Process. 2 2 2004 57 72
57. O. Deniz, M. Castrillon, J. Lorenzo, L. Anton, and G. Bueno Smile detection for user interfaces G. Bebis, R. Boyle, B. Parvin, D. Koracin, P. Remagnino, F. Porikli, J. Peters, J. Klosowski, L. Arns, Y.K. Chun, T.M. Rhyne, L. Monroe, Advances in Visual Computing Lecture Notes in Computer Science Volume 5359 2008 Springer Berlin Heidelberg Berlin, Heidelberg 602 611
58. D.S. Messinger, T.D. Cassel, S.I. Acosta, Z. Ambadar, and J.F. Cohn Infant smiling dynamics and perceived positive emotion J. Nonverbal Behav. 32 3 2008 133 155
59. J.B. Bavelas, and N. Chovil Faces in dialogue J.A. Russell, J.M. Fernandez-Dols, The Psychology of Facial Expression 1997 Cambridge University Press New York 334 346
60. U. Hess, S. Blairy, and R.E. Kleck The influence of facial emotion displays, gender, and ethnicity on judgments of dominance and affiliation J. Nonverbal Behav. 24 4 2000 265 283
61. U. Hess, R.B. Adams Jr, and R.E. Kleck Who may frown and who should smile? Dominance, affiliation, and the display of happiness and anger Cognit. Emotion 19 4 2005 515 536
62. X. Zhang, L. Yin, J.F. Cohn, S. Canavan, M. Reale, A. Horowitz, P. Liu, and J.M. Girard BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database Image Vis. Comput. 32 10 2014 692 706 10.1016/j.imavis.2014.06.002
63. American Psychiatric Association Diagnostic and Statistical Manual of Mental Disorders 4th ed. 1994 American Psychiatric Association Washington, DC
64. M. Hamilton Development of a rating scale for primary depressive illness Br. J. Soc. Clin. Psychol. 6 4 1967 278 296
65. C.J. van Rijsbergen Information Retrieval 2nd ed. 1979 Butterworth London
66. P.E. Shrout, and J.L. Fleiss Intraclass correlations: uses in assessing rater reliability Psychol. Bull. 86 2 1979 420
67. Image Metrics, Live Driver SDK, 2013, URL: http://image-metrics.com/.
68. T.F. Cootes, G.J. Edwards, and C.J. Taylor Active appearance models IEEE Trans. Pattern Anal. Mach. Intell. 23 6 2001 681 685
69. I. Matthews, and S. Baker Active appearance models revisited Int. J. Comput. Vis. 60 2 2004 135 164
70. J.G. Daugman Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression IEEE Trans. Acoust. Speech Sig. Process. 36 7 1988 1169 1179
71. W. Fellenz, J.G. Taylor, N. Tsapatsoulis, and S. Kollias Comparing template-based, feature-based and supervised classification of facial expressions from static images Proceedings of Circuits, Systems, Communications and Computers 1999
72. J.P. Jones, and L.A. Palmer An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex J. Neurophysiol. 58 6 1987 1233 1258
73. C. Liu, S. Louis, and H. Wechsler A Gabor feature classifier for face recognition IEEE International Conference on Computer Vision 2001 270 275
74. D.G. Lowe Object recognition from local scale-invariant features IEEE International Conference on Computer Vision 1999 1150 1157
75. A. Vedali, B. Fulkerson, VLFeat: An open and portable library of computer vision algorithms, in: Proceedings of the ACM International Conference on Multimedia, 2010, pp. 1469-1472.
76. M. Belkin, and P. Niyogi Laplacian eigenmaps for dimensionality reduction and data representation Neural Comput. 15 6 2003 1373 1396
77. D. Cai, X. He, W.V. Zhang, and J. Han Regularized locality preserving indexing via spectral regression Conference on Information and Knowledge Management 2007 ACM 741 750
78. I. Jolliffe Principal component analysis Encyclopedia of Statistics in Behavioral Science 2005 John Wiley & Sons, Ltd.
79. B. Fasel, and J. Luettin Automatic facial expression analysis: a survey Pattern Recognit. 36 1 2003 259 275
80. V. Vapnik The Nature of Statistical Learning Theory 1995 Springer New York, NY
81. C.W. Hsu, C.C. Chang, and C.J. Lin A practical guide to support vector classification Technical Report 2003
82. J.C. Platt Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods A.J. Smola, P.J. Bartlett, Advances in large margin classifiers 1999 MIT Press 61 74
83. M. Tipping Sparse Bayesian learning and the relevance vector machine J. Mach. Learn. Res. 1 2001 211 244
84. C.W. Hsu, and C.J. Lin A comparison of methods for multiclass support vector machines IEEE Trans. Neural Netw. 13 4 2002 415 425
85. S. Geisser Predictive Inference 1993 Chapman and Hall New York, NY
86. L.A. Jeni, J.F. Cohn, and F. De la Torre Facing imbalanced data: recommendations for the use of performance metrics International Conference on Affective Computing and Intelligent Interaction 2013
87. T. Fawcett An introduction to ROC analysis Pattern Recognit. Lett. 27 8 2006 861 874
88. IBM Corp, IBM SPSS Statistics for Windows, Version 21.0, 2012.
89. M. Frigge, D. Hoaglin, and B. Iglewicz Some implementations of the boxplot Am. Stat. 43 1 1989 50 54
90. J.M. Girard, J.F. Cohn, M.H. Mahoor, S.M. Mavadati, Z. Hammal, and D.P. Rosenwald Nonverbal social withdrawal in depression: evidence from manual and automatic analyses Image Vis. Comput. 32 10 2014 641 647 10.1016/j.imavis.2013.12.007