Stress and anxiety detection using facial cues from videos

Biomedical Signal Processing and Control

Volumn 31, Issue , 2017, Pages 89-101

  Giannakakis, G ^a   Pediaditis, M ^a   Manousos, D ^a   Kazantzaki, E ^a   Chiarugi, F ^a   Simos, P G ^a,b   Marias, K ^a   Tsiknakis, M ^a,c  

^a INSTITUTE OF COMPUTER SCIENCE   (Greece)

^b UNIVERSITY OF CRETE   (Greece)

^c TECHNOLOGICAL EDUCATIONAL INSTITUTE OF CRETE   (Greece)

Author keywords

Anxiety; Blink rate; Emotion recognition; Facial cues; Head motion; Stress

Indexed keywords

CAMERAS; EYE MOVEMENTS; STRESSES;

ANXIETY; BLINK RATES; EMOTION RECOGNITION; FACIAL CUES; HEAD MOTION;

FACE RECOGNITION;

ANXIETY; ARTICLE; CAMERA; CLASSIFICATION; EMOTION ASSESSMENT; FACIAL EXPRESSION; FACIES; HEAD MOVEMENT; HEART RATE; HUMAN; MOUTH; OPTIC FLOW; PARAMETERS; PHOTOELECTRIC PLETHYSMOGRAPHY; PRIORITY JOURNAL; STRESS; VIDEORECORDING;

EID: 84979747864     PISSN: 17468094     EISSN: 17468108     Source Type: Journal    
DOI: 10.1016/j.bspc.2016.06.020     Document Type: Article

References (100)

1. [1] Schneiderman, N., Ironson, G., Siegel, S.D., Stress and health: psychological, behavioral, and biological determinants. Ann. Rev. Clin. Psychol., 1, 2005, 607.
2. [2] Selye, H., The stress syndrome. AJN Am. J. Nursing 65 (1965), 97–99.
3. [3] Vitetta, L., Anton, B., Cortizo, F., Sali, A., Mind-Body medicine: stress and its impact on overall health and longevity. Ann. N. Y. Acad. Sci. 1057 (2005), 492–505.
4. [4] R., Donatelle (2013). My Health: An Outcomes Approach, Pearson Education.
5. [5] Lazarus, R.S., Folkman, S., Stress, Appraisal, and Coping. 1986, Pringer, New York.
6. [6] Selye, H., The Stress of Life. 1956, McGraw-Hill, New York, NY, US.
7. [7] Esch, T., Fricchione, G.L., Stefano, G.B., The therapeutic use of the relaxation response in stress-related diseases. Med. Sci. Monitor 9 (2003), RA23–RA34.
8. [8] Palmer, S., Dryden, W., Counselling for Stress Problems. 1995, SAGE Publications Ltd.
9. [9] Shin, L.M., Liberzon, I., The neurocircuitry of fear, stress, and anxiety disorders. Neuropsychopharmacology 35 (2010), 169–191.
10. [10] Harrigan, J.A., O'Connell, D.M.O., How do you look when feeling anxious? Facial displays of anxiety. Personality Individ. Differences 21 (1996), 205–212.
11. [11] Understanding anxiety disorders and effective treatment, American Psychological Association, http://www.apapracticecentral.org/outreach/anxiety-disorders.pdf, June 2010.
12. [12] Gelder, M., Juan, J., Nancy, A., New Oxford Textbook of Psychiatry, vol. 1 & 2, 2004, Oxford University Press, Oxford.
13. [13] Helgoe, L.A., Wilhelm, L.R., Kommor, M.J., The Anxiety Answer Book. 2005, Sourcebooks Inc.
14. [14] Esch, T., Health in stress: change in the stress concept and its significance for prevention, health and life style. Gesundheitswesen (Bundesverband der Arzte des Offentlichen Gesundheitsdienstes (Germany)) 64 (2002), 73–81.
15. [15] Reiche, E.M.V., Nunes, S.O.V., Morimoto, H.K., Stress, depression, the immune system, and cancer. Lancet Oncol. 5 (2004), 617–625.
16. [16] Schwabe, L., Wolf, O.T., Oitzl, M.S., Memory formation under stress: quantity and quality. Neurosci. Biobehav. Rev. 34 (2010), 584–591.
17. [17] Dias-Ferreira, E., Sousa, J.C., Melo, I., Morgado, P., Mesquita, A.R., Cerqueira, J.J., Costa, R.M., Sousa, N., Chronic stress causes frontostriatal reorganization and affects decision-making. Science 325 (2009), 621–625.
18. [18] McEwen, B.S., Sapolsky, R.M., Stress and cognitive function. Curr. Opin. Neurobiol. 5 (1995), 205–216.
19. [19] Alonso, J., Romero, S., Ballester, M., Antonijoan, R., Mañanas, M., Stress assessment based on EEG univariate features and functional connectivity measures. Physiol. Meas., 36, 2015, 1351.
20. [20] Giannakakis, G., Grigoriadis, D., Tsiknakis, M., Detection of stress/anxiety state from EEG features during video watching. 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2015, 6034–6037.
21. [21] Lewis, R.S., Weekes, N.Y., Wang, T.H., The effect of a naturalistic stressor on frontal EEG asymmetry, stress, and health. Biol. Psychol. 75 (2007), 239–247.
22. [22] Nidal, K., Malik, A.S., EEG/ERP Analysis: Methods and Applications. 2014, CRC Press.
23. [23] E., Hoffmann (2005). Brain Training Against Stress. Mental Fitness & Forsknings center ApS Symbion Science Park.
24. [24] Boucsein, W., Electrodermal Activity. 2012, Springer Science & Business Media.
25. [25] Setz, C., Arnrich, B., Schumm, J., La Marca, R., Troster, G., Ehlert, U., Discriminating stress from cognitive load using a wearable EDA device. IEEE Trans. Inf. Technol. Biomed. 14 (2010), 410–417.
26. [26] de Santos Sierra, A., Ávila, C.S., Guerra Casanova, J., del Pozo, G.B., A stress-detection system based on physiological signals and fuzzy logic. IEEE Trans. Ind. Electron. 58 (2011), 4857–4865.
27. [27] Masaoka, Y., Homma, I., The effect of anticipatory anxiety on breathing and metabolism in humans. Respir. Physiol. 128 (2001), 171–177.
28. [28] Nilsen, K.B., Sand, T., Stovner, L.J., Leistad, R.B., Westgaard, R.H., Autonomic and muscular responses and recovery to one-hour laboratory mental stress in healthy subjects. BMC Musculoskelet Disord., 8, 2007, 1.
29. [29] Wijsman, J., Grundlehner, B., Penders, J., Hermens, H., Trapezius muscle EMG as predictor of mental stress. ACM Trans. Embedded Comput. Syst. (TECS) 12 (2013), 1–20.
30. [30] Healey, J.A., Picard, R.W., Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6 (2005), 156–166.
31. [31] Karthikeyan, P., Murugappan, M., Yaacob, S., EMG signal based human stress level classification using wavelet packet transform. Trends in Intelligent Robotics Automation, and Manufacturing, 2012, Springer, 236–243.
32. [32] Murray, I.R., Baber, C., South, A., Towards a definition and working model of stress and its effects on speech. Speech Commun. 20 (1996), 3–12.
33. [33] Wittels, P., Johannes, B., Enne, R., Kirsch, K., Gunga, H.-C., Voice monitoring to measure emotional load during short-term stress. Eur. J. Appl. Physiol. 87 (2002), 278–282.
34. [34] Sharma, N., Gedeon, T., Objective measures, sensors and computational techniques for stress recognition and classification: a survey. Comput. Methods Prog. Biomed. 108 (2012), 1287–1301.
35. [35] Darwin, C., The Expression of the Emotions in Man and Animals. 1998, Oxford University Press, USA.
36. [36] Sharma, N., Gedeon, T., Modeling observer stress for typical real environments. Expert Syst. Appl. 41 (2014), 2231–2238.
37. [37] Dinges, D.F., Rider, R.L., Dorrian, J., McGlinchey, E.L., Rogers, N.L., Cizman, Z., Goldenstein, S.K., Vogler, C., Venkataraman, S., Metaxas, D.N., Optical computer recognition of facial expressions associated with stress induced by performance demands. Aviat. Space Environ. Med. 76 (2005), 172–182.
38. [38] Hamilton, M., The assessment of anxiety-States by rating. Br. J. Med. Psychol. 32 (1959), 50–55.
39. [39] Liao, W., Zhang, W., Zhu, Z., Ji, Q., A real-time human stress monitoring system using dynamic bayesian network. Computer Society Conference in Computer Vision and Pattern Recognition IEEE, 2005 70–70.
40. [40] Hadar, U., Steiner, T., Grant, E., Rose, F.C., Head movement correlates of juncture and stress at sentence level. Lang. Speech 26 (1983), 117–129.
41. [41] Adams, A., Mahmoud, M., Baltrusaitis, T., Robinson, P., Decoupling facial expressions and head motions in complex emotions. Affective Computing and Intelligent Interaction (ACII), 2015 International Conference On, IEEE, 2015, 274–280.
42. [42] Argyle, M., Cook, M., Gaze and Mutual Gaze. 1976, Cambridge University Press, Cambridge, England.
43. [43] Harris, C.S., Thackray, R.I., Shoenber, R.W., Blink rate as a function of induced muscular tension and manifest anxiety. Percept. Motor Skill, 22, 1966 155-&.
44. [44] Leal, S., Vrij, A., Blinking during after lying. J. Nonverbal Behav. 32 (2008), 187–194.
45. [45] Mackintosh, J.H., Kumar, R., Kitamura, T., Blink rate in psychiatric-illness. Br. J. Psychiatry 143 (1983), 55–57.
46. [46] Chan, R.C.K., Chen, E.Y.H., Blink rate does matter: a study of blink rate, sustained attention, and neurological signs in schizophrenia. J. Nerv. Mental Dis. 192 (2004), 781–783.
47. [47] Ousler, G.W., Hagberg, K.W., Schindelar, M., Welch, D., Abelson, M.B., The ocular protection index. Cornea 27 (2008), 509–513.
48. [48] Taylor, J.A., The relationship of anxiety to the conditioned eyelid response. J. Exp. Psychol. 41 (1951), 81–92.
49. [49] Fox, E., Mathews, A., Calder, A.J., Yiend, J., Anxiety and sensitivity to gaze direction in emotionally expressive faces. Emotion 7 (2007), 478–486.
50. [50] Staab, J.P., The influence of anxiety on ocular motor control and gaze. Curr. Opin. Neurol. 27 (2014), 118–124.
51. [51] Laretzaki, G., Plainis, S., Vrettos, I., Chrisoulakis, A., Pallikaris, I., Bitsios, P., Threat and trait anxiety affect stability of gaze fixation. Biol. Psychol. 86 (2011), 330–336.
52. [52] Derakshan, N., Ansari, T.L., Hansard, M., Shoker, L., Eysenck, M.W., Anxiety, inhibition, efficiency, and effectiveness an investigation using the antisaccade task. Exp. Psychol. 56 (2009), 48–55.
53. [53] Holmes, A., Richards, A., Green, S., Anxiety and sensitivity to eye gaze in emotional faces. Brain Cognition 60 (2006), 282–294.
54. [54] Mogg, K., Garner, M., Bradley, B.P., Anxiety and orienting of gaze to angry and fearful faces. Biol. Psychol. 76 (2007), 163–169.
55. [55] Honma, M., Hyper-volume of eye-contact perception and social anxiety traits. Conscious. Cognition 22 (2013), 167–173.
56. [56] Simpson, H.M., Molloy, F.M., Effects of audience anxiety on pupil size. Psychophysiology, 8, 1971 491-&.
57. [57] Kimble, M.O., Fleming, K., Bandy, C., Kim, J., Zambetti, A., Eye tracking and visual attention to threating stimuli in veterans of the Iraq war. J. Anxiety Disorders 24 (2010), 293–299.
58. [58] Partala, T., Surakka, V., Pupil size variation as an indication of affective processing. Int. J. Hum.-Comput. Stud. 59 (2003), 185–198.
59. [59] Metaxas, D., Venkataraman, S., Vogler, C., Image-based stress recognition using a model-based dynamic face tracking system. Computational Science-ICCS 2004, 2004, Springer, 813–821.
60. [60] Liao, W., Zhang, W., Zhu, Z., Ji, Q., A Decision Theoretic Model for Stress Recognition and User Assistance. 2005, AAAI, 529–534.
61. [61] Kalra, P., Magnenat-Thalmann, N., Modeling of vascular expressions in facial animation. Computer Animation'94., Proceedings of IEEE, 1994, 50–58 (201).
62. [62] Gorman, J.M., Sloan, R.P., Heart rate variability in depressive and anxiety disorders. Am. Heart J. 140 (2000), S77–S83.
63. [63] Vrijkotte, T.G., Van Doornen, L.J., De Geus, E.J., Effects of work stress on ambulatory blood pressure, heart rate, and heart rate variability. Hypertension 35 (2000), 880–886.
64. [64] Katsis, C.D., Katertsidis, N.S., Fotiadis, D.I., An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders. Biomed. Signaling Process. 6 (2011), 261–268.
65. [65] Healey, J.A., Picard, R.W., Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6 (2005), 156–166.
66. [66] Steptoe, A., Willemsen, G., Natalie, O., Flower, L., Mohamed-Ali, V., Acute mental stress elicits delayed increases in circulating inflammatory cytokine levels. Clin. Sci. 101 (2001), 185–192.
67. [67] McDuff, D., Gontarek, S., Picard, R., Remote measurement of cognitive stress via heart rate variability. 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2014, 2957–2960.
68. [68] Schubert, C., Lambertz, M., Nelesen, R., Bardwell, W., Choi, J.-B., Dimsdale, J., Effects of stress on heart rate complexity—a comparison between short-term and chronic stress. Biol. Psychol. 80 (2009), 325–332.
69. [69] Friesen, W.V., Ekman, P., Emfacs-7: Emotional Facial Action Coding System. 1983, University of California at San Francisco Unpublished manuscript.
70. [70] Lerner, J.S., Dahl, R.E., Hariri, A.R., Taylor, S.E., Facial expressions of emotion reveal neuroendocrine and cardiovascular stress responses. Biol. Psychiatry 61 (2007), 253–260.
71. [71] Melfsen, S., Osterlow, J., Florin, I., Deliberate emotional expressions of socially anxious children and their mothers. J. Anxiety Disorders 14 (2000), 249–261.
72. [72] Ekman, P., Are there basic emotions. Psychol. Rev. 99 (1992), 550–553.
73. [73] Liu-sheng, W., Yue, X., Attention bias during processing of facial expressions in trait anxiety: an eye-tracking study. Electronics and Optoelectronics (ICEOE), 2011 International Conference On, IEEE, 2011 pp. V1-347-V341-350.
74. [74] Perkins, A.M., Inchley-Mort, S.L., Pickering, A.D., Corr, P.J., Burgess, A.P., A facial expression for anxiety. J. Pers. Soc. Psychol., 102, 2012, 910.
75. [75] Vanin, J.R., Overview of anxiety and the anxiety disorders. Anxiety Disorders, 2008, Springer, 1–18.
76. [76] Hirokawa, K., Yagi, A., Miyata, Y., An examination of the effects of linguistic abilities on communication stress measured by blinking and heart rate, during a telephone situation. Soc. Behav. Personality 28 (2000), 343–353.
77. [77] Lang, P.J., Bradley, M.M., Cuthbert, B.N., International Affective Picture System (iaps): Technical Manual and Affective Ratings. 1997, NIMH Center for the Study of Emotion and Attention, 39–58.
78. [78] Stroop, J.R., Studies of interference in serial verbal reactions. J. Exp. Psychol. 18 (1935), 643–662.
79. [79] Yang, M.H., Kriegman, D.J., Ahuja, N., Detecting faces in images: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 24 (2002), 34–58.
80. [80] C. Zhang, Z. Zhang, A survey of recent advances in face detection, In: Tech. rep., Microsoft Research, 2010.
81. [81] Viola, P., Jones, M., Rapid object detection using a boosted cascade of simple features. Computer Vision and Pattern Recognition (CVPR) 2001 IEEE, 2001, 511–518.
82. [82] C.Z. Zhang, Z. Zhang, A Survey of Recent Advances in Face Detection, In: Microsoft Research Technical Report, 2010.
83. [83] Cootes, T.F., Edwards, G.J., Taylor, C.J., Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001), 681–685.
84. [84] Matthews, I., Baker, S., Active appearance models revisited. Int. J. Comput. Vision 60 (2004), 135–164.
85. [85] Gower, J.C., Generalized procrustes analysis. Psychometrika 40 (1975), 33–51.
86. [86] Thejaswi, N., Sengupta, S., Lip localization and viseme recognition from video sequences. Fourteenth National Conference on Communications, India, 2008.
87. [87] Farnebäck, G., Two-frame motion estimation based on polynomial expansion. Image Analysis, 2003, Springer, 363–370.
88. [88] Poh, M., McDuff, D., Picard, R., Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18 (2010), 10762–10774.
89. [89] Kranjec, J., Beguš, S., Geršak, G., Drnovšek, J., Non-contact heart rate and heart rate variability measurements: a review. Biomed. Signal Process. 13 (2014), 102–112.
90. [90] Felinger, A., Data Analysis and Signal Processing in Chromatography. 1998, Elsevier.
91. [91] Yang, C., He, Z., Yu, W., Comparison of public peak detection algorithms for MALDI mass spectrometry data analysis. BMC Bioinf., 10, 2009.
92. [92] Girden, E.R., ANOVA. Repeated Measures. 1992, Sage.
93. [93] Holmgard, C., Yannakakis, G.N., Karstoft, K.-I., Andersen, H.S., Stress detection for ptsd via the startlemart game. Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference On, IEEE, 2013, 523–528.
94. [94] Yannakakis, G.N., Hallam, J., Ranking vs. preference: a comparative study of self-reporting. Affective Computing and Intelligent Interaction, 2011, Springer, 437–446.
95. [95] Pudil, P., Novovičová, J., Kittler, J., Floating search methods in feature selection. Pattern Recogn. Lett. 15 (1994), 1119–1125.
96. [96] Fawcett, T., An introduction to ROC analysis. Pattern Recogn. Lett. 27 (2006), 861–874.
97. [97] Dash, M., Liu, H., Feature selection for classification. Intell. Data Anal. 1 (1997), 131–156.
98. [98] Gao, H., Yuce, A., Thiran, J.-P., Detecting emotional stress from facial expressions for driving safety. Image Processing (ICIP) 2014 IEEE International Conference On, IEEE, 2014, 5961–5965.
99. [99] Zhai, J., Barreto, A., Stress recognition using non-invasive technology. FLAIRS Conference, 2006, 395–401.
100. [100] Barreto, A., Zhai, J., Adjouadi, M., Non-intrusive physiological monitoring for automated stress detection in human-computer interaction. Human–Computer Interaction, 2007, Springer, 29–38.