Automatic temporal segment detection and affect recognition from face and body display

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 39, Issue 1, 2009, Pages 64-84

  Gunes, Hatice ^a   Piccardi, Massimo ^a  

^a UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

Author keywords

Affect recognition; Affective face and body display; Phase synchronization; Selective fusion; Temporal segment detection

Indexed keywords

DYNAMICS; PHYSIOLOGY; SYNCHRONIZATION;

AFFECT RECOGNITION; AFFECTIVE FACE AND BODY DISPLAY; PHASE SYNCHRONIZATION; SELECTIVE FUSION; TEMPORAL SEGMENT DETECTION;

FACE RECOGNITION;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; FACIAL EXPRESSION; GESTURE; HUMAN; IMAGE PROCESSING; METHODOLOGY; MOVEMENT (PHYSIOLOGY); NORMAL DISTRIBUTION; PHYSIOLOGY; RECOGNITION; TIME; VIDEORECORDING;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; FACIAL EXPRESSION; GESTURES; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MOVEMENT; NORMAL DISTRIBUTION; PATTERN RECOGNITION, AUTOMATED; RECOGNITION (PSYCHOLOGY); TIME FACTORS; VIDEO RECORDING;

EID: 61549119152     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2008.927269     Document Type: Article

References (81)

1. The Facial Recognition Technology (FERET) Database. Online. Available: http://www.itl.nist.gov/iad/humanid/feret/feret_master.html
2. J. Allman, J. T. Cacioppo, R. J. Davidson, P. Ekman, W. V. Friesen, C. E. Izard, and M. Phillips, NSF Report-Facial Expression Understanding 2003. Report. Online. Available: http://face-and-emotion.com data face nsfrept basicℬience.html
3. N. Ambady and R. Rosenthal, "Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis," Psychol. Bull., vol. 11, no. 2, pp. 256-274, 1992.
4. M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, "A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking," IEEE Trans. Signal Process., vol. 50, no. 2, pp. 174-188, Feb. 2002.
5. T. Balomenos, A. Raouzaiou, S. Ioannou, A. Drosopoulos, and K. Karpouzis, "Emotion analysis in man-machine interaction systems," in Proc. Workshop Multimodal Interaction Related Mach. Learn. Algorithms 2004, pp. 318-328.
6. M. S. Bartlett, G. Littlewort, M. G. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Fully automatic facial action recognition in spontaneous behavior," in Proc. Int. Conf. Comput. Vis. Pattern Recognit., 2006, pp. 223-230.
7. M. S. Bartlett, G. C. Littlewort, M. G. Frank, C. Lainscsek, I. Fasel, and J. R. Movellan, "Automatic recognition of facial actions in spontaneous expressions," J. Multimedia, vol. 1, no. 6, pp. 22-35, Sep. 2006.
8. S. Bengio, "Multimodal speech processing using asynchronous hidden Markov models," Inf. Fusion, vol. 5, no. 2, pp. 81-89, Jun. 2004.
9. T. Bänziger and K. R. Scherer, "Using actor portrayals to systematically study multimodal emotion expression: The GEMEP corpus," in Proc. 2nd Int. Conf. Affective Comput. Intell. Interaction, 2007, pp. 476-487.
10. G. R. Bradski, "Computer vision face tracking for use in a perceptual user interface," Intel Technol. J., 2nd Quarter 1998.
11. J. K. Burgoon, M. L. Jensen, T. O. Meservy, J. Kruse, and J. F. Nunamaker, "Augmenting human identification of emotional states in video," in Proc. Int. Conf. Intell. Data Anal., 2005. Online. Available: https://analysis.mitre.org/proceedings/Final_Papers_Files/ 344_Camera_Ready_Paper.pdf
12. A. Camurri, I. Lager, and G. Volpe, "Recognizing emotion from dance movement: Comparison of spectator recognition and automated techniques," Int. J. Hum.-Comput. Stud., vol. 59, no. 1/2, pp. 213-225, Jul. 2003.
13. Y. Chen, Y. Rui, and T. S. Huang, "Multicue HMM-UKF for real-time contour tracking," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 9, pp. 1525-1529, Sep. 2006.
14. G. S. Christensen and S. A. Soliman, "A new technique for linear static state estimation based on weighted least absolute value approximations," J. Optim. Theory Appl., vol. 61, no. 1, pp. 123-136, Apr. 1989.
15. I. Cohen et al., "Facial expression recognition from video sequences: Temporal and static modeling," Comput. Vis. Image Underst., vol. 91, no. 1, pp. 160-187, Jul. 2003.
16. J. F. Cohn, J. F. Reed, Z. Ambadar, J. Xiao, and T. Moriyama, "Automatic analysis and recognition of brow actions in spontaneous facial behavior," in Proc. IEEE Int. Conf. Syst., Man Cybern., 2004, vol. 1, pp. 610-616.
17. J. F. Cohn, L. I. Reed, T.Moriyama, X. Jing, K. Schmidt, and Z. Ambadar, "Multimodal coordination of facial action, head rotation, and eye motion during spontaneous smiles," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2004, pp. 129-135.
18. A. Corradini, M. Mehta, N. O. Bernsen, and J.-C. Martin, "Multimodal input fusion in human computer interaction on the example of the on-going nice project," in Proc. NATO-ASI Conf. Data Fus. Situation Monitoring, Incident Detection, Alert Response Manage., 2003, pp. 223-234.
19. M. Coulson, "Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence," Nonverbal Behav. vol. 28, no. 2, pp. 117-139, Jun. 2004.
20. E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, L. Lowry, M. McRorie, L. Jean-Claude Martin, J.-C. Devillers, A. Abrilian, S. Batliner, A. Noam, and K. Karpouzis, "The humaine database: Addressing the needs of the affective computing community," in Proc. 2nd Int. Conf. Affective Comput. Intell. Interaction, 2007, pp. 488-500.
21. P. Dreuw, T. Deselaers, D. Rybach, D. Keysers, and H. Ney, "Tracking using dynamic programming for appearance-based sign language recognition," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog. 2006, pp. 293-298.
22. J. Driver and C. Spence, "Multisensory perception: Beyond modularity and convergence," Curr. Biol., vol. 10, no. 20, pp. 731-735, Oct. 2000.
23. P. Ekman, "About brows: Emotional and conversational signals," in Human Ethology: Claims and Limits of a New Discipline: Contributions to the Colloquium, M. V. Cranach, K. Foppa, W. Lepenies, and D. Ploog, Eds. New York: Cambridge Univ. Press, 1979, pp. 169-248.
24. P. Ekman and W. V. Friesen, Unmasking the Face: A Guide to Recognizing Emotions From Facial Clues. Englewood Cliffs, NJ: Prentice-Hall, 1975.
25. A. Elgammal, V. Shet, Y. Yacoob, and L. S. Davis, "Learning dynamics for exemplar-based gesture recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2003, pp. 571-578.
26. S. D. Villalba, G. Castellano, and A. Camurri, "Recognising human emotions from body movement and gesture dynamics," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2007, pp. 71-82.
27. H. Gu and Q. Ji, "An automated face reader for fatigue detection," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2004, pp. 111-116.
28. H. Gunes, "Vision-based multimodal analysis of affective face and upper-body behaviour," Ph.D. dissertation, Univ. Technol. Sydney (UTS), Sydney, Australia, 2007.
29. H. Gunes and M. Piccardi, "A bimodal face and body gesture data base for automatic analysis of human nonverbal affective behavior," in Proc. Int. Conf. Pattern Recog., 2006, vol. 1, pp. 1148-1153.
30. H. Gunes and M. Piccardi, "Creating and annotating affect data bases from face and body display: A contemporary survey," in Proc. IEEE Int. Conf. Syst., Man Cybern., 2006, pp. 2426-2433.
31. H. Gunes and M. Piccardi, "Observer annotation of affective display and evaluation of expressivity: Face vs. face-and-body," in Proc. HCSNet Workshop Use Vis. Hum.-Comput. Interaction, 2006, pp. 35-42.
32. H. Gunes and M. Piccardi, "Bi-modal emotion recognition from expressive face and body gestures," J. Netw. Comput. Appl., vol. 30, no. 4, pp. 1334-1345, Nov. 2007.
33. B. Hartmann, M. Mancini, S. Buisine, and C. Pelachaud, "Design and evaluation of expressive gesture synthesis for embodied conversational agents," in Proc. 3rd Int. Joint Conf. Auton. Agents Multi-Agent Syst., 2005, pp. 1095-1096.
34. E. Hudlicka, "To feel or not to feel: The role of affect in human-computer interaction," Int. J. Hum.-Comput. Stud., vol. 59, no. 1/2, pp. 1-32, Jul. 2003.
35. R. Duin, A. Jain, and J. Mao, "Statistical pattern recognition: A review," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 1, pp. 4-37, Jan. 2000.
36. R. El Kaliouby and P. Robinson, "Real-time inference of complex mental states from facial expressions and head gestures," in Proc. Int. Conf. Comput. Vis. Pattern Recog., 2004, vol. 3, p. 154.
37. A. Kapoor and R. W. Picard, "Probabilistic combination of multiple modalities to detect interest," in Proc. Int. Conf. Pattern Recog. 2004, vol. 3, pp. 969-972.
38. A. Kapoor and R. W. Picard, "Multimodal affect recognition in learning environments," in Proc. ACM Int. Conf. Multimedia, 2005, pp. 677-682.
39. K. Karpouzis, G. Caridakis, L. Kessous, N. Amir, A. Raouzaiou, L. Malatesta, and S. Kollias, "Modeling naturalistic affective states via facial vocal and bodily expressions recognition," in Artificial Intelligence for Human Computing, vol. 4451. New York: Springer-Verlag, 2007, pp. 11-92.
40. J. Kittler, M. Hatef, R. P. W. Duin, and J. Matas, "On combining classifiers," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 3, pp. 226-239, Mar. 1998.
41. R. Laban and L. Ullmann, The Mastery of Movement, 4th ed. London, U.K.: Princeton Book Company, 1988.
42. R. Lienhart and J. Maydt, "An extended set of Haar-like features for rapid object detection," in Proc. IEEE Int. Conf. Image Process., 2002, vol. 1, pp. 900-903.
43. C. L. Lisetti and F. Nasoz, "Maui: A multimodal affective user interface," in Proc. ACM Int. Conf. Multimedia, 2002, pp. 161-170.
44. J.-C. Martin, R. Niewiadomski, L. Devillers, S. Buisine, and C. Pelachaud, "Multimodal complex emotions: Gesture expressivity and blended facial expressions," J. Humanoid Robot., vol. 3, pp. 831-843, 2006.
45. R. Masters, "Compassionate wrath: Transpersonal approaches to anger," J. Transpers. Psychol., vol. 32, no. 1, pp. 31-51, 2000.
46. T. O. Meservy, M. L. Jensen, J. Kruse, J. K. Burgoon, J. F. Nunamaker, D. P. Twitchell, G. Tsechpenakis, and D. N. Metaxas, "Deception detection through automatic, unobtrusive analysis of nonverbal behavior," IEEE Intell. Syst., vol. 20, no. 5, pp. 36-43, Sep./Oct. 2005.
47. S. Mitra and T. Acharya, "Gesture recognition: A survey," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 37, no. 3, pp. 311-324, May 2007.
48. L. Nigay and J. Coutaz, "A generic platform for addressing the multimodal challenge," in Proc. Conf. Human Factors Comput. Syst. (CHI), 1995, pp. 98-105.
49. H. Ning, T. X. Han, Y. Hu, Z. Zhang, Y. Fu, and T. S. Huang, "A realtime shrug detector," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2006, pp. 505-510.
50. T. Otsuka and J. Ohya, "Spotting segments displaying facial expression from image sequences using HMM," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 1998, pp. 442-447.
51. M. Paleari and C. L. Lisetti, "Toward multimodal fusion of affective cues," in Proc. 1st ACM Int. Workshop Human-Centered Multimedia, 2006, pp. 99-108.
52. H. Pan, T. S. Levinson, S. E. Huang, and Z.-P. Liang, "A fused hidden Markov model with application to bimodal speech processing," IEEE Trans. Signal Process., vol. 52, no. 3, pp. 573-581, Mar. 2004.
53. M. Pantic and I. Patras, "Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 36, no. 2, pp. 433-449, Apr. 2006.
54. M. Pantic, A. Pentland, A. Nijholt, and T. Huang, "Machine understanding of human behavior," in Proc. Int. Joint Conf. Artif. Intell., Workshop AI Hum. Comput., 2007, pp. 13-24.
55. M. Pantic and L. J. M. Rothkrantz, "Toward an affect-sensitive multimodal human-computer interaction," Proc. IEEE, vol. 91, no. 9, pp. 1370-1390, Sep. 2003.
56. M. Piccardi and O. Perez, "Hidden Markov models with kernel density estimation of emission probabilities and their use in activity recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007, pp. 1-8.
57. A. Pikovsky, M. Rosenblum, and J. Kurths, "Phase synchronization in regular and chaotic systems: A tutorial," Int. J. Bifurc. Chaos, vol. 10, no. 10, pp. 2291-2305, 2000.
58. J. Platt, "Fast training of support vector machines using sequential minimal optimization," in Advances in Kernel Methods-Support Vector Learning, B. Scholkopf, C. Burges, and A. Smola, Eds. Cambridge, MA: MIT Press, 1977.
59. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989.
60. J. A. Russell, "A circumplex model of affect," J. Pers. Soc. Psychol., vol. 39, no. 6, pp. 1161-1178, 1980.
61. S. Salvador and P. Chan, "FastDTW: Toward accurate dynamic time warping in linear time and space," in Proc. KDD Workshop Mining Temporal Sequential Data, Seattle, WA, 2004, pp. 70-80.
62. K. L. Schmidt and J. F. Cohn, "Human facial expressions as adaptations: Evolutionary questions in facial expression research," Yearb. Phys. Anthropol., vol. 44, pp. 3-24, 2001.
63. N. Sebe, I. Cohen, T. Gevers, and T. S. Huang, "Emotion recognition based on joint visual and audio cues," in Proc. Int. Conf. Pattern Recog., 2006, vol. 1, pp. 1136-1139.
64. N. Sebe, M. S. Lew, I. Cohen, Y. Sun, T. Gevers, and T. S. Huang, "Authentic facial expression analysis," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2004, pp. 517-522.
65. C. Shan, S. Gong, and P.W. McOwan, "Dynamic facial expression recognition using a Bayesian temporal manifold model," in Proc. Conf. Brit. Mach. Vis. Assoc., 2006, pp. 297-306.
66. C. Shan, S. Gong, and P. W. McOwan, "Beyond facial expressions: Learning human emotion from body gestures," in Proc. Brit. Mach. Vis. Conf., 2007. Online. Available: Www.dcs.warwick.ac.uk/bmvc2007/ proceedings/CD-ROM/papers/paper-276.pdf
67. D. X. Sun, "Feature dimension reduction using reduced-rank maximum likelihood estimation for hidden Markov model," in Proc. Int. Conf. Spoken Language Process., 1996, pp. 244-247.
68. Y. L. Tian, T. Kanade, and J. F. Cohn, Handbook of Face Recognition New York: Springer-Verlag, 2005.
69. Y. L. Tian, T. Kanade, and J. F. Cohn, "Facial expression analysis," in The Handbook of Face Recognition, S. Z. Li and A.K. Jain, Eds. New York: Springer-Verlag, 2005.
70. Y. Tong, W. Liao, and Q. Ji, "Facial action unit recognition by exploiting their dynamic and semantic relationships," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 10, pp. 1683-1699, Oct. 2007.
71. M. Valstar and M. Pantic, "Fully automatic facial action unit detection and temporal analysis," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2006, pp. 149-154.
72. M. F. Valstar, H. Gunes, and M. Pantic, "How to distinguish posed from spontaneous smiles using geometric features," in Proc. ACM Int. Conf. Multimodal Interfaces, 2007, pp. 38-45.
73. J. Van den Stock, R. Righart, and B. De Gelder, "Body expressions influence recognition of emotions in the face and voice," Emotion vol. 7, no. 3, pp. 487-494, Aug. 2007.
74. P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," in Proc. IEEE Conf. Comput. Vis. Pattern Recog. 2001, vol. 1, pp. 511-518.
75. M. E. Wall, A. Rechtsteiner, and L. M. Rocha, "Singular value decomposition and principal component analysis," in A Practical Approach to Microarray Data Analysis, D. P. Berrar, W. Dubitzky, and M. Granzow, Eds. Norwell, MA: Kluwer, 2003, pp. 91-109.
76. A. D. Wilson, A. F. Bobick, and J. Cassell, "Temporal classification of natural gesture and application to video coding," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 1997, pp. 948-954.
77. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools With Java Implementations. San Francisco, CA: Morgan Kaufmann, 2000.
78. L. Wu, S. L. Oviatt, and P. R. Cohen, "Multimodal integration - A statistical view," IEEE Trans. Multimedia, vol. 1, no. 4, pp. 334-341, 1999.
79. M. Yeasin, B. Bullot, and R. Sharma, "From facial expression to level of interests: A spatio-temporal approach," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2004, pp. 922-927.
80. Z. Zeng, Y. Fu, G. I. Roisman, Z.Wen, Y. Hu, and T. S. Huang, "One-class classification for spontaneous facial expression analysis," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., 2006, pp. 281-286.
81. Y. Zhang and Q. Ji, "Active and dynamic information fusion for facial expression understanding from image sequences," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 5, pp. 699-714, May 2005.