Spotting agreement and disagreement: A survey of nonverbal audiovisual cues and tools

Proceedings - 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, ACII 2009

Volumn , Issue , 2009, Pages

  Bousmalis, Konstantinos ^a   Mehu, Marc ^b   Pantic, Maja ^c  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF GENEVA   (Switzerland)

^c UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

BEHAVIORAL CUES; TEMPORAL PATTERN;

SURVEYS;

INTELLIGENT COMPUTING;

EID: 77949366790     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ACII.2009.5349477     Document Type: Conference Paper

References (83)

1. M. Argyle. Bodily Communication. 2 edn., 1988. Chapter 7.
2. K. Audhkhasi, K. Kandhway, O. Deshmukh, and A. Verma. Formant-based technique for automatic filled-pause detection in spoken english. Proc. IEEE Int'l Conf. Acoustics, Speech and Signal Processing, pp. 4857-4860. 2009.
3. S. Ba and J. Odobez. Recognizing Visual Focus of Attention from Head Pose in Natural Meetings. IEEE Trans. Systems, Man, and Cybernetics, Part B, 39(1):16-33, 2009.
4. D. Baron, E. Shriberg, and A. Stolcke. Automatic punctuation and disfluency detection in multi-party meetings using prosodic and lexical cues. Proc. Int'l Conf. Spoken Language Processing, pp. 949-952. 2002.
5. M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan. Fully automatic facial action recognition in spontaneous behavior. Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 223-230. 2006.
6. A. Bobick and J. Davis. The recognition of human movement using temporal templates. IEEE Trans. Pattern Analysis and Machine Intelligence, 23(3):257-267, 2001.
7. L. J. Brunner. Smiles can be back channels. Journal of Personality and Social Psychology, 37(5):728-734, 1979.
8. P. Buehler, M. Everingham, D. Huttenlocher, and A. Zisserman. Long term arm and hand tracking for continuous sign language TV broadcasts. Proc. Conf. British Machine Vision, pp. 1105-1114. 2008.
9. P. Bull. Posture and Gesture, chap. 5: The Encoding of Disagreement and Agreement, pp. 62-69. Pergamon Press, 1987.
10. P. Bull. Posture and Gesture, chap. 6: The Decoding of Interest/Boredom and Disagreement/Agreement, pp. 70-84. Pergamon Press, 1987.
11. J. Carletta. Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus. Language Resources and Evaluation Journal, 41(2):181-190, 2007.
12. T. Chartrand and J. Bargh. The chameleon effect: the perception-behavior link and social interaction. Journal of Personality and Social Psychology, 76(6):893-910, 1999.
13. S. Cohen. A computerized scale for monitoring levels of agreement during a conversation. University of Pennsylvania Working Papers in Linguistics, 8(1):57-70, 2003.
14. C. Darwin. The expression of emotions in man and animals. Oxford University Press, USA, 2002.
15. L. Ding and A. Martinez. Modelling and recognition of the linguistic components in american sign language. Image and Vision Computing Journal, 27(12), 2009.
16. A. Dittmann. Develpmental factors in conversational behavior. Journal of Communication, 22(4):404-423, 1972.
17. E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, O. Lowry, M. McRorie, J. Martin, L. Devillers, S. Abrilian, A. Batliner, et al. The HUMAINE database: Addressing the collection and annotation of naturalistic and induced emotional data. Lecture Notes in Computer Science, 4738:483-500, 2007.
18. P. Ekman. Human Ethology, chap. About Brows: Emotional and Conversational Signals. Cambridge Univ. Press, 1979.
19. P. Ekman, W. V. Friesen, and J. C. Hager. Facial action coding system. Salt Lake City: Research Nexus, 2002.
20. R. el Kaliouby and P. Robinson. Real-time inference of complex mental states from facial expressions and head gestures. Proc. IEEE Int'l Conf. Computer Vision & Pattern Recognition, vol. 3, p. 154. 2004.
21. Z. Fu,W. Li, X. Li, F. Li, and Y.Wang. Automatic tongue location and segmentation. Proc. Int'l Conf. Audio, Language and Image Processing, pp. 1050-1055. 2008.
22. S. Fujie, Y. Ejiri, K. Nakajima, Y. Matsusaka, and T. Kobayashi. A conversation robot using head gesture recognition as para-linguistic information. Proc. IEEE Int'l Workshop Robot and Human Interactive Communication, pp. 159-164. 2004.
23. M. Gabrea and D. O'Shaughnessy. Detection of filled pauses in spontaneous conversational speech. Proc. Int'l Conf. Spoken Language Processing, vol. 3, pp. 678-681. 2000.
24. M. Galley, K. McKeown, J. Hirschberg, and E. Shriberg. Identifying agreement and disagreement in conversational speech: use of bayesian networks to model pragmatic dependencies. Proc. Meeting Association for Computational Linguistics, pp. 669-676. 2004.
25. D. B. Givens. The nonverbal dictionary of gestures, signs and body language cue. Center for Nonverbal Studies Press, Sokane, WA, 2002.
26. M. Goto, K. Itou, and S. Hayamizu. A real-time filled pause detection system for spontaneous speech recognition. Proc. European Conf. Speech Communication and Technology, pp. 227-230. 1999.
27. J. Gottman, H. Markman, and C. Notarius. The topography of marital conflict: A sequential analysis of verbal and nonverbal behavior. Journal of Marriage and the Family, 39(3):461-477, 1977.
28. D. Greatbatch. Talk at Work, chap. 9: On the management of disagreement between news interviewees. Cambridge University Press, 1992.
29. H. Gunes and M. Pantic. Automatic, dimensional and continuous emotion recognition. Int'l Journal of Synthetic Emotion, 1(1), 2009.
30. U. Hadar, T. Steiner, and F. C. Rose. Head movement during listening turns in conversation. Journal of Nonverbal Behavior, 9(4):214-228, 1985.
31. F. Haumer and W. Donsbach. The rivalry of nonverbal cues on the perception of politicians by television viewers. Journal of Broadcasting and Electronic Media, 53(2):262-279, 2009.
32. D. Hillard, M. Ostendorf, and E. Shriberg. Detection of agreement vs. disagreement in meetings: training with unlabeled data. Proc. Conf. North American Chapter of the Association for Computational Linguistics on Human Language Technology, pp. 34-36. 2003.
33. T. Ike, N. Kishikawa, and B. Stenger. A Real-Time Hand Gesture Interface Implemented on a Multi-Core Processor. Proc. IAPR Conf. Machine Vision Applications, pp. 9-12. 2007.
34. S. Kawato and J. Ohya. Real-time detection of nodding and head-shaking by directly detecting and tracking the between-eyes. Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 40-45. 2000.
35. E. Keller and W. Tschacher. Prosodic and gestural expression of interactional agreement. Lecture Notes in Computer Science, 4775:85-98, 2007.
36. S. Koelstra and M. Pantic. Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics. Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition. 2008.
37. I. Laptev and P. Perez. Retrieving actions in movies. Proc. Int'l Conf. Computer Vision, pp. 1-8. 2007.
38. C.-C. Lee, S. Lee, and S. Narayanan. An analysis of multimodal cues of interruption in dyadic spoken interactions. Proc. European Conf. Speech Communication and Technology, pp. 1678-1681. 2008.
39. G. Littlewort, M. S. Bartlett, I. Fasel, J. Susskind, and J. Movellan. Dynamics of facial expression extracted automatically from video. Image and Vision Computing, 24(6):615-625, 2006.
40. Y. Liu, E. Shriberg, A. Stolcke, D. Hillard, M. Ostendorf, and M. Harper. Enriching speech recognition with automatic detection of sentence boundaries and disfluencies. IEEE Trans. Audio, Speech, and Language Processing, 14(5):1526-1540, 2006.
41. V. Manusov and A. R. Trees. "Are You Kidding Me?": The Role of Nonverbal Cues in the Verbal Accounting Process. The Journal of Communication, 52(3):640-656, 2002.
42. M. Marszałek, I. Laptev, and C. Schmid. Actions in context. Proc. IEEE Conf. Computer Vision & Pattern Recognition. 2009.
43. M. Marzinzik and B. Kollmeier. Speech pause detection for noise spectrum estimation by tracking power envelope dynamics. IEEE Trans. Speech and Audio Processing, 10(2):109-118, 2002.
44. S. Matos, S. Birring, I. Pavord, and H. Evans. Detection of cough signals in continuous audio recordings using hidden markov models. IEEE Trans. Biomedical Engineering, 53(6):1078-1083, 2006.
45. T. Meservy, M. Jensen, J. Kruse, J. Burgoon, J. Nunamaker, D. Twitchell, G. Tsechpenakis, and D. Metaxas. Deception detection through automatic, unobtrusive analysis of nonverbal behavior. IEEE Intelligent Systems, 20(5):36-43, 2005.
46. K. Mikolajczyk and H. Uemura. Action recognition with motion-appearance vocabulary forest. Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8. 2008.
47. L.-P. Morency, C. Sidner, C. Lee, and T. Darrell. Contextual recognition of head gestures. Proc. ACM Int'l Conf. Multimodal Interfaces, pp. 18-24. 2005.
48. L.-P. Morency, C. Sidner, C. Lee, and T. Darrell. Head gestures for perceptual interfaces: The role of context in improving recognition. Artificial Intelligence, 171(8-9):568-585, 2007.
49. C. Morimoto and M. Mimica. Eye gaze tracking techniques for interactive applications. Computer Vision and Image Understanding, 98(1):4-24, 2005.
50. D. Morris. Bodytalk: A world guide to gestures. Jonathan Cape, 1994.
51. E. Murphy-Chutorian and M. Trivedi. Head pose estimation in computer vision: A survey. IEEE Trans. Pattern Analysis and Machine Intelligence, 31(4):607-626, 2009.
52. J. Niebles and L. Fei-Fei. A hierarchical model of shape and appearance for human action classification. Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8. 2007.
53. R. Ogden. Phonetics and social action in agreements and disagreements. Journal of Pragmatics, 38(10):1752-1775, 2006.
54. A. Oikonomopoulos, M. Pantic, and I. Patras. Sparse Bspline polynomial descriptors for human activity recognition. Image and Vision Computing, 27(12), 2009.
55. M. Pantic. Machine analysis of facial behaviour: Naturalistic and dynamic behaviour. Philosophical Transactions of Royal Society B, 2009.
56. 2): How far are we from attaining it? Journal of Autonomous and Adaptive Communications Systems, 1(2):168-187, 2008.
57. M. Pantic and I. Patras. Detecting facial actions and their temporal segments in nearly frontal-view face image sequences. Proc. IEEE Int'l Conf. Systems, Man and Cybernetics, vol. 4, pp. 3358-3363. 2005.
58. M. Pantic, A. Pentland, A. Nijholt, and T. S. Huang. Human computing and machine understanding of human behavior: A survey. Lecture Notes in Computer Science, 4451:47-71, 2007.
59. A. Pentland. Socially aware, computation and communication. Computer, 38(3):33-40, 2005.
60. S. Petridis and M. Pantic. Audiovisual laughter detection based on temporal features. Proc. ACM Int'l Conf. Multimodal Interfaces, pp. 37-44. 2008.
61. S. Petridis and M. Pantic. Fusion of audio and visual cues for laughter detection. Proc. ACM Int'l Conf. Content-based Image and Video Retrieval, pp. 329-338. 2008.
62. I. Poggi. Mind, hands, face and body: Goal and belief view of multimodal communication. Weidler, 2007.
63. A. M. Pomerantz. Second Assessments: A Study of Some Features of Agreements/Disagreements. General sociology, University of California, Irvine, 1975.
64. A. M. Pomerantz. Structures of Social Action: Studies in Conversation Analysis, chap. Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes. Studies in Emotion and Social Interaction. Cambridge University Press, 1984.
65. W. Rinn. The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. Psychological Bulletin, 95(1):52-77, 1984.
66. H. M. Rosenfeld and M. Hancks. The Relationship of Verbal and Nonverbal Communication, chap. The Nonverbal Context of Verbal Listener Responses, pp. 193-206. Walter de Gruyter, 1980.
67. J. Seiter. Does communicating nonverbal disagreement during an opponent's speech affect the credibility of the debater in the background? Psychological Reports, 84:855-861, 1999.
68. J. S. Seiter, H. J. Kinzer, and H. Weger. Background behavior in live debates: The effects of the implicit ad hominem fallacy. Communication Reports, 19(1):57-69, 2006.
69. J. S. Seiter and H. Weger. Audience perceptions of candidates' appropriateness as a function of nonverbal behaviors displayed during televised political debates. The Journal of Social Psychology, 145(2):225-236, 2005.
70. E. Shechtman and M. Irani. Matching local self-similarities across images and videos. Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8. 2007.
71. W. Tan and G. Rong. A real-time head nod and shake detector using HMMs. Expert Systems with Applications, 25(3):461 - 466, 2003.
72. Y.-I. Tian, T. Kanade, and J. Cohn. Recognizing action units for facial expression analysis. IEEE Trans. Pattern Analysis and Machine Intelligence, 23(2):97-115, 2001.
73. Y. Tong, W. Liao, and Q. Ji. Affective Information Processing, chap. 10: Automatic Facial Action Unit Recognition by Modeling Their Semantic and Dynamic Relationships, pp. 159-180. Springer London, 2009.
74. K. P. Truong and D. A. van Leeuwen. Automatic discrimination between laughter and speech. Speech Communication, 49(2):144 - 158, 2007.
75. M. F. Valstar, H. Gunes, and M. Pantic. How to distinguish posed from spontaneous smiles using geometric features. Proc. ACM Int'l Conf. Multimodal Interfaces, pp. 38-45. 2007.
76. M. F. Valstar and M. Pantic. Combined support vector machines and hidden markov models for modeling facial action temporal dynamics. Lecture Notes in Computer Science, 4796:118-127, 2007.
77. A. Vinciarelli, A. Dielmann, S. Favre, and H. Salamin. Canal9: A database of political debates for analysis of social interactions. Proc. IEEE Int'l Conf. Affective Computing and Intelligent Interfaces, vol. 2. 2009.
78. A. Vinciarelli, M. Pantic, and H. Bourlard. Social signal processing: Survey of an emerging domain. Image and Vision Computing, 27(12), 2009.
79. E. Vural, M. Cetin, A. Ercil, G. Littlewort, M. Bartlett, and J. Movellan. Automated drowsiness detection for improved driving safety. Proc. Int'l Conf. Automotive Technologies. 2008.
80. C.-H. Wu and G.-L. Yan. Acoustic feature analysis and discriminative modeling of filled pauses for spontaneous speech recognition. The Journal of VLSI Signal Processing, 36(2-3):91-104, 2004.
81. P. Yang, Q. Liu, and D. Metaxas. Boosting coded dynamic features for facial action units and facial expression recognition. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 1-6. 2007.
82. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. Pattern Analysis and Machine Intelligence, 31(1):39-58, 2009.
83. L. Zhi, J.-Q. Yan, T. Zhou, and Q.-L. Tang. Tongue Shape Detection Based on B-Spline. Proc. Int'l Conf. Machine Learning and Cybernetics, pp. 3829-3832. 2006.