Towards a technology of nonverbal communication: Vocal behavior in social and affective phenomena

Affective Computing and Interaction: Psychological, Cognitive and Neuroscientific Perspectives

Volumn , Issue , 2010, Pages 133-156

  Vinciarelli, Alessandro ^a,b   Mohammadi, Gelareh ^b  

^a UNIVERSITY OF GLASGOW   (United Kingdom)

^b IDIAP RESEARCH INSTITUTE   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84863753243     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4018/978-1-61692-892-6.ch007     Document Type: Chapter

References (112)

1. Albrecht, K. (2005). Social intelligence: The new science of success. New York: John Wiley & Sons Ltd.
2. Ambady, N., Bernieri, F., & Richeson, J. (2000). Towards a histology of social behavior: judgmental accuracy from thin slices of behavior. In Zanna, M. (Ed.), Advances in experimental social psychology (pp. 201-272).
3. Ambady, N., Krabbenhoft, M., & Hogan, D. (2006). The 30-sec sale: Using thin-slice judgments to evaluatesales effectiveness. Journal of Consumer Psychology, 16(1), 4-13. doi:10.1207/ s15327663jcp1601_2
4. Ambady, N., & Rosenthal, R. (1992). Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychological Bulletin, 111(2), 256-274. doi:10.1037/0033-2909.111.2.256
5. Amir, N., Ron, S., & Laor, N. (2000). Analysis of an emotional speech corpus in Hebrew based on objective criteria. In Proceedings of the ISCA Workshop on Speech & Emotion (pp. 29-33).
6. Ang, J., Dhillon, R., Krupski, A., Shriberg, E., & Stolcke, A. (2002). Prosody-based automatic detection of annoyance and frustration in humancomputer dialog. In. Proceedings of ICSLP, 02, 2037-2040.
7. Argyle, M., & Trower, P. (1979). Person to person: ways of communicating. HarperCollins Publishers.
8. Bales, R. (1950). Interaction process analysis: A method for the study of small groups. Addison-Wesley.
9. Banerjee, S., & Rudnicky, A. (2004). Using simple speech based features to detect the state of a meeting and the roles of the meeting participants. In Proceedings of international conference on spoken language processing (pp. 221-231).
10. Barzilay, R., Collins, M., Hirschberg, J., & Whittaker, S. (2000). The rules behind the roles: identifying speaker roles in radio broadcasts. In Proceedings of the 17th national conference on artificial intelligence (pp. 679-684).
11. Bhatti, M. W., Wang, Y., & Guan, L. (2004). A neural network approach for human emotion recognition in speech. In Proceedings of the International Symposium on Circuits & Systems, ISCAS'04, (Vol. 2, pp. II-181-4).
12. Bickmore, T., & Cassell, J. (2005). Social dialogue with embodied conversational agents. In van Kuppevelt, J., Dybkjaer, L., & Bernsen, N. (Eds.), Advances in natural, multimodal, dialogue systems (pp. 23-54). New York: Kluwer. doi:10.1007/1-4020-3933-6_2
13. Bilmes, J. (1988). The concept of preference in conversation analysis. Language in Society, 17(2), 161-181. doi:10.1017/S0047404500012744
14. Bousmalis, K., Mehu, M., & Pantic, M. (2009). Spotting agreement and disagreement: A survey of nonverbal audiovisual cues and tools. In Proceedings of the international conference on affective computing and intelligent interaction (Vol. II, pp. 121-129).
15. Buchanan, M. (2007). The science of subtle signals. Strategy+Business, 48, 68-77.
16. Busso, C., Lee, S., & Narayanan, S. (2009). Analysis of emotionally salient aspects of fundamental frequency for emotion detection. IEEE Transaction on Audio. Speech and Language Processing, 17(4), 582-596. doi:10.1109/ TASL.2008.2009578
17. Cohn, J. (2006). Foundations of human computing: facial expression and emotion. In Proceedings of the ACM international conference on multimodal interfaces (pp. 233-238).
18. Cortes, J., & Gatti, F. (1965). Physique and self-description of temperament. Journal of Consulting Psychology, 29(5), 432-439. doi:10.1037/h0022504
19. Costa, M., Dinsbach, W., Manstead, A., & Bitti, P. (2001). Social presence, embarrassment, and non-verbal behavior. Journal of Nonverbal Behavior, 25(4), 225-240. doi:10.1023/A:1012544204986
20. Coulson, M. (2004). Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence. Journal of Non-verbal Behavior, 28(2), 117-139. doi:10.1023/B:JONB.0000023655.25550.be
21. Crowley, J. L. (2006). Social Perception. ACM Queue; Tomorrow's Computing Today, 4(6), 43-48. doi:10.1145/1147518.1147531
22. Crystal, D. (1969). Prosodic systems and intonation in English. Cambridge: Cambridge University Press.
23. Cunningham, D., Kleiner, M., Bültho, H., & Wallraven, C. (2004). The components of conversational facial expressions. Proceedings of the Symposium on Applied Perception in Graphics and Visualization (pp. 143-150).
24. Dion, K., Berscheid, E., & Walster, E. (1972). What is beautiful is good. Journal of Personality and Social Psychology, 24(3), 285-290. doi:10.1037/ h0033731
25. Dong, W., Lepri, B., Cappelletti, A., Pentland, A., Pianesi, F., & Zancanaro, M. (2007). Using the influence model to recognize functional roles in meetings. In Proceedings of the 9th international conference on multimodal interfaces (pp. 271-278).
26. Dovidio, J., & Ellyson, S. (1982). Decoding visual dominance: Attributions of power based on relative percentages of looking while speaking and looking while listening. Social Psychology Quarterly, 45(2), 106-113. doi:10.2307/3033933
27. Dumais, S., Cutrell, E., Cadiz, J. J., Jancke, G., Sarin, R., & Robbins, D. C. (2003). Stuff I've seen: a system for personal information retrieval and re-use. Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 72-79).
28. Ekman, P., & Rosenberg, E. (2005). What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (facs). New York: Oxford University Press.
29. Favre, S., Dielmann, A., & Vinciarelli, A. (2009). Automatic role recognition in multiparty recordings using social networks and probabilistic sequential models. In Proceedings of ACM international conference on multimedia.
30. Forsyth, D., Arikan, O., Ikemoto, L., O'Brien, J., & Ramanan, D. (2006). Computational studies of human motion part 1: Tracking and motion synthesis. Foundations and Trends in Computer Graphics and Vision, 1(2), 77-254. doi:10.1561/0600000005
31. France, D. J., Shiavi, R. G., Silverman, S., Silverman, M., & Wilkes, D. M. (2000). Acoustical Properties of speech as indicators of depression and suicidal risk. IEEE Transactions on Bio-Medical Engineering, 47(7), 829-837. doi:10.1109/10.846676
32. Galley, M., McKeown, K., Hirschberg, J., & Shriberg, E. (2004). Identifying agreement and disagreement in conversational speech: use of Bayesian Networks to model pragmatic dependencies. In Proceedings of meeting of the association for computational linguistics (pp. 669-676).
33. Garg, N., Favre, S., Salamin, H., Hakkani-Tur, D., & Vinciarelli, A. (2008). Role recognition for meeting participants: an approach based on lexical information and Social Network Analysis. In Proceedings of the acm international conference on multimedia (pp. 693-696).
34. Gladwell, M. (2005). Blink: The power of thinking without thinking. New York: Little Brown & Company.
35. Glass, C., Merluzzi, T., Biever, J., & Larsen, K. (1982). Cognitive assessment of social anxiety: Development and validation of a self-statement questionnaire. Cognitive Therapy and Research, 6(1), 37-55. doi:10.1007/BF01185725
36. Goleman, D. (2005). Emotional intelligence. New York: Random House Publishing Group.
37. Grahe, J., & Bernieri, F. (1999). The importance of nonverbal cues in judging rapport. Journal of Nonverbal Behavior, 23(4), 253-269. doi:10.1023/A:1021698725361
38. Hall, E. (1959). The silent language. New York: Doubleday.
39. Hecht, M., De Vito, J., & Guerrero, L. (1999). Perspectives on nonverbal communication-codes, functions, and contexts. In L. Guerrero, J. De Vito, & M. Hecht (Eds.), Hecht, M., De Vito, J., & Guerrero, L (p. 3-18). Waveland Press.
40. Hillard, D., Ostendorf, M., & Shriberg, E. (2003). Detection of agreement vs. disagreement in meetings: Training with unlabeled data. In Proceedings of the north American chapter of the association for computational linguistics - human language technologies conference.
41. Hyun, H. K., Kim, E. H., & Kwak, Y. K. (2005). Improvement of emotion recognition by Bayesian classifier using non-zero-pitch concept. IEEE International Workshop on Robots and Human Interactive Communication, ROMAN 2005, (pp. 312-316).
42. Jayagopi, D., Hung, H., Yeo, C., & Gatica-Perez, D. (2009). Modeling dominance in group conversations from non-verbal activity cues. IEEE Transactions on Audio. Speech and Language Processing, 17(3), 501-513. doi:10.1109/TASL.2008.2008238
43. Kapoor, A., Picard, R., & Ivanov, Y. (2004). Probabilistic combination of multiple modalities to detect interest. In Proceedings of the international conference on pattern recognition (pp. 969-972).
44. Knapp, M., & Hall, J. (1972). Nonverbal communication in human interaction. Harcourt Brace College Publishers.
45. Kunda, Z. (1999). Social cognition. Boston: MIT Press.
46. Kwon, O. W., Chan, K., Hao, J., & Lee, T. W. (2003). Emotion Recognition by Speech Signals. In Proceedings of International Conference EUROSPEECH (pp. 125-128).
47. Laskowski, K., Ostendorf, M., & Schultz, T. (2008). Modeling vocal interaction for text independent participant characterization in multi-party conversation. In Proceedings of the 9th isca/acl sigdial workshop on discourse and dialogue (pp. 148-155).
48. Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabasi, A., & Brewer, D. (2009). Computational social science. Science, 323, 721-723. doi:10.1126/ science.1167742
49. Lee, C. M., Narayanan, S., & Pieraccini, R. (2001). Recognition of negative emotions from the speech signals. In Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding (pp. 240-243).
50. Levine, J., & Moreland, R. (1990). Progress in small roup research. Annual Review of Psychology, 41, 585-634. doi:10.1146/annurev.ps.41.020190.003101
51. Levine, J., & Moreland, R. (1998). Small groups. In Gilbert, D., & Lindzey, G. (Eds.), The handbook of social psychology (Vol. 2, pp. 415-469). New York: Oxford University Press.
52. Li, X., Tao, J., Johanson, M. T., Soltis, J., Savage, A., Leong, K. M., & Newman, J. D. (2007). Stress and emotion classification using jitter and shimmer features. In Proceedings of IEEE International Conference on Acoustics, Speech & Signal Processing, ICASSP2007 (Vol. 4, pp. IV-1081-4).
53. Liu, Y. (2006). Initial study on automatic identication of speaker role in broadcast news speech. In Proceedings of the human language technology conference of the naacl, companion volume: Short papers (pp. 81-84).
54. Lott, D., & Sommer, R. (1967). Seating arrangements and status. Journal of Personality and Social Psychology, 7(1), 90-95. doi:10.1037/h0024925
55. Lugger, M., & Yang, B. (2006). Classification of different speaking groups by means of voice quality parameters. ITG-Sprach-Kommunikation.
56. Lugger, M., & Yang, B. (2008). Cascaded emotion classification via psychological emotions using a large set of voice quality parameters. In Proceedings of IEEE International Conference on Acoustics, Speech & Signal Processing, ICASSP2008 (pp. 4945-4948.
57. Mast, M. (2002). Dominance as expressed and inferred through speaking time: A metaanalysis. Human Communication Research, 28(3), 420-450.
58. McCowan, I., Bengio, S., Gatica-Perez, D., Lathoud, G., Monay, F., Moore, D., et al. (2003). Modeling human interaction in meetings. In Proceedings of IEEE international conference on acoustics, speech and signal processing (pp. 748-751).
59. McGilloway, S., Cowie, R., & Douglas-Cowie, E. (2000). Approaching automatic recognition of emotion from voice: a rough benchmark. In Proceedings of the ISCA Workshop on Speech and Emotion (pp. 207-212).
60. Mitra, S., & Acharya, T. (2007). Gesture recognition: A survey. IEEE Transactions on Systems, Man and Cybernetics. Part C, Applications and Reviews, 37(3), 311-324. doi:10.1109/TSMCC.2007.893280
61. Morrison, D., Wang, R., & De Silva, L. C. (2007). Ensemble methods for spoken emotion recognition in call-centres. Speech Communication, 49, 98-112. doi:10.1016/j.specom.2006.11.004
62. th International Congress of Phonetic Sciences (ICPh'95) (Vol.3, pp. 178-181).
63. Murphy-Chutorian, E., & Trivedi, M. (2009). Head pose estimation in computer vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(4), 607-626. doi:10.1109/ TPAMI.2008.106
64. Murray-Smith, R. (2009). (to appear). Empowering people rather than connecting them. International Journal of Mobile HCI.
65. Nass, C., & Brave, S. (2005). Wired for speech: How voice activates and advances the HumanComputer relationship. Boston: The MIT Press.
66. th International Conference on Neural Information Processing (Vol. 2, pp. 495-501).
67. Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003). Speech emotion recognition using hidden Markov models. Speech Communication, 41, 603-623. doi:10.1016/S0167-6393(03)00099-2
68. Nwe, T. L., Wei, F. S., & De Silva, L. C. (2001). Speech based emotion classification. In Proceedings of IEEE Region 10 International Conference on Electrical &. Electron Technology, 1, 291-301.
69. Olguin Olguin, D., Waber, B., Kim, T., Mohan, A., Koji, A., & Pentland, A. (2009). Sensible organizations: technology and methodology for automatically measuring organizational behavior. IEEE Transactions on Systems, Man abd Cybernetics - Part B, 39 (1), 43-55.
70. Otsuka, K., Takemae, Y., & Yamato, J. (2005). A probabilistic inference of multiparty conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances. In Proceedings of ACM international conference on multimodal interfaces (pp. 191-198).
71. Oudeyer, P. Y. (2003). The production and recognition of emotions in speech: features and algorithms. International Journal of Human-Computer Studies, 59, 157-183. doi:10.1016/S1071-5819(02)00141-6
72. Pantic, M., Nijholt, A., Pentland, A., & Huang, T. (2008). Human-Centred Intelligent HumanComputer Interaction (HCI2): how far are we from attaining it? International Journal of Autonomous and Adaptive Communications Systems, 1(2), 168-187. doi:10.1504/IJAACS.2008.019799
73. Pantic, M., Pentland, A., Nijholt, A., & Huang, T. (2007). Human computing and machine understanding of human behavior: A survey. In Lecture notes in articial intelligence (Vol. 4451, pp. 47-71). New York: Springer Verlag.
74. Park, C. H., Lee, D. W., & Sim, K. B. (2002). Emotion recognition of speech based on RNN. In Proceedings of International Conference on Machine Learning & Cybernetics(ICMLC'02) (Vol. 4, pp.2210-2213).
75. Pentland, A. (2008). Honest signals: how they shape our world. Cambridge, MA: MIT Press.
76. Picard, R. (2000). Affective computing. Cambridge, MA: The MIT Press.
77. Poggi, I. (2007). Mind, hands, face and body: A goal and belief view of multimodal communication. Weidler Buchverlag Berlin.
78. Polzin, T. S., & Waibel, A. (2000). Emotion-sensetive human-computer interfaces. In Proceedings of the ISCA Workshop on Speech and Emotion (pp. 201-206).
79. Raento, M., Oulasvirta, A., & Eagle, N. (2009). Smartphones: an emerging tool for social scientists. Sociological Methods & Research, 37(3), 426. doi:10.1177/0049124108330005
80. Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. New York (USA). NY, USA: Cambridge University Press New York.
81. Richmond, V., & McCroskey, J. (1995). Nonverbal behaviors in interpersonal relations. Allyn and Bacon.
82. Rienks, R., & Heylen, D. (2006). Dominance Detection in Meetings Using Easily Obtainable Features. [). New York: Springer.]. Lecture Notes in Computer Science, 3869, 76-86. doi:10.1007/11677482_7
83. Rienks, R., Zhang, D., & Gatica-Perez, D. (2006). Detection and application of in fluence rankings in small group meetings. In Proceedings of the international conference on multimodal interfaces (pp. 257-264).
84. Rong, J., Li, G., & Chen, Y. P. (2009). Acoustic feature selection for automatic emotion recognition from speech. Information Processing & Management, 45, 315-328. doi:10.1016/j.ipm.2008.09.003
85. Salamin, H., Favre, S., & Vinciarelli, A. (2009). Automatic role recognition in multiparty recordings: Using social aliation networks for feature extraction. IEEE Transactions on Multimedia, 11(7), 1373-1380. doi:10.1109/TMM.2009.2030740
86. Scheflen, A. (1964). The significance of posture in communication systems. Psychiatry, 27, 316-331.
87. Schegloff, E. (1987). Single episodes of interaction: an exercise in conversation analysis. Social Psychology Quarterly, 50(2), 101-114. doi:10.2307/2786745
88. Scherer, K. (1979). Personality markers in speech. Cambridge, MA: Cambridge University Press.
89. Scherer, K. (2003). Vocal communication of emotion: a review of research paradigms. Speech Communication, 40, 227-256. doi:10.1016/ S0167-6393(02)00084-5
90. Schuller, B., Reiter, S., Muller, R., Al-Hames, M., Lang, M., & Rigoll, G. (2005). Speaker independent speech emotion recognition by ensemble classification. In Proceedings of IEEE International Conference on Multimedia & Expo (ICME'05).
91. Shami, M. T., & Kamel, M. S. (2005). Segmentbased approach to the recognition of emotions in speech. IEEE International Conference on Multimedia & Expo (ICME'05), Amsterdam, The Netherlands.
92. Shami, M. T., & Verhelst, W. (2007). An evaluation of the robustness of existing supervised machine learning approaches to the classification of emotions in speech. Speech Communication, 49, 201-212. doi:10.1016/j.specom.2007.01.006
93. Surawski, M., & Osso, E. (2006). The eects of physical and vocal attractiveness on impression formation of politicians. Current Psychology (New Brunswick, N.J.), 25(1), 15-27. doi:10.1007/ s12144-006-1013-5
94. Tischler, H. (1990). Introduction to sociology. Harcourt Brace College Publishers.
95. Tranter, S., & Reynolds, D. (2006). An overview of automatic speaker diarization systems. IEEE Transactions on Audio, Speech, and Language Processing, 14(5), 1557-1565. doi:10.1109/ TASL.2006.878256
96. Truong, K., & Van Leeuwen, D. (2007). Automatic discrimination between laughter and speech. Speech Communication, 49(2), 144-158. doi:10.1016/j. specom.2007.01.001
97. Van den Stock, J., Righart, R., & de Gelder, B. (2007). Body expressions influence recognition of emotions in the face and voice. Emotion (Washington, D.C.), 7(3), 487-494. doi:10.1037/1528-3542.7.3.487
98. Ververidis, D., & Kotropoulos, C. (2006). Emotional speech recognition: resources, features, and methods. Speech Communication, 48, 1162-1181. doi:10.1016/j.specom.2006.04.003
99. Ververidis, D., Kotropoulos, C., & Pitas, I. (2004). Automatic emotional speech classification. In proceedings of IEEE International Conference on Acoustics, Speech &. Signal Processing, 1, I-593-I-596.
100. Vinciarelli, A. (2007). Speakers role recognition in multiparty audio recordings using social network analysis and duration distribution modeling. IEEE Transactions on Multimedia, 9(9), 1215-1226. doi:10.1109/TMM.2007.902882
101. Vinciarelli, A. (2009). Capturing order in social interactions. IEEE Signal Processing Magazine, 26(5), 133-137. doi:10.1109/MSP.2009.933382
102. Vinciarelli, A., Pantic, M., & Bourlard, H. (2009). Social Signal Processing: Survey of an emerging domain. Image and Vision Computing Journal, 27(12), 1743-1759. doi:10.1016/j.imavis.2008.11.007
103. Vinciarelli, A., Pantic, M., Bourlard, H., & Pentland, A. (2008). In Social Signal Processing: State-of-the-art and future perspectives of an emerging domain. Proceedings of the ACM international conference on multimedia (pp. 1061-1070).
104. Waibel, A., Schultz, T., Bett, M., Denecke, M., Malkin, R., Rogina, I., & Stiefelhagen, R. (2003). SMaRT: the Smart Meeting Room task at ISL. In Proceedings of IEEE international conference on acoustics, speech, and signal processing (pp. 752-755).
105. Watzlawick, P., Beavin, J., & Jackson, D. (1967). The pragmatics of human communication. New York: Norton.
106. Williams, A. (2003). Facial expression of pain: An evolutionary account. The Behavioral and Brain Sciences, 25(4), 439-455.
107. Womack, B., & Hansen, J. L. H. (1996). Classification of speech under stress using target driven features. Speech Communication, 20, 131-150. doi:10.1016/S0167-6393(96)00049-0
108. Yang, B., & Lugger, M. (2009). Emotion recognition from speech signals using new harmony features. Article in Press. Signal Processing, 1-9.
109. Yang, M., Kriegman, D., & Ahuja, N. (2002). Detecting faces in images: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(1), 34-58. doi:10.1109/34.982883
110. Zancanaro, M., Lepri, B., & Pianesi, F. (2006). Automatic detection of group functional roles in face to face interactions. In Proceedings of international conference on multimodal interfaces (pp. 47-54).
111. Zeng, Z., Pantic, M., Roisman, G., & Huang, T. (2009). A survey of affect recognition methods: audio, visual and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39-58. doi:10.1109/TPA-MI.2008.52
112. Zhou, G., Hansen, J. H. L., & Kaiser, J. F. (2001). Nonlinear feature based classification of speech under stress. IEEE Transactions on Speech and Audio Processing, 9(3), 201-216. doi:10.1109/89.905995