Spotting laughter in natural multiparty conversations: A comparison of automatic online and offline approaches using audiovisual data

ACM Transactions on Interactive Intelligent Systems

Volumn 2, Issue 1, 2012, Pages 1-31

  Scherer, Stefan ^a   Glodek, Michael ^b   Schwenker, Friedhelm ^b   Campbell, Nick ^a   Palm, Günther ^b  

^a TRINITY COLLEGE DUBLIN   (Ireland)

^b UNIVERSITY OF ULM   (Germany)

Author keywords

Echo state networks; Gaussian mixture model supervectors; Hidden Markov models; Laughter detection; Multimodal; Natural discourse; Support vector machines

Indexed keywords

GAUSSIAN DISTRIBUTION; INTERFACE STATES; SUPPORT VECTOR MACHINES; TRELLIS CODES;

AFFECT AWARE APPLICATIONS; ECHO STATE NETWORKS; GAUSSIAN MIXTURE MODEL; GENERALIZATION CAPABILITY; MULTI-MODAL; MULTI-MODAL INTERFACES; MULTI-PARTY CONVERSATIONS; NATURAL DISCOURSE;

HIDDEN MARKOV MODELS;

EID: 84983561587     PISSN: 21606455     EISSN: 21606463     Source Type: Journal    
DOI: 10.1145/2133366.2133370     Document Type: Article

References (47)

1. AKINORI, I., WANG, X., SUZUKI, M., AND SHOZO, M. 2005. Smile and laughter recognition using speech processing and face recognition from conversation video. In Proceedings of the International Conference on Cyberworlds (CW'05). IEEE, 437-444..
2. BACHOROWSKI, J.-A., SMOSKI, M. J., AND OWREN, M. J. 2001. The acoustic features of human laughter. J. Acoust. Soc. Amer. 110, 3, 1581-1597..
3. BATLINER, A., STEIDL, S., EYBEN, F., AND SCHULLER, B. 2010. Onlaughter and speech laugh, basedonobservations of child-robot interaction. The Phonetics of Laughting, Trends in Linguistics. Mouton de Gruyter. To appear..
4. BENNETT, K. P. AND CAMPBELL, C. 2000. Support vector machines: hype or hallelujah? ACM Knowl. Discor. Data Min. Explor. Newslett. 2, 2, 1-13..
5. BISHOP, C. M. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics) 1st ed. Springer..
6. BOUSMALIS, K., MEHU, M., AND PANTIC, M. 2009. Spotting agreement and disagreement: A survey of nonverbal audiovisual cues and tools. In Proceedings of the 3rd International Conference on Affective Computing andIntelligentInteraction (ACII'09).Vol. 2. IEEE, 1-9..
7. CAMPBELL, N., KASHIOKA, H., AND OHARA, R. 2005. No laughing matter. In Proceedings of the Interspeech Conference. 465-468..
8. CAMPBELL, W. M., STURIM, D. E., AND REYNOLDS, D. A. 2006a. Support vector machines using GMM supervectors for speaker verification. IEEE Signal Process. Lett. 13, 5, 308-311..
9. CAMPBELL, W. M., STURIM, D. E., REYNOLDS, D. A., AND SOLOMONOFF, A. 2006b. SVM-Based speaker verification using a GMM supervector kernel and NAP variability compensation. In Proceedings of the IEEE International Conference onAcoustics, Speech, and Signal Processing (ICASSP'06). IEEE, 97-100..
10. CAMPBELL, W. N. 2007. On the use of non verbal speech sounds in human communication. In Lecture Notes in Computer Science Series, vol. 4775, Springer, 117-128..
11. CAMPBELL, W. N. 2008. Tools and resources for visualising conversational-speech interaction. In Proceedings of the 6th International Language Resources and Evaluation (LREC'08). 231-234..
12. CAMPBELL, W. N. AND SCHERER, S. 2010. Comparing measures ofsynchrony and alignment in dialogue speech timing with respect to turn-taking activity. In Proceedings of the Conference. 2546-2549..
13. CARIDAKIS, G., KARPOUZIS, K., WALLACE, M., KESSOUS, L., AND AMIR, N. 2010. Multimodal user's affective state analysis in naturalistic interaction. J. Multimodal User Interfac. 3, 49-66..
14. CARLETTA, J., ASHBY, S., BOURBAN, S., FLYNN, M., GUILLEMOT, M., HAIN, T., KADLEC, J., KARAISKOS, V., KRAAIJ, W., KRONENTHAL, M., LATHOUD, G., LINCOLN, M., LISOWSKA, A., MCCOWAN, I., POST, W., REIDSMA, D., AND WELLNER, P. 2005. The AMI meeting corpus: A pre-announcement. In Machine Learningfor Multimodal Interaction: Second International Workshop (MLMI'05). Lecture Notes in Computer Science Series, vol. 3869, Springer, 28-39..
15. DOUXCHAMPS, D. AND CAMPBELL, W. N. 2007. Robust Real Time Face Tracking for the Analysis of Human Behaviour. Lecture Notes in Computer Science Series, vol. 4892, Springer, 1-10..
16. EYBEN, F., PETRIDIS, S., SCHULLER, B., TZIMIROPOULOS, G., ZAFEIRIOU, S., AND PANTIC, M. 2011. Audiovisual classification ofvocal outbursts in human conversation using long-short-term memory networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'11). IEEE, 5844-5847..
17. GAUSS, C. F. 1809. Theoria motus corporum coelestium in sectionibus conicis solem ambientium. Latin Ed. F. Perthes and I. H. Besser, Hamburgi..
18. GLODEK, M., SCHERER, S., SCHWENKER, F., AND PALM, G. 2011. Conditioned hidden markov model fusion for multimodal classification. InProceedings ofInterspeech Conference. 2269-2272..
19. HERMANSKY, H. 1990. Perceptual linear predictive (PLP) analysis of speech. J. Acoust. Soc. Amer. 87, 4, 1738-1752..
20. HERMANSKY, H. 1997. The modulation spectrum in automatic recognition of speech. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU'97). IEEE, 140-147..
21. HERMANSKY, H., HANSON, B., AND WAKITA, H. 1985. Perceptually based linear predictive analysis of speech. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'85). Vol. 10. IEEE, 509-512..
22. HERMANSKY, H., MORGAN, N., BAYYA, A., AND KOHN, P. 1992. Rasta-PLP speech analysis technique. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'92). Vol. 1. IEEE, 121-124..
23. JAEGER, H. 2002. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the echo state network approach. Tech. rep. 159, Fraunhofer-Gesellschaft, St. Augustin, Germanyy
24. JAEGER, H. AND HAAS, H. 2004. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Sci. 304, 78-80..
25. JANIN, A., BARON, D., EDWARDS, J., ELLIS, D., GELBART, D., MORGAN, N., PESKIN, B., PFAU, T., SHRIBERG, E., STOLCKE, A., AND WOOTERS, C. 2003. The ICSI meetingcorpus. InProceedings of the IEEE International Conference onAcoustics, Speech, and Signal Processing (ICASSP'03). IEEE, 364-367..
26. KENNEDY, L. AND ELLIS, D. 2004. Laughter detection in meetings. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'04), Meeting Recognition Workshop. IEEE, 118-121..
27. KNOX, M. AND MIRGHAFORI, N. 2007. Automatic laughter detection using neural networks. In Proceedings of the Interspeech Conference. 2973-2976..
28. KOLLER, D. AND FRIEDMAN, N. 2009. Probabilistic Graphical Models:Principles and Techniques. The MIT Press..
29. LASKOWSKI, K. 2008. Modeling vocal interaction for text-independent detection of involvement hotspots in multi-partymeetings. InProceedings of the2ndIEEE/ISCA/ACL Workshop on SpokenLanguage Technology (SLT'08). IEEE/ISCA/ACL, 81-84..
30. LASKOWSKI, K. 2009. Contrasting emotion-bearing laughter types in multiparticipant vocal activity detection for meetings. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'09). IEEE, 4765-4768..
31. MAGANTI, H. K., SCHERER, S., AND PALM, G. 2007. A novel feature for emotion recognition in voice based applications. In Proceedings of the 2nd International Conference on Affective Computing and Intelligent Interaction (ACII'07). Springer, 710-711..
32. PETRIDIS, S. AND PANTIC, M. 2009. Is this joke really funny? Judging the mirth by audiovisual laughter analysis. In Proceedings ofIEEE International Conference on Multimedia and Expo (ICME'09). IEEE, 1444-1447..
33. PROVINE, R. AND YONG, L. 1991. Laughter : A stereotyped human vocalization. English 89, 2, 115-124..
34. RABINER, L. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 2, 257-286..
35. RISKO, E. F AND KINGSTONE, A. 2010. Eyes wide shut: Implied social presence, eye tracking and attention. J. Attent. Percept. Psychophys. 1-6..
36. ROSSI, F AND VILLA, N. 2006. Support vector machine for functional data classification. Neurocomput. 69, 7-9, 730-742..
37. SCHERER, S., HOFMANN, H., LAMPMANN, M., PFEIL, M., RHINOW, S., SCHWENKER, F., AND PALM, G. 2008a. Emotion recognition from speech: Stress experiment. InProceedings of the 6th International Language Resources and Evaluation (LREC'08). 1325-1330..
38. SCHERER, S., OUBBATI, M., SCHWENKER, F., AND PALM, G. 2008b. Real-Time emotion recognition from speech using echo state networks. In Proceedings of the 3rd IAPR Workshop on Artificial Neural Networks in Pattern Recognition (ANNPR'08). Springer, 205-216..
39. SCHERER, S., SCHELS, M., AND PALM, G. 2011. How low level observations can help to reveal the user's state in hci. In Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction (ACII'11). S. D'Mello, A. Graesser, B. Schuller, and J.-C. Martin, Eds., Vol. 2, Springer, 81-90..
40. SCHÖLKOPF, B. AND SMOLA, A. J. 2001. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA..
41. SCHULLER, B., STEIDL, S., BATLINER, A., BURKHARDT, F., DEVILLERS, L., MÜLLER, C., AND NARAYANAN, S. S. 2010. The interspeech 2010 paralinguistic challenge. In Proceedings of the Interspeech Conference. 2794-2797..
42. STRAUSS, P.-M., HOFFMANN, H., AND SCHERER, S. 2007. Evaluation and user acceptance of a dialogue system using wizard-of-oz recordings. In Proceedings of the 3rd IET International Conference on Intelligent Environments 2007 (IE'07). IEEE, 521-524..
43. TRUONG, K. P. AND VAN LEEUWEN, D.A. 2005.Automatic detection of laughter. In Proceedings of the Inter speech Conference . 485-488..
44. TRUONG, K. P. AND VAN LEEUWEN, D. A. 2007a. Automatic discrimination between laughter and speech. Speech Comm. 49, 144-158..
45. TRUONG, K. P. AND VAN LEEUWEN, D. A. 2007b. Evaluating laughter segmentation in meetings with acoustic and acoustic-phonetic features. In Proceedings of the Workshop on the Phonetics of Laughter. 49-53..
46. VIOLA, P. AND JONES, M. J. 2004. Robust real-time face detection. Int. J. Comput. Vis. 57, 2, 137-154..
47. ZHAO, J., DONG, Y., ZHAO, X., YANG, H., LU, L., AND WANG, H. 2008. Advances in SVM-based system using GMM super vectors for text-independent speaker verificationsuper vectors for text-independent speaker verification. Tsinghua Sci. Technol. 13, 4, 522-527..