Audiovisual discrimination between speech and laughter: Why and when visual information might help

IEEE Transactions on Multimedia

Volumn 13, Issue 2, 2011, Pages 216-234

  Petridis, Stavros ^a   Pantic, Maja ^a,b  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

Human behavior analysis; laughter versus speech discrimination; neural networks; principal components analysis (PCA)

Indexed keywords

AUDIO AND VIDEO; AUDIO INFORMATION; AUDIO-BASED; CEPSTRAL; CROSS VALIDATION; DATA SETS; FACIAL EXPRESSIONS; HEAD MOVEMENTS; HEAD POSE; HUMAN BEHAVIORS; LAUGHTER-VERSUS-SPEECH DISCRIMINATION; PRINCIPAL COMPONENTS ANALYSIS (PCA); PROSODIC FEATURES; SINGLE-MODAL; TESTING CONDITIONS; TRAINING CONDITIONS; TWO-STREAM; VISUAL CHANNELS; VISUAL INFORMATION;

BEHAVIORAL RESEARCH; EXPERIMENTS; NEURAL NETWORKS; STATISTICAL TESTS;

PRINCIPAL COMPONENT ANALYSIS;

EID: 79952983096     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2010.2101586     Document Type: Article

References (68)

1. [Online]. Available: http://www.doc.ic.ac.uk/~maja/AMI-SAL-annotations. xls.
2. Nist (2004), Rich Transcription 2004 Spring Meeting Recognition Evaluation, Documentation. [Online]. Available: http://www.nist.gov/ speech/tests/rt/rt2004/spring/.
3. J. Bachorowski and M. Owren, "Not all laughs are alike: Voiced but not unvoiced laughter readily elicits positive affect," Psychol. Sci., vol. 12, no. 3, pp. 252-257, 2001. (Pubitemid 33653539)
4. J. A. Bachorowski, M. J. Smoski, and M. J. Owren, "The acoustic features of human laughter," J. Acoust. Soc. Amer., vol. 110, no. 1, pp. 1581-1597, 2001.
5. P. Boersma, "Accurate short-term analysis of the fundamental frequency and the harmonics-to-noise ratio of a sampled sound," Proc. Inst. Phonet. Sci., vol. 17, pp. 97-110, 1993.
6. K. Bousmalis, M. Mehu, and M. Pantic, "Spotting agreement and disagreement: A survey of nonverbal audiovisual cues and tools," in Proc. IEEE Int. Conf. Affective Computing and Intelligent Interfaces, 2009, vol. 2.
7. C. Busso, S. Lee, and S. Narayanan, "Analysis of emotionally salient aspects of fundamental frequency for emotion detection," IEEE Trans. Audio, Speech, Lang. Process., vol. 17, no. 4, pp. 582-596, 2009.
8. R. Cai, L. Lu, H.-J. Zhang, and L.-H. Cai, "Highlight sound effects detection in audio stream," in Proc. Int. Conf. Multimedia and Expo, 2003, vol. 3, pp. 37-40.
9. N. Campbell, "Recording techniques for capturing natural everyday speech," in Proc. Language Resources and Evaluation Conf., 2002.
10. N. Campbell, "Whom we laugh with affects how we laugh," in Proc. Workshop Phonetics of Laughter, 2007, pp. 61-65.
11. N. Campbell, H. Kashioka, and R. Ohara, "No laughing matter," in Proc. Eur. Conf. Speech Communication and Technology, 2005, pp. 465-468. (Pubitemid 43908100)
12. J. F. Cohn and K. L. Schmidt, "The timing of facial motion in posed and spontaneous smiles," Int. J. Wavelets Multires. Inf. Process., vol. 2, pp. 121-132, 2005.
13. T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, "Active shape models-their training and application," Comput. Vis. Image Understand., vol. 61, no. 1, p. 38, 1995.
14. E. Douglas-Cowie, R. Cowie, C. Cox, N. Amir, and D. Heylen, "The sensitive artificial listener: An induction technique for generating emotionally coloured conversation," in Proc. Workshopn Corpora for Research on Emotion and Affect, pp. 1-4.
15. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition," IEEE Trans. Multimedia, vol. 2, no. 3, pp. 141-151, 2000.
16. I. Eibl-Eibesfeldt, "The expressive behavior of the deaf-and-blind born," in Social Communication and Movement: Studies of Interaction and Expression in Man and Chimpanzee. New York: Academic, 1973, pp. 163-193.
17. D. P. W. Ellis, PLP and RASTA (and MFCC and Inversion) in Matlab, 2005. [Online]. Available: http://www.ee.columbia.edu/dpwe/resources/ matlab/rastamat.
18. G. Furnas, T. Landauer, L. Gomez, and S. Dumais, "The vocabulary problem in human-system communication," Commun. ACM, vol. 30, no. 11, pp. 964-972, 1987.
19. D. Gonzalez-Jimenez and J. L. Alba-Castro, "Toward pose-invariant 2-D face recognition through point distribution models and facial symmetry," IEEE Trans. Inf. Forensics Security, vol. 2, no. 3, pp. 413-429, 2007. (Pubitemid 47290576)
20. M. Graciarena, E. Shriberg, A. Stolcke, F. Enos, J. Hirschberg, and S. Kajarekar, "Combining prosodic lexical and cepstral systems for deceptive speech detection," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, 2006, vol. 1, pp. 1033-1036.
21. K. Grammer and I. Eibl-Eibesfeldt, "The ritualisation of laughter," in Die Natürlichkeit der Sprache und der Kultur. Bochum, Germany: Brockmeyer, 1990, pp. 192-214.
22. G. Guo and S. Li, "Content-based audio classification and retrieval by support vector machines," IEEE Trans. Neural Netw., vol. 14, no. 1, pp. 209-215, 2003.
23. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computat., vol. 9, no. 8, pp. 1735-1780, 1997. (Pubitemid 127462305)
24. W. Hudenko, W. Stone, and J. Bachorowski, "Laughter differs in children with autism: An acoustic analysis of laughs produced by children with and without the disorder," J. Autism Develop. Disorders, vol. 39, no. 10, pp. 1392-1400, 2009.
25. A. Ito, W. Xinyue, M. Suzuki, and S. Makino, "Smile and laughter recognition using speech processing and face recognition from conversation video," in Proc. Int. Conf. Cyberworlds, 2005, pp. 8-15.
26. A. Janin et al., "The ICSI meeting corpus," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, 2003, vol. 1, pp. 364-367.
27. L. Kennedy and D. Ellis, "Laughter detection in meetings," in Proc. NIST Meeting Recognition Workshop, 2004.
28. S. Kipper and D. Todt, "The role of rhythm and pitch in the evaluation of human laughter," J. Nonverb. Behav., vol. 27, no. 4, pp. 255-272, 2003. (Pubitemid 39015682)
29. M. Knox, N. Morgan, and N. Mirghafori, "Getting the last laugh: Automatic laughter segmentation in meetings," in Proc. INTERSPEECH, 2008, pp. 797-800.
30. K. Laskowski and S. Burger, "Analysis of the occurrence of laughter in meetings," in Proc. INTERSPEECH, 2007, pp. 1258-1261.
31. K. Laskowski and T. Schultz, "Detection of laughter-in-Interaction in multichannel close-talk microphone recordings of meetings," Lecture Notes in Computer Science, vol. 5237, pp. 149-160, 2008.
32. Y. Liu, N. Chawla, M. Harper, E. Shriberg, and A. Stolcke, "A study in machine learning from imbalanced data for sentence boundary detection in speech," Comput. Speech Lang., vol. 20, no. 4, pp. 468-494, 2006. (Pubitemid 44142004)
33. A. Lockerd and F. Mueller, "LAFCAM: Leveraging affective feedback camcorder," in Proc. CHI, Human Factors in Computing Systems, 2002, pp. 574-575.
34. I. Marks, K. Cavanagh, and L. Gega, Hands-on Help: Computer-Aided Psychotherapy. East Sussex, U.K.: Psychology Press Hove, 2007.
35. I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, and V. Karaiskos, "The AMImeeting corpus," in Proc. Int. Conf. Methods and Techniques in Behavioral Research, 2005, pp. 137-140.
36. N. Oostdijk, "The spoken Dutch corpus: Overview and first evaluation," in Proc. Int. Conf. Language Resources and Evaluation, 2000, pp. 887-894.
37. M. Pantic, A. Pentland, A. Nijholt, and T. Huang, "Human computing and machine understanding of human behavior: A survey," Lecture Notes in Computer Science, vol. 4451, pp. 47-71, 2007.
38. M. Pantic and A. Vinciarelli, "Implicit human-centered tagging," IEEE Signal Process. Mag., vol. 26, no. 6, pp. 173-180, 2009.
39. I. Patras and M. Pantic, "Particle filtering with factorized likelihoods for tracking facial features," in Proc. Int. Conf. Automatic Face and Gesture Recognition, 2004, pp. 97-104.
40. S. Petridis, H. Gunes, S. Kaltwang, and M. Pantic, "Static vs. dynamic modeling of human nonverbal behavior from multiple cues and modalities," in Proc. ICMI, 2009, pp. 23-30.
41. S. Petridis and M. Pantic, "Audiovisual discrimination between laughter and speech," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, 2008, pp. 5117-5120.
42. S. Petridis and M. Pantic, "Audiovisual laughter detection based on temporal features," in Proc. ACM Int. Conf. Multimodal Interfaces, 2008, pp. 37-44.
43. S. Petridis and M. Pantic, "Fusion of audio and visual cues for laughter detection," in Proc. ACM Int. Conf. Image and Video Retrieval, 2008, pp. 329-337.
44. S. Petridis and M. Pantic, "Is this joke really funny? Judging the mirth by audiovisual laughter analysis," in Proc. IEEE Int. Conf. Multimedia & Expo, 2009, pp. 1444-1447.
45. G. Potamianos, C. Neti, G. Gravier,A. Garg, and A. W. Senior, "Recent advances in the automatic recognition of audiovisual speech," Proc. IEEE, vol. 91, no. 9, pp. 1306-1326, 2003.
46. R. Provine, "Laughter punctuates speech: Linguistic, social and gender contexts of laughter," Ethology, vol. 95, no. 4, pp. 291-298, 1993.
47. R. Provine, Laughter: A Scientific Investigation. New York: Viking, 2000.
48. R. Provine and Y. Yong, "Laughter: A stereotyped human vocalization," Ethology, vol. 89, no. 2, pp. 115-124, 1991.
49. L. Rabiner and R. Schafer, Digital Processing of Speech Signals. Englewood Cliffs, NJ: Prentice-Hall, 1978.
50. B. Reuderink, M. Poel, K. Truong, R. Poppe, and M. Pantic, "Decision-level fusion for audio-visual laughter detection," Lecture Notes in Computer Science, vol. 5237, pp. 137-148, 2008.
51. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm," in Proc. IEEE Int. Conf. Neural Networks, 1993, vol. 1, pp. 586-591.
52. H. Rothgänger, G. Hauser, A. Cappellini, and A. Guidotti, "Analysis of laughter and speech sounds in Italian and German students," Naturwissenschaften, vol. 85, no. 8, pp. 394-402, 1998. (Pubitemid 28402160)
53. W. Ruch and P. Ekman, "The expressive pattern of laughter," in Emotions, Qualia and Consciousness. Singapore:World Scientific, 2001, pp. 426-443.
54. K. Scherer, "Affect bursts," in Emotions: Essays on Emotion Theory, S. van Goozen, N. van de Poll, and J. Sergeant, Eds. Hillsdale, NJ: Erlbaum, 1994, pp. 161-193.
55. M. Schroeder, D. Heylen, and I. Poggi, "Perception of non-verbal emotional listener feedback," in Proc. Speech Prosody, Dresden, Germany, 2006, pp. 1-4.
56. B. Schueller, A. Batliner, D. Seppi, S. Steidl, T. Vogt, J. Wagner, L. Devillers, L. Vidrascu, N. Amir, L. Kessous, and V. Aharonson, "The relevance of feature type for the automatic classification of emotional user states: Low level descriptors and functionals," in Proc. INTERSPEECH, 2007, pp. 2253-2256.
57. B. Schueller, F. Eyben, and G. Rigoll, "Static and dynamic modelling for the recognition of non-verbal vocalisations in conversational speech," Lecture Notes in Computer Science, vol. 5078, pp. 99-110, 2008.
58. B. Schueller, R. Mueller, B. Hoernler, A. Hoethker, H. Konosu, and G. Rigoll, "Audiovisual recognition of spontaneous interest within conversations," in Proc. ACM Int. Conf. Multimodal Interfaces, 2007, pp. 30-37.
59. J. Trouvain, "Segmenting phonetic units in laughter," in Proc. Int. Conf. Phonetic Sciences, 2003, pp. 2793-2796.
60. K. P. Truong and D. A. van Leeuwen, "Automatic discrimination between laughter and speech," Speech Commun., vol. 49, no. 2, pp. 144-158, 2007. (Pubitemid 46241514)
61. K. P. Truong and D. A. van Leeuwen, "Evaluating laughter segmentation in meetings with acoustic and acoustic-phonetic features," in Proc. Workshop Phonetics of Laughter, 2007.
62. M. F.Valstar, H. Gunes, and M. Pantic, "Howto distinguish posed from spontaneous smiles using geometric features," in Proc. ACM Int. Conf. Multimodal Interfaces, 2007, pp. 38-45.
63. J.Vettin and D. Todt, "Laughter in conversation: Features of occurrence and acoustic structure," J. Nonverb. Behav., vol. 28, no. 2, pp. 93-115, 2004. (Pubitemid 39015675)
64. A. Vinciarelli, M. Pantic, and H. Bourlard, "Social signal processing: Survey of an emerging domain," Image Vis. Comput., vol. 27, no. 12, pp. 1743-1759, 2009.
65. B. Vlasenko, B. Schueller, A. Wendemuth, and G. Rigoll, "Frame vs. turn-level: Emotion recognition from speech considering static and dynamic processing," in Proc. ACII, 2007, pp. 139-147.
66. H. Yehia, T. Kuratate, and E. Vatikiotis-Bateson, "Linking facial animation, head motion and speech acoustics," J. Phonet., vol. 30, no. 3, pp. 555-568, 2002.
67. B. Yin, E. Ambikairajah, and F. Chen, "Combining cepstral and prosodic features in language identification," in Proc. Int. Conf. Pattern Recognition, 2006, vol. 4, pp. 254-257. (Pubitemid 46553409)
68. Z. Zeng, M. Pantic, G. Roisman, and T. Huang, "A survey of affect recognition methods: Audio, visual and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, 2009.