The MAHNOB Laughter database

Image and Vision Computing

Volumn 31, Issue 2, 2013, Pages 186-202

  Petridis, Stavros ^a   Martinez, Brais ^a   Pantic, Maja ^a,b  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

Audiovisual; Audiovisual automatic laughter speech; Database; Discrimination; Laughter; Thermal

Indexed keywords

ACOUSTIC NOISE; EXPERIMENTS; SPEECH; SPEECH RECOGNITION;

AUDIOVISUAL; AUDIOVISUAL AUTOMATIC LAUGHTER-SPEECH; DISCRIMINATION; LAUGHTER; THERMAL;

DATABASE SYSTEMS;

EID: 84886398059     PISSN: 02628856     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.imavis.2012.08.014     Document Type: Article

References (79)

1. C. Niemitz, Visuelle zeichen, sprache und gehirn in der evolution des menscheneine entgegnung auf McFarland (Visual signs, language and the brain in the evolution of humans - a reply to McFarland, Z. Sem. 12 (1990) 323-336.
2. W. Ruch, P. Ekman, The expressive pattern of laughter, Emotions, Qualia, and Consciousness, 2001, pp. 426-443.
3. A. Pentland, S. Pentland, Honest Signals: How They Shape Our World, The MIT Press, 2008.
4. M.J. Owren, J.-A. Bachorowski, The evolution of emotional expression: A selfish-gene account of smiling and laughter in early hominids and humans, T.J. Mayne, G.A. Bonanno (Eds.), Emotion: Current Issues and Future Directions, Guilford, New York, 2001, pp. 152-191.
5. R. Provine, Laughter: A Scientific Investigation, Viking, New York, 2000.
6. Eibl-Eibesfeldt, The expressive behavior of the deaf-and-blind born, Social Communication and Movement: Studies of Interaction and Expression in Man and Chimpanzee, 1973, pp. 163-193.
7. M. Makagon, E. Funayama, M. Owren, An acoustic analysis of laughter produced by congenitally deaf and normally hearing college students, J. Acoust. Soc. Am. 124 (2008) 472-483. (Pubitemid 352019336)
8. S. Kipper, D. Todt, The role of rhythmand pitch in the evaluation of human laughter, J. Nonverbal Behav. 27 (4) (2003) 255-272. (Pubitemid 39015682)
9. J.A. Russell, J.A. Bachorowski, J.M. Fernandez-Dols, Facial and vocal expressions of emotion, Annu. Rev. Psychol. 54 (2003) 329-349. (Pubitemid 37417497)
10. M. Pantic, A. Pentland, A. Nijholt, T. Huang, Human computing and machine understanding of human behavior: A survey, Lect. Notes Comput. Sci. 4451 (2007) 47-71.
11. K. Bousmalis, M. Mehu, M. Pantic, Spotting agreement and disagreement: A survey of nonverbal audiovisual cues and tools, IEEE Intl Conf. Affective Computing and Intelligent Interfaces, Vol. 2, 2009.
12. D. Gatica-Perez, Automatic nonverbal analysis of social interaction in small groups: A review, Image Vision Comput. 27 (12) (2009) 1775-1787.
13. M. Pantic, A. Vinciarelli, Implicit human-centered tagging, IEEE Signal Process. Mag. 26 (6) (2009) 173-180, http://dx.doi.org/10.1109/MSP.2009.934186.
14. J. Bachorowski, M. Owren, Not all laughs are alike: voiced but not unvoiced laughter readily elicits positive affect, Psychol. Sci. 12 (3) (2001) 252-257. (Pubitemid 33653539)
15. J. Vettin, D. Todt, Laughter in conversation: features of occurrence and acoustic structure, J. Nonverbal Behav. 28 (2) (2004) 93-115. (Pubitemid 39015675)
16. J.A. Bachorowski, M.J. Smoski, M.J. Owren, The acoustic features of human laughter, J. Acoust. Soc. Am. 110 (1) (2001) 1581-1597.
17. H. Rothgänger, G. Hauser, A. Cappellini, A. Guidotti, Analysis of laughter and speech sounds in Italian and German students, Naturwissenschaften 85 (8) (1998) 394-402. (Pubitemid 28402160)
18. D. Szameitat, K. Alter, A. Szameitat, D. Wildgruber, A. Sterr, C. Darwin, Acoustic profiles of distinct emotional expressions in laughter, J. Acoust. Soc. Am. 126 (2009) 354-366.
19. K.P. Truong, D.A. van Leeuwen, Automatic discrimination between laughter and speech, Speech Commun. 49 (2) (2007) 144-158. (Pubitemid 46241514)
20. B. Schueller, F. Eyben, G. Rigoll, Static and dynamic modelling for the recognition of non-verbal vocalisations in conversational speech, Lect. Notes Comput. Sci. 5078 (2008) 99-110.
21. M. Knox, N. Morgan, N. Mirghafori, Getting the last laugh: Automatic laughter segmentation in meetings, Interspeech, 2008, pp. 797-800.
22. K. Laskowski, T. Schultz, Detection of laughter-in-interaction in multi-channel close-talk microphone recordings of meetings, Lect. Notes Comput. Sci. 5237 (2008) 149-160.
23. L. Kennedy, D. Ellis, Laughter detection in meetings, NIST Meeting Recognition Workshop, 2004.
24. S. Petridis, M. Pantic, Audiovisual discrimination between speech and laughter: why and when visual information might help, IEEE Trans. Multimedia 13 (2) (2011) 216-234.
25. J. Whitehill, G. Littlewort, I. Fasel, M. Bartlett, J. Movellan, Toward practical smile detection, IEEE Trans. Pattern Anal. Mach. Intell. 31 (11) (2009) 2106-2111.
26. M.F. Valstar, H. Gunes, M. Pantic, How to distinguish posed from spontaneous smiles using geometric features, Proc. ACM Int'l Conf. Multimodal Interfaces, 2007, pp. 38-45.
27. S. Petridis, A. Asghar, M. Pantic, Classifying laughter and speech using audio-visual feature prediction, IEEE Int'l Conf. on Acoustics Speech and Signal Processing, 2010, pp. 5254-5257.
28. A. Ito, W. Xinyue, M. Suzuki, S. Makino, Smile and laughter recognition using speech processing and face recognition from conversation video, Intern. Conf. on Cyberworlds, 2005, 2005, pp. 8-15.
29. B. Reuderink, M. Poel, K. Truong, R. Poppe, M. Pantic, Decision-level fusion for audio-visual laughter detection, Lect. Notes Comput. Sci. 5237 (2008) 137-148.
30. S. Scherer, F. Schwenker, N. Campbell, G. Palm, Multimodal laughter detection in natural discourses, Human Centered Robot Systems, 2009, pp. 111-120.
31. J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, et al., The AMI meeting corpus: A pre-announcement, Machine Learning for Multimodal Interaction, 2006, pp. 28-39. (Pubitemid 43979723)
32. N. Campbell, Tools and resources for visualising conversational-speech interaction, Multimodal Corpora, Lecture Notes in Computer Science, 5509, 2009, pp. 176-188.
33. J. Urbain, E. Bevacqua, T. Dutoit, A. Moinet, R. Niewiadomski, C. Pelachaud, B. Picart, J. Tilmanne, J. Wagner, The AVL aughter Cycle database, Int'l Conf. on Language Resources and Evaluation, 2010.
34. M. Valstar, M. Pantic, Induced disgust, happiness and surprise: An addition to the MMI facial expression database, Proceedings of Int'l Conf. Language Resources and Evaluation, Workshop on Emotion, 2010, pp. 65-70.
35. P. Tsiamyrtzis, J. Dowdall, D. Shastri, I. Pavlidis, M.G. Frank, P. Ekman, Imaging facial physiology for the detection of deceit, Int. J. Comput. Vis. 71 (2) (2007) 197-214. (Pubitemid 44359121)
36. S. Wang, Z. Liu, S. Lv, Y. Lv, G. Wu, P. Peng, F. Chen, X. Wang, A natural visible and infrared facial expression database for expression recognition and emotion inference, IEEE Trans. Multimedia 12 (7) (2010) 682-691.
37. K. Grammer, I. Eibl-Eibesfeldt, The ritualisation of laughter, Die Naturlichkeit der Sprache und der Kultur, Brockmeyer, Bochum, 1990, pp. 192-214.
38. W. Hudenko, W. Stone, J. Bachorowski, Laughter differs in children with autism: An acoustic analysis of laughs produced by children with and without the disorder, J. Autism Dev. Disord. 39 (10) (2009) 1392-1400.
39. S. Petridis, M. Pantic, Is this joke really funny? Judging the mirth by audiovisual laughter analysis, Proc. IEEE Intl Conf. Multimedia & Expo, 2009, pp. 1444-1447.
40. B. Martinez, X. Binefa, M. Pantic, Facial component detection in thermal imagery, IEEE Int'l Conf. Computer Vision and Pattern Recognition Workshops, Vol. 3, 2010, pp. 48-54.
41. B. Hern'andez, G. Olague, R. Hammoud, L. Trujillo, E. Romero, Visual learning of texture descriptors for facial expression recognition in thermal imagery, Comput. Vision Image Understanding 106 (2-3) (2007) 258-269. (Pubitemid 46670742)
42. S. Petridis, Audiovisual discrimination between laughter and speech, Ph.D. thesis, Imperial College London (2011).
43. http://mahnob-db.eu/laughter/.
44. http://tcts.fpms.ac.be/~urbain/.
45. http://www.mmifacedb.com/.
46. J. Urbain, R. Niewiadomski, E. Bevacqua, T. Dutoit, A. Moinet, C. Pelachaud, B. Picart, J. Tilmanne, J.Wagner, AVL aughter Cycle, J. Multimodal User Interfaces (2010) 1-12.
47. http://www.speech-data.jp/corpora.html.
48. http://corpus.amiproject.org/.
49. S. Petridis, M. Pantic, J.F. Cohn, Prediction-based classification for audiovisual discrimination between laughter and speech, IEEE Int'l Conf. on Automatic Face and Gesture Recognition, 2011.
50. G. McKeown, M.F. Valstar, R. Cowie, M. Pantic, M. Schroeder, IEEE Transactions on Affective Computing. 3 (1) (2012) pp. 5-17.
51. B. Schueller, R. Mueller, F. Eyben, J. Gast, B. Hoernler, M. Woellmer, G. Rigoll, A. Hoethker, H. Konosu, Being bored? Recognising natural interest by extensive audiovisual integration for real-life application, Image Vision Comput. 27 (12) (2009) 1760-1774.
52. J. Cohn, T. Kruez, I. Matthews, Y. Yang, M. Nguyen, M. Padilla, F. Zhou, F. De la Torre, Detecting depression from facial actions and vocal prosody, Intern. Conf. on ACII 2009, IEEE, 2009, pp. 1-7.
53. http://semaine-db.eu/.
54. F. Eyben, S. Petridis, B. Schueller, G. Tzimiropoulos, S. Zafiriou, M. Pantic, Audiovisual classification of vocal outburst in human conversation using long-short-term memory networks, IEEE Int'l Conf. on Acoustics Speech and Signal Processing, 2011, pp. 5844-5847.
55. S. Sundaram, S. Narayanan, Automatic acoustic synthesis of human-like laughter, J. Acoust. Soc. Am. 121 (1) (2007) 527-535. (Pubitemid 46102798)
56. D. Mowrer, L. La Pointe, J. Case, Analysis of five acoustic correlates of laughter, J. Nonverbal Behav. 11 (3) (1987) 191-199.
57. J. Lichtenauer, J. Shen, M.F. Valstar, M. Pantic, Image and Vision Computing. 29 (2011) pp. 666-680.
58. J. Trouvain, Segmenting phonetic units in laughter, Proc. Int'l Conf. Phonetic Sciences, 2003, pp. 2793-2796.
59. Max Planck Institute for Psycholinguistics Nijmegen, http://www.lat-mpi. eu/tools/elan/.
60. H. Brugman, A. Russel, Annotating multimedia/multi-modal resources with ELAN, Int'l Conf. on Language Resources and Evaluation, 2004, pp. 2065-2068.
61. K. Laskowski, Contrasting emotion-bearing laughter types in multiparticipant vocal activity detection for meetings, IEEE Int'l Conf. on Acoustics Speech and Signal Processing, 2009, pp. 4765-4768.
62. A. Batliner, S. Steidl, F. Eyben, B. Schuller, On laughter and speech laugh, based on observations of child-robot interaction, The Phonetics of Laughing (2011).
63. E. Nwokah, H. Hsu, P. Davies, A. Fogel, The integration of laughter and speech in vocal communication: A dynamic systems perspective, J. Speech Lang. Hear. Res. 42 (4) (1999) 880.
64. P. Boersma, D. Weenink, Praat: doing phonetics by computer (www.praat.org)), Tech. Rep., 2005.
65. M. Knox, N. Mirghafori, Automatic laughter detection using neural networks, Interspeech, 2007, pp. 2973-2976.
66. C. Bickley, S. Hunnicutt, Acoustic analysis of laughter, Second Int'l Conf. on Spoken Language Processing, 1992.
67. I. Patras, M. Pantic, Particle filtering with factorized likelihoods for tracking facial features, Int'l Conf. on Automatic Face and Gesture Recognition, 2004, pp. 97-104.
68. I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, The AMI meeting corpus, Int'l. Conf. on Methods and Techniques in Behavioral Research, 2005, pp. 137-140.
69. D. Gonzalez-Jimenez, J.L. Alba-Castro, Toward pose-invariant 2-d face recognition through point distribution models and facial symmetry, IEEE Trans. Inf. Forensics Secur. 2 (3) (2007) 413-429. (Pubitemid 47290576)
70. L. Rabiner, R. Schafer, Digital Processing of Speech Signals, Prentice Hall, 1978.
71. M. Riedmiller, H. Braun, A direct adaptive method for faster backpropagation learning: The RPROP algorithm, Proc. IEEE Int'l Conf. on Neural Networks, Vol. 1, 1993, pp. 586-591.
72. G. Potamianos, C. Neti, G. Gravier, A. Garg, A.W. Senior, Recent advances in the automatic recognition of audiovisual speech, Proc. IEEE 91 (9) (2003) 1306-1326.
73. Y. Liu, N. Chawla, M. Harper, E. Shriberg, A. Stolcke, A study in machine learning from imbalanced data for sentence boundary detection in speech, Comput. Speech Lang. 20 (4) (2006) 468-494. (Pubitemid 44142004)
74. R. Fisher, The Design of Experiments, Oliver & Boyd, Edinburgh and London, 1935.
75. M. Smucker, J. Allan, B. Carterette, A comparison of statistical significance tests for information retrieval evaluation, ACM Conf. Information and Knowledge Management, 2007, pp. 623-632.
76. S. Dupont, J. Luettin, Audiovisual speech modeling for continuous speech recognition, IEEE Trans. Multimedia 2 (3) (2000) 141-151.
77. Z. Zeng, J. Tu, B. Pianfetti, T. Huang, Audio-visual affective expression recognition through multistream fused HMM, IEEE Trans. Multimedia 10 (4) (2008) 570-577. (Pubitemid 351711233)
78. S. Petridis, M. Pantic, Audiovisual laughter detection based on temporal features, Proc. ACM Int'l Conf. on Multimodal interfaces, 2008, pp. 37-44.
79. D. Johnson, S. Sinanovic, Symmetrizing the kullback-leibler distance, Technical Report, Rice University (2011), http://www.ece.rice.edu/~dhj/resistor. pdf.