Low-Latency Real-Time Meeting Recognition and Understanding Using Distant Microphones and Omni-Directional Camera

IEEE Transactions on Audio, Speech and Language Processing

Volumn 20, Issue 2, 2012, Pages 499-513

  Hori, Takaaki ^a   Araki, Shoko ^a   Yoshioka, Takuya ^a   Fujimoto, Masakiyo ^a   Watanabe, Shinji ^a   Oba, Takanobu ^a   Ogawa, Atsunori ^a   Kinoshita, Keisuke ^a   Nakatani, Tomohiro ^a   Nakamura, Atsushi ^a   Otsuka, Kazuhiro ^a   Mikami, Dan ^a   Yamato, Junji ^a  

^a NTT Communication Science Laboratories   (Japan)

Author keywords

Distant microphones; meeting analysis; speaker diarization; speech enhancement; speech recognition; topic tracking

Indexed keywords

EID: 85008590333     PISSN: 15587916     EISSN: 15587924     Source Type: Journal    
DOI: 10.1109/TASL.2011.2164527     Document Type: Article

References (56)

1. A. Waibel, M. Bett, and M. Finke, “Meeting browser: Tracking and summarizing meetings,” in Proc. DARPA Broadcast News Transcript. Understanding Workshop, 1998, pp. 281–286.
2. A. Janin, J. Ang, S. Bhagat, R. Dhillon, J. Edwards, J. Maaas-Guarasa, N. Morgan, B. Peskin, E. Shriberg, A. Stolcke, C. Wooters, and B. Wrede, “The ICSI meeting project: Resources and research,” in Proc. ICASSP′04 Meeting Recognit. Workshop, 2004.
3. P. Wellner, M. Flynn, and M. Guillemot, “Browsing recorded meetings with Ferret,” in Proc. ICMI-MLMI, 2004, pp. 12–21.
4. F. Asano, K. Yamamoto, J. Ogata, M. Yamada, and M. Nakamura, “Detection and separation of speech events in meeting recordings using a microphone array,” EURASIP J. Audio, Speech, Music Process., vol. 2007, 2007, 10. 1155/2007/27616, article ID 27616.
5. S. Renals, T. Hain, and H. Bourlard, “Recognition and understanding of meetings the AMI and AMIDA projects,” in Proc. ASRU′07, 2007, pp. 238–247.
6. G. Tur, A. Stolcke, L. Voss, J. Dowding, B. Favre, R. Fernandez, M. Frampton, M. Frandsen, C. Frederickson, M. Graciarena, D. Hakkani-Tür, D. Kintzing, K. Leveque, S. Mason, J. Niekrasz, S. Peters, M. Purver, K. Riedhammer, E. Shriberg, J. Tien, D. Vergyri, and F. Yang, “The CALO meeting speech recognition and understanding system,” in Proc. IEEE Workshop Spoken Lang. Technol. (SLT), 2008, pp. 69–72.
7. K. Otsuka, S. Araki, K. Ishizuka, M. Fujimoto, M. Heinrich, and J. Yamato, “A realtime multimodal system for analyzing group meetings by combining face pose tracking and speaker diarization,” in Proc. ICMI, 2008, pp. 257–264.
8. A. Popescu-Belis, P. Poller, and J. Kilgour, “A multimedia retrieval system using speech input,” in Proc. ICMI-MLMI, 2009, pp. 223–224.
9. P. N. Garner, J. Dines, T. Hain, E. A. Hannani, M. Karafiat, D. Korchagin, M. Lincoln, V. Wan, and L. Zhang, “Real-time ASR from meetings,” in Proc. Interspeech, 2009, pp. 2119–2122.
10. M. Fujimoto, K. Ishizuka, and T. Nakatani, “A study of mutual front-end processing method based on statistical model for noise robust speech recognition,” in Proc. Interspeech′09, Sep. 2009, pp. 1235–1238.
11. T. Hori, “NTT speech recognizer with outlook on the next generation: SOLON,” in Proc. NTT Workshop Commun. Scene Anal., 2004, p. SP-6 [Online]. Available: www.kecl.ntt.co.jp/icl/signal/hori/publi-cations/thori_csa2004.pdf.
12. C. Wooters and M. Huijbregts, “The ICSI RT07s speaker diarization system,” in Multimodal Technologies for Perception of Humans: International Evaluation Workshops CLEAR 2007 and RT 2007, Berlin, Heidelberg, Germany, 2008, pp. 509–519, Springer-Verlag.
13. X. Anguera, C. Wooters, and J. Hernando, “Acoustic beamforming for speaker diarization of meetings,” IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 7, pp. 2011–2022, Sep. 2007.
14. H. K. Maganti, D. Gatica-Perez, and I. McCowan, “Speech enhancement and recognition in meetings with an audio-visual sensor array,” IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2257–2269, Nov. 2007.
15. T. Yoshioka, T. Nakatani, T. Miyoshi, and H. G. Okuno, “Blind separation and dereverberation of speech mixtures by joint optimization,” IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 1, pp. 69–84, Jan. 2011.
16. K. Kinoshita, M. Delcroix, T. Nakatani, and M. Miyoshi, “Suppression of late reverberation effect on speech signal using long-term multiple-steplinearprediction,” IEEE Trans. Audio, Speech, Lang. Process., vol. 17, no. 4, pp. 534–545, May 2009.
17. A. Krueger and R. Haeb-Umbach, “Model-based feature enhancement for reverberant speech recognition,” IEEE Trans. Audio, Speech, Lang. Process., vol. 18, no. 7, pp. 1692–1707, Sep. 2010.
18. S. Araki, M. Fujimoto, K. Ishizuka, H. Sawada, and S. Makino, “Speaker indexing and speech enhancement in real meeting/conversations,” in Proc. ICASSP′08, Apr. 2008, vol. I, pp. 93–96.
19. C. H. Knapp and G. C. Carter, “The generalized correlation method for estimation of time delay,” IEEE Trans. Acoust. Speech, Signal Process., vol. 24, no. 4, pp. 320–327, Aug. 1976.
20. S. Araki, H. Sawada, R. Mukai, and S. Makino, “Underdetermined blind sparse source separation for arbitrarily arranged multiple sensors,” Signal Process., vol. 77, no. 8, pp. 1833–1847, Aug. 2007.
21. M. Fujimoto and K. Ishizuka, “Noise robust voice activity detection based on switching Kalman filter,” in Proc. Interspeech′07, Aug. 2007, pp. 2933–2936.
22. J. Sohn, N. S. Kim, and W. Sung, “A statistical model-based voice activity detection,” IEEE Signal Process. Lett., vol. 6, no. 1, pp. 1–3, Jan. 1999.
23. A. Agarwal and Y. M. Cheng, “Two-stage mel-warped Wiener filter for robust speech recognition,” in Proc. ASRU′99, Dec. 1999.
24. S. Watanabe, Y. Minami, A. Nakamura, and N. Ueda, “Variational Bayesian estimation and clustering for speech recognition,” IEEE Trans. Speech Audio Process., vol. 12, no. 4, pp. 365–381, Jul. 2004.
25. E. McDermott, T. J. Hazen, J. Le Roux, A. Nakamura, and S. Kata-giri, “Discriminative training for large vocabulary speech recognition using minimum classification error,” IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 1, pp. 203–223, Jan. 2007.
26. C. J. Leggetter and P. C. Woodland, “Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models,” Comput. Speech Lang., vol. 9, pp. 171–185, 1995.
27. V. Digalakis, D. Rtischev, and L. Neumeyer, “Speaker adaptation using constrained estimation of Gaussian mixtures,” IEEE Trans. Speech Audio Process., vol. 3, no. 5, pp. 357–366, Sep. 1995.
28. M. Federico, “Bayesian estimation methods for n-gram language model adaptation,” in Proc. ICSLP′96, 1996, pp. 240–243.
29. H. Masataki, Y. Sagisaka, K. Hisaki, and T. Kawahara, “Task adaptation using map estimation in n-gram language modeling,” in Proc. ICASSP′97, 1997, vol. 2, pp. 783–786.
30. B. Roark, M. Saraclar, and M. Collins, “Discriminative n-gram language modeling,” Comput. Speech Lang., vol. 21, no. 2, pp. 373–392, 2007.
31. Z. Zhou, J. Gao, F. K. Soong, and H. Meng, “A comparative study of discriminative methods for reranking LVCSR n-best hypotheses in domain adaptation and generalization,” in Proc. ICASSP′06, 2006, vol. 1, pp. 141–144.
32. T. Oba, T. Hori, and A. Nakamura, “A comparative study on methods of weighted language model training for reranking LVCSR n-best hypotheses,” in Proc. ICASSP′10, 2010, pp. 5126–5129.
33. M. Mohri, F. Pereira, and M. Riley, “Weighted finite-state transducers in speech recognition,” Comput. Speech Lang., vol. 16, pp. 69–88, 2002.
34. T. Hori, C. Hori, and Y. Minami, “Fast on-the-fly composition for weighted finite-state transducers in 1. 8 million-word vocabulary continuous speech recognition,” in Proc. Interspeech-ICSLP, 2004, vol. 1, pp. 289–292.
35. T. Hori, C. Hori, Y. Minami, and A. Nakamura, “Efficient WFST-based one-pass decoding with on-the-fly hypothesis rescoring in extremely large vocabulary continuous speech recognition,” IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 4, pp. 1352–1365, May 2007.
36. T. Hori and A. Nakamura, “Generalized fast on-the-fly composition algorithm for WFST-based speech recognition,” in Proc. InterspeechEurospeech, 2005, pp. 557–560.
37. A. Kobayashi, T. Oku, S. Homma, S. Sato, T. Imai, and T. Takagi, “Discriminative rescoring based on minimization of word errors for transcribing broadcast news,” in Proc. Interspeech′08, 2008, pp. 1574–1577.
38. D. Caseiro and I. Trancoso, “A specialized on-the-fly algorithm for lexicon and language model composition,” IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 4, pp. 1281–1291, Jul. 2006.
39. T. Oonishi, P. Dixon, K. Iwano, and S. Furui, “Generalization of specialized on-the-fly composition,” in Proc. ICASSP′09, 2009, pp. 4317–4320.
40. M. Saraclar, M. Riley, E. Bocchieri, and V. Goffin, “Towards automatic closed captioning: Low latency real time broadcast news transcription,” in Proc. ICSLPV2, 2002, pp. 1741–1744.
41. D. Liu and F. Kubala, “Online speaker clustering,” in Proc. ICASSP, 2003, vol. I, pp. 572–575.
42. W. Wang, P. Lv, Q. Zhao, and Y. Yan, “A decision-tree-based online speaker clustering,” Lecture Notes in Comput. Sci., vol. 4477, pp. 555–562, 2007.
43. H. Hung and G. Friedland, “Towards audio-visual on-line diarization of participants in group meetings,” in Proc. Workshop Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications--M2SFA2, 2008.
44. A. Temko, R. Malkin, C. Zieger, D. Macho, C. Nadeu, and M. Omologo, “CLEAR evaluation of acoustic event detection and classification systems,” Multimodal Technol. Percept. Humans, pp. 311–322, 2007.
45. K. Truong and D. Van Leeuwen, “Automatic discrimination between laughter and speech,” Speech Commun., vol. 49, no. 2, pp. 144–158, 2007.
46. Y. Liu, E. Shriberg, S. Andreas, D. Hillard, M. Ostendorf, and M. Harper, “Enriching speech recognition with automatic detection of sentence boundaries and disfluencies,” IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 5, pp. 1526–1540, Sep. 2006.
47. T. Oba, T. Hori, and A. Nakamura, “Improved sequential dependency analysis integrating labeling-based sentence boundary detection,” IEICE Trans. Inf. Syst., vol. E93-D, no. 5, pp. 1272–1281, 2010.
48. T. Iwata, S. Watanabe, T. Yamada, and N. Ueda, “Topic tracking model for analyzing consumer purchase behavior,” in Proc. IJCAF09, 2009, pp. 1427–1432.
49. D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet allocation,” J. Mach. Learn. Res., vol. 3, pp. 993–1022, 2003.
50. S. Watanabe, T. Iwata, T. Hori, A. Sako, and Y. Ariki, “Topic tracking language model for speech recognition,” Comput. Speech Lang., vol. 25, pp. 440–461, 2011.
51. D. Mikami, K. Otsuka, and J. Yamato, “Memory-based particle filter forface pose tracking robust under complex dynamics,” in Proc. CVPR, 2009, pp. 999–1006.
52. S. Furui, K. Maekawa, and H. Isahara, “A Japanese national project on spontaneous speech corpus and processing technology,” in Proc. ASR, 2000, pp. 244–248.
53. R. Kneser and H. Ney, “Improved backing-off for m-gram language modeling,” in Proc. ICASSP′95, 1995, vol. 1, pp. 181–184.
54. T. Fuchi and S. Takagi, “Japanese morphological analyzer using word co-occurrence -JTAG-,” in Proc. COLING-ACU98, 1998, pp. 409–413.
55. “The 2009 (RT-09) Rich Transcription Meeting Recognition Evaluation Plan.” [Online]. Available: http://www.itl.nist.gov/iad/mig/tests/rt/2009/index.html.
56. X. Wei, J. Sun, and X. Wang, “Dynamic mixture models for multiple time-series,” in Proc. IJCAI ′07, 2007, pp. 2909–2914.