Speech enhancement and recognition in meetings with an audio-visual sensor array

IEEE Transactions on Audio, Speech and Language Processing

Volumn 15, Issue 8, 2007, Pages 2257-2269

  Maganti, Hari Krishna ^a   Gatica Perez, Daniel ^b   McCowan, Iain ^c  

^a UNIVERSITY OF ULM   (Germany)

^b EPFL   (Switzerland)

^c QUEENSLAND UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

Audio x2013; visual fusion; Microphone array processing; Multiobject tracking; Speech enhancement; Speech recognition

Indexed keywords

AUDIO VISUALS; AUDIO-VISUAL SENSORS; BEAM-FORMING TECHNIQUES; INTEGRATED APPROACHES; MICROPHONE ARRAY PROCESSING; MICROPHONE ARRAYS; MULTIOBJECT TRACKING; POST-FILTERING; RECOGNITION PERFORMANCE; SPATIAL FILTERING; SPEAKER TRACKING; SPEECH ACQUISITIONS; SPEECH RECOGNITION SYSTEMS; SPEECH SIGNALS; TABLE TOPS;

ARRAY PROCESSING; BEAMFORMING; COMMUNICATION CHANNELS (INFORMATION THEORY); IMAGE SENSORS; MICROPHONES; QUALITY CONTROL; SPEECH ANALYSIS; SPEECH ENHANCEMENT;

SPEECH RECOGNITION;

EID: 40249089621     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2007.906197     Document Type: Article

References (59)

1. G. Abowd et al., "Living laboratories: The future computing environments group at the Georgia Institute of Technology," in Proc. Conf. Human Factors in Comput. Syst. (CHI), Hague, Apr. 2000, pp. 215-216.
2. F. Asano et al., "Detection and separation of speech event using audio and video information fusion," J. Appl. Signal Process., vol. 11, pp. 1727-1738, 2004.
3. R. Baeza-Yates and B. Ribeiro-Neto,Modern Information Retrieval. New York: ACM, 1999.
4. M. Beal, H. Attias, and N. Jojic, "Audio-video sensor fusion with probabilistic graphical models," in Proc. Eur. Conf. Comput. Vision (ECCV), Copenhagen, May 2002.
5. J. Bitzer, K. S. Uwe, and K. Kammeyer, "Theoretical noise reduction limits of the generalized sidelobe canceller (GSC) for speech enhancement," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 1999, vol. 5, pp. 2965-2968.
6. R. Brunelli et al., "A generative approach to audio-visual person tracking," in Proc. CLEAR Evaluation Workshop, Southampton, U.K., Apr. 2006, pp. 55-68.
7. N. Checka, K. Wilson, M. Siracusa, and T. Darrell, "Multiple person and speaker activity tracking with a particle filter," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Montreal, QC, Canada, May 2004, pp. V-881-V-884.
8. R. Chellapa, C.Wilson, and A. Sirohey, "Human and machine recognition of faces: A survey," Proc. IEEE, vol. 83, no. 5, pp. 705-740, May 1995.
9. Y. Chen and Y. Rui, "Real-time speaker tracking using particle filter sensor fusion," Proc. IEEE, vol. 92, no. 3, pp. 485-494, Mar. 2004.
10. S. M. Chu, E. Marcheret, and G. Potamianos, "Automatic speech recognition and speech activity detection in the chil smart room," in Proc. JointWorkshop Multimodal Interaction and Related Machine Learning Algorithms (MLMI), Edinburgh, U.K., Jul. 2005, pp. 332-343.
11. R. K. Cook, R. V. Waterhouse, R. D. Berendt, S. Edelman, and M. C. Thompson, Jr, "Measurement of correlation coefficients in reverberant sound fields," J. Acoust. Soc. Amer., vol. 27, pp. 1072-1077, 1955.
12. H. Cox, R. Zeskind, and M. Owen, "Robust adaptive beamforming," IEEE Trans. Acoust., Speech. Signal Process., vol. ASSP-35, no. 10, pp. 1365-1376, Oct. 1987.
13. H. Cox, R. Zeskind, and I. Kooij, "Practical supergain," IEEE Trans. Acoust., Speech. Signal Process., vol. ASSP-34, no. 3, pp. 393-397, Jun. 1986.
14. J. Crowley and P. Berard, "Multi-modal tracking of faces for video communications," in Proc. Conf. Comput. Vision Pattern Recognition (CVPR), San Juan, Puerto Rico, Jun. 1997, pp. 640-645.
15. J. DiBiase, "A high-accuracy, low-latency technique for talker localization in reverberant environments," Ph.D. dissertation, Brown Univ., Providence, RI, 2000.
16. J. DiBiase, H. Silverman, and M. Brandstein, "Robust localization in reverberant rooms," in Microphone Arrays. New York: Springer, 2001, vol. 8, pp. 157-180.
17. G. W. Elko, "Superdirectional microphone arrays," in Acoustic Signal Processing for Telecommunication, S. Gay and J. Benesty, Eds. Norwell, MA: Kluwer, 2000, ch. 10, pp. 181-237.
18. A. Doucet, N. de Freitas, and N. Gordon, Sequential Monte Carlo Methods in Practice. New York: Springer-Verlag, 2001.
19. J. Fisher, T. Darrell, W. T. Freeman, and P. Viola, "Learning joint statistical models for audio-visual fusion and segregation," in Proc. Neural Inf. Process. Syst. (NIPS), Denver, CO, Dec. 2000, pp. 772-778.
20. D. Gatica-Perez, G. Lathoud, I. McCowan, and J.-M. Odobez, "A mixed-state i-Particle filter for multi-camera speaker tracking," in Proc. IEEE Conf. Comput. Vision, Workshop on Multimedia Technologies for E-learning and Collaboration(ICCV-WOMTEC), Nice, France, Oct. 2003.
21. D. Gatica-Perez, G. Lathoud, J.-M. Odobez, and I. McCowan, "Multimodal multispeaker probabilistic tracking in meetings," in Proc. IEEE Conf. Multimedia Interfaces (ICMI), Trento, Italy, Oct. 2005, pp. 183-190.
22. D. Gatica-Perez, G. Lathoud, J.-M. Odobez, and I. McCowan, "Audiovisual probabilistic tracking of multiple speakers in meetings," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 2, pp. 601-616, Feb. 2007.
23. J.-L. Gauvain and C.-H. Lee, "Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains," IEEE Trans. Acoust., Speech. Signal Process., vol. 2, no. 2, pp. 291-298, Apr. 1994.
24. S. M. Griebel and M. S. Brandstein, "Microphone array source localization using realizable delay vectors," in Proc. IEEE Workshop Applicat. Signal Process. Audio Acoust. (WASPAA), New York, Oct. 2001, pp. 71-74.
25. T. Hain et al., "The development of the AMI system for the transcription of speech in meetings," in Proc. Joint Workshop Multimodal Interaction and Related Mach. Learn. Algorithms (MLMI), Edinburgh, U.K., Jul. 2005, pp. 344-356.
26. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. Cambridge, U.K.: Cambridge Univ. Press, 2001.
27. M. Isard and A. Blake, "CONDENSATION: Conditional density propagation for visual tracking," Proc. Int. J. Comput. Vision, vol. 29, no. 1, pp. 5-28, 1998.
28. B. Kapralos, M. Jenkin, and E. Milios, "Audio-visual localization of multiple speakers in a video teleconferencing setting," Int. J. Imaging Syst. Technol., vol. 13, pp. 95-105, 2003.
29. N. Katsarakis et al., "3D audiovisual person tracking using Kalman filtering and information theory," in Proc. CLEAR Evaluation Workshop, Southampton, U.K., Apr. 2006, pp. 45-54.
30. Z. Khan, T. Balch, and F. Dellaert, "An MCMC-based particle filter for tracking multiple interacting targets," in Proc. Eur. Conf. Comput. Vision (ECCV), Prague, May 2004, pp. 279-290.
31. J. Kleban and Y. Gong, "HMM adaptation and microphone array processing for distant speech recognition," in Proc. Int. Conf. Acoust. , Speech, Signal Process. (ICASSP), Istanbul, Turkey, Jun. 2000, pp. 1411-1414.
32. C. Knapp and G. Carter, "The generalized correlation method for estimation of time delay," IEEE Trans. Acoust., Speech. Signal Process., vol. ASSP-24, no. 4, pp. 320-327, Aug. 1976.
33. H. Krim and M. Viberg, "Two decades of array signal processing research: The parametric approach," IEEE Signal Process. Mag., vol. 13, no. 4, pp. 67-94, Jul. 1996.
34. G. Lathoud and I. McCowan, "A sector-based approach for localization of multiple speakers with microphone arrays," in Proc. ISCA Workshop Statistical and Perceptual Audio Process. (SAPA), Jeju, Korea, Oct. 2004.
35. C. J. Leggetter and P. C. Woodland, "Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models," Comput. Speech Lang., vol. 9, no. 2, pp. 171-185, 1995.
36. J. S. Liu, Monte Carlo Strategies in Scientific Computing. New York: Springer-Verlag, 2001.
37. M. Lincoln, I. McCowan, J. Vepa, and H. K. Maganti, "The multichannelWall Street Journal audio-visual corpus (MC-WSJ-AV): Specification and initial experiments," in IEEE Autom. Speech Recognition Understanding Workshop (ASRU), San Juan, Puerto Rico, Dec. 2005, pp. 357-362.
38. K. S. Uwe, J. Bitzer, and C. Marro, "Post-filtering techniques," in Microphone Arrays. New York: Springer, 2001, vol. 3, pp. 36-60.
39. M. Wolfel, K. Nickel, and J. McDonough, "Microphone array driven speech recognition: Influence of localization on the word error rate," in Proc. Joint Workshop Multimodal Interaction and Related Mach. Learn. Algorithms (MLMI), Edinburgh, U.K., Jul. 2005, pp. 320-331.
40. C. Marro, Y. Mahieux, and K. U. Simmer, "Analysis of noise reduction and dereverberation techniques based on microphone arrays with postfiltering," IEEE Trans. Speech Audio Process., vol. 6, no. 3, pp. 240-259, May 1998.
41. I. McCowan and H. Bourlard, "Microphone array post-filter based on noise field coherence," IEEE Trans. Speech Audio Process., vol. 11, no. 6, pp. 709-716, Nov. 2003.
42. I. McCowan, M. Hari-Krishna, D. Gatica-Perez, D. Moore, and S. Ba, "Speech acquisition in meetings with an audio-visual sensor array," in Proc. IEEE Int. Conf. Multimedia (ICME), Amsterdam, The Netherlands, Jul. 2005, pp. 1382-1385.
43. J.Meyer and K. U. Simmer, "Multi-channel speech enhancment in a car environment using Wiener filtering and spectral subtraction," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Munich, Germany, Apr. 1997, pp. 1167-1170.
44. N. Morgan, D. Baron, J. Edwards, D. Ellis, D. Gelbart, A. Janin, T. Pfau, E. Shriberg, and A. Stolcke, "The meeting project at ICSI," in Proc. Human Lang. Technol. Conf., San Diego, CA, Mar. 2001, pp. 1-7.
45. D. Moore and I. McCowan, "Microphone array speech recognition: Experiments on overlapping speech in meetings," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Hong Kong, Apr. 2003, pp. V-497-V-500.
46. K. Nickel, T. Gehrig, H. K. Ekenel, J. McDonough, and R. Stiefelhagen, "An audio-visual particle filter for speaker tracking on the CLEAR'06 evaluation dataset," in Proc. CLEAR Evaluation Workshop, Southampton, U.K., Apr. 2006, pp. 69-80.
47. J. G. Fiscus, N. Radde, J. S. Garofolo, A. Le, J. Ajot, and C. Laprun, "The rich transcription 2005 spring meeting recognition evaluation," in Proc. NIST MLMI Meeting Recognition Workshop, Edinburgh, U.K., Jul. 2005, pp. 369-389.
48. M. Omologo, M. Matassoni, P. Svaizer, and D. Giuliani, "Microphone array based speech recognition with different talker-array positions," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP),Munich, Germany, Apr. 1997, pp. 227-230.
49. T. R. al, "WSJCAM0: A British English speech corpus for large vocabulary continuous speech recognition," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Detroit, MI, Apr. 1995, pp. 81-84.
50. S. Roweis, "Factorial models and refiltering for speech separation and denoising," in Proc. Eurospeech Conf. Speech Commun. Technol. (Eurospeech- 2003), Geneva, Switzerland, Sep. 2003, pp. 1009-1012.
51. D. Sturim, M. Brandstein, and H. Silverman, "Tracking multiple talkers using microphone-array measurements," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Munich, Germany, Apr. 1997, pp. 371-374.
52. B. D. V.Veen and K. M. Buckley, "Beamforming:A versatile approach to spatial filtering," IEEE Acoust., Speech, Signal Process. Mag., vol. 5, no. 2, pp. 4-24, Apr. 1988.
53. J. Vermaak, M. Gagnet, A. Blake, and P. Perez, "Sequential Monte Carlo fusion of sound and vision for speaker tracking," in Proc. Int. Conf. Comput. Vision (ICCV), Vancouver, BC, Canada, Jul. 2001, pp. 741-746.
54. A. Waibel, T. Schultz, M. Bett, R. Malkin, I. Rogina, R. Stiefelhagen, and J. Yang, "Smart: The smart meeting room task at ISL," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Hong Kong, Apr. 2003, pp. IV-752-IV-754.
55. D. Ward and R. Williamson, "Particle filter beamforming for acoustic source localization in a reverberant environment," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Orlando, FL, May 2002, pp. 1777-1780.
56. S. J. Young et al., "Multilingual large vocabulary speech recognition: The European SQUALE project," Comput. Speech Lang., vol. 11, no. 1, pp. 73-89, 1997.
57. R. Zelinski, "A microphone array with adaptive post-filtering for noise reduction in reverberant rooms," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), New York, Apr. 1988, pp. 2578-2581.
58. Z. Zhang, "Flexible camera calibration by viewing a plane from unknown orientations," in Proc. Int. Conf. Computer Vision (ICCV), Kerkyra, Greece, Sep. 1999, pp. 666-673.
59. D. Zotkin, R. Duraiswami, and L. Davis, "Multimodal 3-D tracking and event detection via the particle filter," in Proc. Int. Conf. Comput. Vision, Workshop on Detection and Recognition of Events in Video (ICCV-EVENT), Vancouver, BC, Canada, Jul. 2001, pp. 20-27.