Learning bimodal structure in audiovisual data

IEEE Transactions on Neural Networks

Volumn 20, Issue 12, 2009, Pages 1898-1910

  Monaci, Gianluca ^a,b   Vandergheynst, Pierre ^c   Sommer, Friedrich T ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b PHILIPS RESEARCH LABORATORIES   (Netherlands)

^c EPFL   (Switzerland)

Author keywords

Audio visual source localization; Dictionary learning; Matching pursuit (MP); Multimodal data processing; Sparse representation

Indexed keywords

AUDIO-VISUAL; DICTIONARY LEARNING; MATCHING PURSUIT; MULTI-MODAL DATA; SOURCE LOCALIZATION; SPARSE REPRESENTATION;

DATA STRUCTURES; EDUCATION; LEARNING ALGORITHMS;

DATA PROCESSING;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUDITORY STIMULATION; COMPUTER SIMULATION; DISCRIMINATION LEARNING; HEARING; HUMAN; LEARNING; PHOTOSTIMULATION; PHYSIOLOGY; RECOGNITION; SPEECH; VISION;

ACOUSTIC STIMULATION; ALGORITHMS; ARTIFICIAL INTELLIGENCE; AUDITORY PERCEPTION; COMPUTER SIMULATION; DISCRIMINATION LEARNING; HUMANS; LEARNING; PHOTIC STIMULATION; RECOGNITION (PSYCHOLOGY); SPEECH; VISUAL PERCEPTION;

EID: 72149118713     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2009.2032182     Document Type: Article

References (58)

1. J. Driver and T. Noesselt, "Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgements," Neuron, vol.57, no.1, pp. 11-23, 2008.
2. D. A. Bulkin and J. M. Groh, "Seeing sounds: Visual and auditory interactions in the brain," Current Opinion Neurobiol., vol.16, no.4, pp. 415-419, 2006.
3. C. E. Schroeder and J. J. Foxe, "Multisensory contributions to lowlevel, 'unisensory' processing," Current Opinion Neurobiol., vol.15, no.4, pp. 454-458, 2005.
4. S. Shimojo and L. Shams, "Sensory modalities are not separate modalities: Plasticity and interactions," Current Opinion Neurobiol., vol.11, no.4, pp. 505-509, 2001.
5. R. Sekuler, A. Sekuler, and R. Lau, "Sound alters visual motion perception," Nature, vol.385, no.6614, pp. 308-308, 1997.
6. H. McGurk and J. W. MacDonald, "Hearing lips and seeing voices," Nature, vol.264, no.5588, pp. 746-748, 1976.
7. J. Driver, "Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading," Nature, vol.381, pp. 66-68, 1996.
8. J.-P. Bresciani, F. Dammeier, and M. Ernst, "Vision and touch are automatically integrated for the perception of sequences of events," J. Vis., vol.6, no.5, pp. 554-564, 2006.
9. G. Potamianos, C. Neti, G. Gravier,A. Garg, and A. W. Senior, "Recent advances in the automatic recognition of audiovisual speech," Proc. IEEE, vol.91, no.9, pp. 1306-1326, Sep. 2003.
10. C. Zhang, P. Yin, Y. Rui, R. Cutler, and P. Viola, "Boosting-based multimodal speaker detection for distributed meetings," in Proc. IEEE Int. Workshop Multimedia Signal Process., 2006, pp. 1541-1552.
11. P. Besson, V. Popovici, J.-M. Vesin, J.-P. Thiran, and M. Kunt, "Extraction of audio features specific to speech production for multimodal speaker detection," IEEE Trans. Multimedia, vol.10, no.1, pp. 63-73, Jan. 2008.
12. R. Goecke, G. Potamianos, and C. Neti, "Noisy audio feature enhancement using audio-visual speech data," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2002, pp. 2025-2028.
13. W. Wang, D. Cosker, Y. Hicks, S. Saneit, and J. Chambers, "Video assisted speech source separation," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2005, vol.5, pp. 425-428.
14. B. Rivet, L. Girin, and C. Jutten, "Mixing audiovisual speech processing and blind source separation for the extraction of speech signals from convolutive mixtures," IEEE Trans. Audio Speech Lang. Process., vol.15, no.1, pp. 96-108, Jan. 2007.
15. B. Rivet, L. Girin, and C. Jutten, "Visual voice activity detection as a help for speech source separation from convolutive mixtures," Speech Commun., vol.49, no.7, pp. 667-677, 2007.
16. A. Llagostera, G. Monaci, P. Vandergheynst, and R. Gribonval, "Blind audiovisual source separation using overcomplete dictionaries," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2008, pp. 1841-1844.
17. J. Hershey and J. Movellan, "Audio-vision: Using audio-visual synchrony to locate sounds," in Proc. Neural Inf. Process. Signal, 1999, vol.12, pp. 813-819.
18. M. Slaney and M. Covell, "FaceSync: A linear operator for measuring synchronization of video facial images and audio tracks," in Proc. Neural Inf. Process. Signal, 2000, vol.13, pp. 814-820.
19. H. J. Nock, G. Iyengar, and C. Neti, "Speaker localisation using audiovisual synchrony: An empirical study," in Proc. ACM Int. Conf. Image Video Retrieval, 2003, pp. 488-499.
20. J. W. Fisher, III and T. Darrell, "Speaker association with signal-level audiovisual fusion," IEEE Trans. Multimedia, vol.6, no.3, pp. 406-413, Jun. 2004.
21. E. Kidron, Y. Schechner, and M. Elad, "Cross-modal localization via sparsity," IEEE Trans. Signal Process., vol.55, no.4, pp. 1390-1404, Apr. 2007.
22. M. Gurban and J.-P. Thiran, "Multimodal speaker localization in a probabilistic framework," in Proc. Eur. Signal Process. Conf., 2006.
23. M. R. Siracusa and J. W. Fisher, "Dynamic dependency tests: Analysis and applications to multi-modal data association," in Proc. Int. Conf. Artif. Intell. Statist., 2007.
24. G. Monaci, Ò. D. Escoda, and P. Vandergheynst, "Analysis of multimodal sequences using geometric video representations," Signal Process., vol.86, no.12, pp. 3534-3548, 2006.
25. G. Monaci and P.Vandergheynst, "Audiovisual gestalts," in Proc. IEEE CVPR Workshop Perceptual Organization Comput. Vis., 2006.
26. G. Monaci, P. Jost, P. Vandergheynst, B. Mailhe, S. Lesage, and R. Gribonval, "Learning multi-modal dictionaries," IEEE Trans. Image Process., vol.16, no.9, pp. 2272-2283, Sep. 2007.
27. S. Mallat and Z. Zhang, "Matching pursuits with time-frequency dictionaries," IEEE Trans. Signal Process., vol.41, no.12, pp. 3397-3415, Dec. 1993.
28. S. Mallat, A Wavelet Tour of Signal Processing. New York: Academic, 1998.
29. S. S. Chen, D. L. Donoho, and M. A. Saunders, "Atomic decomposition by basis pursuit," SIAM J. Sci. Comput., vol.20, no.1, pp. 33-61, 1998.
30. P. Frossard, P. Vandergheynst, R. M. F. i Ventura, and M. Kunt, "A posterior quantization of progressive matching pursuit streams," IEEE Trans. Signal Process., vol.52, no.2, pp. 525-535, Feb. 2004.
31. D. L. Donoho and M. Elad, "Optimal sparse representation in general (nonorthogoinal) dictionaries via l1 minimization," Proc. Nat. Acad. Sci., vol.100, pp. 2197-2202, 2003.
32. E. J. Candès and D. L. Donoho, "Curvelets-A surprisingly effective nonadaptive representation for objects with edges," in Curve and Surface Fitting, A. Cohen, C. Rabut, and L. Schmaker, Eds. Nashville, TN: Vanderbilt Univ. Press, 1999.
33. H. B. Barlow, "The coding of sensory messages," in Current Problems in Animal Behaviour, W. H. Thorpe and O. L. Zangwill, Eds. Cambridge, U.K.: Cambridge Univ. Press, 1961.
34. B. A. Olshausen and D. J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by V1?," Vis. Res., vol.37, pp. 3311-3327, 1997.
35. M. Rehn and F. T. Sommer, "A network that uses few active neurons to code visual input predicts the diverse shapes of cortical receptive fields," J. Comput. Neurosci., vol.22, no.2, pp. 135-146, 2007.
36. M. S. Lewicki and T. J. Sejnowski, "Learning overcomplete representations," Neural Comput., vol.12, no.2, pp. 337-365, 2000.
37. E. C. Smith and M. S. Lewicki, "Efficient auditory coding," Nature, vol.439, no.7079, pp. 978-982, 2006.
38. T. Blumensath and M. Davies, "Sparse and shift-invariant representations of music," IEEE Trans. Audio Speech Lang. Process., vol.14, no.1, pp. 50-57, Jan. 2006.
39. R. Grosse, R. Raina, H. Kwong, and A. Y. Ng, "Shift-invariant sparse coding for audio classification," in Proc. Conf. Uncertainty Artif. Intell., 2007, pp. 149-158.
40. B. Mailhé, S. Lesage, R. Gribonval, and F. Bimbot, "Shift-invariant dictionary learning for sparse representations: Extending K-SVD," in Proc. Eur. Signal Process. Conf., 2008.
41. P. Smaragdis, B. Raj, and M. V. Shashanka, "Sparse and shift-invariant feature extraction from non-negative data," in Proc. Int. Conf. Acoust. Speech Signal Process., 2008, pp. 2069-2072.
42. H. Wersing, J. Eggert, and E. Körner, "Sparse coding with invariance constraints," in Proc. Int. Conf. Artif. Neural Netw., 2003, pp. 385-392.
43. M. Elad and M. Aaron, "Image denoising via sparse and redundant representations over learned dictionaries," IEEE Trans. Image Process., vol.15, no.12, pp. 3736-3745, Dec. 2006. (Pubitemid 44811686)
44. B. A. Olshausen, "Sparse codes and spikes," in Probabilistic Models of the Brain: Perception and Neural Function, R. P. N. Rao, B. A. Olshausen, and M. S. Lewicki, Eds. Cambridge, MA:MIT Press, 2002.
45. M. Elad, M. Aharon, and A. M. Bruckstein, "The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation," IEEE Trans. Signal Process., vol.54, no.11, pp. 4311-4322, Nov. 2006.
46. G. Davis, S. Mallat, and M. Avellaneda, "Adaptive greedy approximations," J. Constructive Approx., vol.13, no.1, pp. 57-98, 1997.
47. I. F. Gorodnitsky and B. D. Rao, "Sparse signal reconstruction from limited data using focuss: Are-weighted minimum norm algorithm," IEEE Trans. Signal Process., vol.45, no.3, pp. 600-616, Mar. 1997.
48. J. A. Tropp, A. C. Gilbert, and M. J. Strauss, "Algorithms for simultaneous sparse approximation. Part I: Greedy pursuit," Signal Process., vol.86, no.3, pp. 572-588, 2006.
49. H. B. Barlow, "What is the computational goal of the neocortex?," in Large-Scale Neuronal Theories of the Brain, C. Koch and J. L. Davis, Eds. Cambridge, MA: MIT Press, 1994.
50. M. McGrath and Q. Summerfield, "Intermodal timing relations and audiovisual speech recognition by normal-hearing adults," J. Acoust. Soc. Amer., vol.77, no.2, pp. 678-685, 1985.
51. L. M. Miller and M. D'Esposito, "Perceptual fusion and stimulus coincidence in the cross-modal integration of speech," J. Neurosci., vol.25, no.25, pp. 5884-5893, 2005.
52. J. Chen and X. Huo, "Theoretical results on sparse representations of multiple-measurement vectors," IEEE Trans. Signal Process., vol.54, no.12, pp. 4634-4643, Dec. 2006.
53. S. F. Cotter, B. D. Rao, K. Engan, and K. Kreutz-Delgado, "Sparse solutions to linear inverse problems with multiple measurement vectors," IEEE Trans. Signal Process., vol.53, no.7, pp. 2477-2488, Jul. 2005.
54. D. Leviatan and V. Temlyakov, "Simultaneous approximation by greedy algorithms," Adv. Comput. Math., vol.25, no.1-3, pp. 73-90, 2006.
55. E. K. Patterson, S. Gurbuz, Z. Tufekci, and J. N. Gowdy, "Moving-talker, speaker-independent feature study, and baseline results using the CUAVE multimodal speech corpus," EURASIP J. Appl. Signal Process., vol.2002, no.11, pp. 1189-1201, 2002.
56. M. Riesenhuber and T. Poggio, "Hierarchical models of object recognition in cortex," Nature Neurosci., vol.2, pp. 1019-1025, 1999.
57. T. Masquelier and S. J. Thorpe, "Unsupervised learning of visual features through spike timing dependent plasticity," PLoS Comput. Biol., vol.3, no.2, p. e31, 2007.
58. R. E. Turner and M. Sahani, "Modeling natural sounds with modulation cascade processes," in Proc. Neural Inf. Process. Syst., 2008, vol.20, pp. 1545-1552.