Visually Indicated Sounds

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn 2016-December, Issue , 2016, Pages 2405-2413

  Owens, Andrew ^a   Isola, Phillip ^a,b   McDermott, Josh ^a   Torralba, Antonio ^a   Adelson, Edward H ^a   Freeman, William T ^a,c  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; RECURRENT NEURAL NETWORKS;

EXAMPLE-BASED SYNTHESIS; PHYSICAL INTERACTIONS; PSYCHOPHYSICAL EXPERIMENTS; VISUAL SCENE; WAVE FORMS;

PATTERN RECOGNITION;

EID: 84986249782     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2016.264     Document Type: Conference Paper

References (46)

1. P. Agrawal, J. Carreira, and J. Malik. Learning to see by moving. In ICCV, 2015.
2. A. Arnab, M. Sapienza, S. Golodetz, J. Valentin, O. Miksik, S. Izadi, and P. H. S. Torr. Joint object-material category segmentation from audio-visual cues. In BMVC, 2015.
3. R. Baillargeon. The acquisition of physical knowledge in infancy: A summary in eight lessons. Blackwell handbook of childhood cognitive development, 1:46-83, 2002.
4. S. Bell, P. Upchurch, N. Snavely, and K. Bala. Material recognition in the wild with the materials in context database. CoRR, abs/1412.0623, 2014.
5. J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury, M. Davies, and M. B. Sandler. A tutorial on onset detection in music signals. Speech and Audio Processing, IEEE Transactions on, 13(5):1035-1047, 2005.
6. T. Bonebright. Were those coconuts or horse hoofs? visual context effects on identification and perceived veracity of everyday sounds. In International Conference on Auditory Display, 2012.
7. S. Cavaco and M. S. Lewicki. Statistical modeling of intrinsic structures in impacts sounds. The Journal of the Acoustical Society of America, 121(6):3558-3568, 2007.
8. A. Davis, K. L. Bouman, M. Rubinstein, F. Durand, and W. T. Freeman. Visual vibrometry: Estimating material properties from small motion in video. In CVPR, 2015.
9. A. Davis, M. Rubinstein, N. Wadhwa, G. J. Mysore, F. Durand, and W. T. Freeman. The visual microphone: passive recovery of sound from video. ACM Transactions on Graphics (TOG), 2014.
10. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
11. C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. ICCV, 2015.
12. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. CVPR, 2015.
13. K. Fukunaga and L. D. Hostetler. The estimation of the gradient of a density function, with applications in pattern recognition. Information Theory, IEEE Transactions on, 21(1):32-40, 1975.
14. W.W. Gaver. What in the world do we hear?: An ecological approach to auditory event perception. Ecological psychology, 1993.
15. M. Gemici and A. Saxena. Learning haptic representation for manipulating deformable food objects. In IROS, 2014.
16. B. R. Glasberg and B. C. Moore. Derivation of auditory filter shapes from notched-noise data. Hearing research, 47(1):103-138, 1990.
17. R. Goroshin, J. Bruna, J. Tompson, D. Eigen, and Y. LeCun. Unsupervised feature learning from temporal data. arXiv preprint arXiv:1504.02518, 2015.
18. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
19. P.-J. Hsieh, J. T. Colas, and N. Kanwisher. Spatial pattern of bold fmri activation reveals cross-modal information in auditory cortex. Journal of neurophysiology, 2012.
20. Y. Hu and P. C. Loizou. Speech enhancement based on wavelet thresholding the multitaper spectrum. Speech and Audio Processing, IEEE Transactions on, 12(1):59-67, 2004.
21. P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Learning visual groups from co-occurrences in space and time. arXiv preprint arXiv:1511.06811, 2015.
22. D. Jayaraman and K. Grauman. Learning image representations tied to ego-motion. In ICCV, December 2015.
23. S. Ji, W. Xu, M. Yang, and K. Yu. 3d convolutional neural networks for human action recognition. IEEE TPAMI, 2013.
24. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia, pages 675-678. ACM, 2014.
25. M. Kac. Can one hear the shape of a drum? The american mathematical monthly, 1966.
26. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
27. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
28. Z.-H. Ling, S.-Y. Kang, H. Zen, A. Senior, M. Schuster, X.-J. Qian, H. M. Meng, and L. Deng. Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends. IEEE Signal Processing Magazine, 2015.
29. R. A. Lutfi. Human sound source identification. In Auditory perception of sound sources, pages 13-42. Springer, 2008.
30. J. H. McDermott and E. P. Simoncelli. Sound texture perception via statistics of the auditory periphery: evidence from sound synthesis. Neuron, 71(5):926-940, 2011.
31. H. Mobahi, R. Collobert, and J. Weston. Deep learning from temporal coherence in video. In ICML, 2009.
32. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodal deep learning. In ICML, 2011.
33. L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. arXiv preprint arXiv:1509.06825, 2015.
34. L. Schulz. The origins of inquiry: Inductive inference and exploration in early childhood. Trends in cognitive sciences, 16(7):382-389, 2012.
35. A. A. Shabana. Theory of vibration: an introduction. Springer Science & Business Media, 1995.
36. L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson. Recognizing materials using perceptually inspired features. International journal of computer vision, 103(3):348-371, 2013.
37. M. H. Siegel, R. Magid, J. B. Tenenbaum, and L. E. Schulz. Black boxes: Hypothesis testing via indirect perceptual evidence. Proceedings of the 36th Annual Conference of the Cognitive Science Society, 2014.
38. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems, 2014.
39. J. Sinapov, M. Wiemer, and A. Stoytchev. Interactive learning of the acoustic properties of household objects. In ICRA, 2009.
40. M. Slaney. Pattern playback in the 90s. In NIPS, pages 827-834, 1994.
41. L. Smith and M. Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13-29, 2005.
42. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958, 2014.
43. K. Van Den Doel, P. G. Kry, and D. K. Pai. Foleyautomatic: physically-based sound effects for interactive simulation and animation. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 537-544. ACM, 2001.
44. C. Vondrick, H. Pirsiavash, and A. Torralba. Anticipating the future by watching unlabeled video. arXiv preprint arXiv:1504.08023, 2015.
45. X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
46. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In NIPS, 2014.