The visual microphone: Passive recovery of sound from video

ACM Transactions on Graphics

Volumn 33, Issue 4, 2014, Pages

  Davis, Abe ^a   Rubinstein, Michael ^a,b   Wadhwa, Neal ^a   Mysore, Gautham J ^c   Durand, Frédo ^a   Freeman, William T ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b MICROSOFT RESEARCH   (United States)

^c ADOBE RESEARCH   (United States)

Author keywords

Remote sound acquisition; Sound from video; Visual acoustics

Indexed keywords

DATA VISUALIZATION; INTERACTIVE COMPUTER GRAPHICS; MICROPHONES; RECOVERY; AUDIO ACOUSTICS; HIGH SPEED CAMERAS;

AUDIO SAMPLES; FRAME-RATE; HIGH-SPEED VIDEO; REMOTE SOUND ACQUISITION; ROLLING SHUTTERS; SPATIAL RESOLUTION; VIBRATION MODES; HIGH SPEED; HIGH-SPED VIDEO; OBJECT SURFACE; PROPERTY; SOUND FROM VIDEO; VISUAL ACOUSTIC;

AUDIO ACOUSTICS; RECOVERY;

EID: 84905749657     PISSN: 07342071     EISSN: 15577333     Source Type: Journal    
DOI: 10.1145/2601097.2601119     Document Type: Conference Paper

References (29)

1. AIT-AIDER, O., BARTOLI, A., AND ANDREFF, N. 2007. Kinematics from lines in a single rolling shutter image. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on, IEEE, 1-6.
2. BOLL, S. 1979. Suppression of acoustic noise in speech using spectral subtraction. Acoustics, Speech and Signal Processing, IEEE Transactions on 27, 2, 113-120.
3. CHEN, J., WADHWA, N., CHA, Y.-J., DURAND, F., FREEMAN, W. T., AND BUYUKOZTURK, O. 2014. Structural modal identification through high speed camera video: Motion magnification. Proceedings of the 32nd International Modal Analysis Conference (to appear).
4. DE CHEVEIGNÉ, A., AND KAWAHARA, H. 2002. Yin, a fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America 111, 4, 1917-1930.
5. D'EMILIA, G., RAZZÈ, L., AND ZAPPA, E. 2013. Uncertainty analysis of high frequency image-based vibration measurements. Measurement 46, 8, 2630-2637.
6. FISHER, W. M., DODDINGTON, G. R., AND GOUDIE-MARSHALL, K. M. 1986. The darpa speech recognition research database: specifications and status. In Proc. DARPA Workshop on speech recognition, 93-99.
7. GAUTAMA, T., AND VAN HULLE, M. 2002. A phase-based approach to the estimation of the optical flow field using spatial filtering. Neural Networks, IEEE Transactions on 13, 5 (sep), 1127-1136.
8. GRUNDMANN, M., KWATRA, V., CASTRO, D., AND ESSA, I. 2012. Calibration-free rolling shutter removal. In Computational Photography (ICCP), 2012 IEEE International Conference on, IEEE, 1-8.
9. HANSEN, J. H., AND PELLOM, B. L. 1998. An effective quality evaluation protocol for speech enhancement algorithms. In ICSLP, vol. 7, 2819-2822.
10. JANSSEN, A., VELDHUIS, R., AND VRIES, L. 1986. Adaptive interpolation of discrete-time signals that can be modeled as autoregressive processes. Acoustics, Speech and Signal Processing, IEEE Transactions on 34, 2, 317-330.
11. JANSSON, E., MOLIN, N.-E., AND SUNDIN, H. 1970. Resonances of a violin body studied by hologram interferometry and acoustical methods. Physica scripta 2, 6, 243.
12. LIU, C., TORRALBA, A., FREEMAN, W. T., DURAND, F., AND ADELSON, E. H. 2005. Motion magnification. ACM Trans. Graph. 24 (Jul), 519-526.
13. LOIZOU, P. C. 2005. Speech enhancement based on perceptually motivated bayesian estimators of the magnitude spectrum. Speech and Audio Processing, IEEE Transactions on 13, 5, 857-869.
14. MEINGAST, M., GEYER, C., AND SASTRY, S. 2005. Geometric models of rolling-shutter cameras. arXiv preprint cs/0503076.
15. MORLIER, J., SALOM, P., AND BOS, F. 2007. New image processing tools for structural dynamic monitoring. Key Engineering Materials 347, 239-244.
16. NAKAMURA, J. 2005. Image sensors and signal processing for digital still cameras. CRC Press.
17. POE, E. A. 1845. The Raven.
18. PORTILLA, J., AND SIMONCELLI, E. P. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int. J. Comput. Vision 40, 1 (Oct.), 49-70.
19. POWELL, R. L., AND STETSON, K. A. 1965. Interferometric vibration analysis by wavefront reconstruction. JOSA 55, 12, 1593-1597.
20. QUACKENBUSH, S. R., BARNWELL, T. P., AND CLEMENTS, M. A. 1988. Objective measures of speech quality. Prentice Hall Englewood Cliffs, NJ.
21. ROTHBERG, S., BAKER, J., AND HALLIWELL, N. A. 1989. Laser vibrometry: pseudo-vibrations. Journal of Sound and Vibration 135, 3, 516-522.
22. RUBINSTEIN, M. 2014. Analysis and Visualization of Temporal Variations in Video. PhD thesis, Massachusetts Institute of Technology.
23. SIMONCELLI, E. P., FREEMAN, W. T., ADELSON, E. H., AND HEEGER, D. J. 1992. Shiftable multi-scale transforms. IEEE Trans. Info. Theory 2, 38, 587-607.
24. STANBRIDGE, A., AND EWINS, D. 1999. Modal testing using a scanning laser doppler vibrometer. Mechanical Systems and Signal Processing 13, 2, 255-270.
25. TAAL, C. H., HENDRIKS, R. C., HEUSDENS, R., AND JENSEN, J. 2011. An algorithm for intelligibility prediction of time-frequency weighted noisy speech. Audio, Speech, and Language Processing, IEEE Transactions on 19, 7, 2125-2136.
26. WADHWA, N., RUBINSTEIN, M., DURAND, F., AND FREEMAN, W. T. 2013. Phase-based video motion processing. ACM Transactions on Graphics (TOG) 32, 4, 80.
27. WADHWA, N., RUBINSTEIN, M., DURAND, F., AND FREEMAN, W. T. 2014. Riesz pyramid for fast phase-based video magnification. In Computational Photography (ICCP), 2014 IEEE International Conference on, IEEE.
28. WU, H.-Y., RUBINSTEIN, M., SHIH, E., GUTTAG, J., DURAND, F., AND FREEMAN, W. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics (TOG) 31, 4, 65.
29. ZALEVSKY, Z., BEIDERMAN, Y., MARGALIT, I., GINGOLD, S., TEICHER, M., MICO, V., AND GARCIA, J. 2009. Simultaneous remote extraction of multiple speech sources and heart beats from secondary speckles pattern. Opt. Express 17, 24, 21566-21580.