Full-frame video stabilization with motion inpainting

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 28, Issue 7, 2006, Pages 1150-1163

  Matsushita, Yasuyuki ^a   Ge, Weina ^b   Tang, Xiaoou ^a   Shum, Heung Yeung ^a  

^a MICROSOFT RESEARCH ASIA   (China)

^b The Pennsylvania State University   (United States)

Author keywords

motion inpainting; sharpning and deblurring; Video analysis; video completion; video enhancement; video stabilization

Indexed keywords

ALGORITHM; ARTICLE; ARTIFACT; COMPUTER ASSISTED DIAGNOSIS; COMPUTER GRAPHICS; IMAGE ENHANCEMENT; IMAGE SUBTRACTION; INFORMATION RETRIEVAL; MATHEMATICAL COMPUTING; METHODOLOGY; MOTION; PAINTING; PHOTOGRAPHY; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY; SIGNAL PROCESSING; VIDEORECORDING;

ALGORITHMS; ARTIFACTS; COMPUTER GRAPHICS; IMAGE ENHANCEMENT; IMAGE INTERPRETATION, COMPUTER-ASSISTED; INFORMATION STORAGE AND RETRIEVAL; MOTION; NUMERICAL ANALYSIS, COMPUTER-ASSISTED; PAINTINGS; PHOTOGRAPHY; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY; SIGNAL PROCESSING, COMPUTER-ASSISTED; SUBTRACTION TECHNIQUE; VIDEO RECORDING;

EID: 33746390629     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2006.141     Document Type: Article

References (44)

1. I. Zoghlami, O. Faugeras, and R. Deriche, “Using Geometric Corners to Build a 2D Mosaic from a Set of Images,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 421–425, 1997.
2. D. Capel and A. Zisserman, “Automated Mosaicing with Super-Resolution Zoom,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 885–891, 1998.
3. A. Cenci, A. Fusiello, and V. Roberto, “Image Stabilization by Feature Tracking,” Proc. 10th Int'l Conf. Image Analysis and Processing, pp. 665–670, 1999.
4. M. Brown and D. Lowe, “Recognizing Panoramas,” Proc. Int'l Conf. Computer Vision, pp. 1218–1225, 2003.
5. J. Bergen, P. Anandan, K. Hanna, and R. Hingorani, “Hierarchical Model-Based Motion Estimation,” Proc. Second European Conf. Computer Vision, pp. 237–252, 1992.
6. R. Szeliski, “Image Mosaicing for Tele-Reality Applications,” technical report, CRL-Digital Equipment Corp., 1994.
7. T.-J. Cham and R. Cipolla, “A Statistical Framework for Longrange Feature Matching in Uncalibrated Image Mosaicing,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 442-447,1998.
8. Z. Zhu, G. Xu, Y. Yang, and J. Jin, “Camera Stabilization Based on 2.5D Motion Estimation and Inertial Motion Filtering,” Proc. IEEE Int'l Conf. Intelligent Vehicles, vol. 2, pp. 329–334, 1998.
9. R. Szeliski, “Image Alignment and Stitching: A Tutorial,” Technical Report MSR-TR-2004-92, Microsoft Corp., 2004.
10. M. Hansen, P. Anandan, K. Dana, G. van der Wal, and P. Burt, “Real-Time Scene Stabilization and Mosaic Construction,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 54-62,1994.
11. C. Buehler, M. Bosse, and L. McMillan, “Non-Metric Image-Based Rendering for Video Stabilization,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 609–614, 2001.
12. Z. Duric and A. Resenfeld, “Shooting a Smooth Video with a Shaky Camera,” J. Machine Vision and Applications, vol. 13, pp. 303–313, 2003.
13. M. Pilu, “Video Stabilization as a Variational Problem and Numerical Solution with the Viterbi Method,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 625–630, 2004.
14. A. Litvin, J. Konrad, and W. Karl, “Probabilistic Video Stabilization Using Kalman Filtering and Mosaicking,” Proc. ISεT/SPIE Symp. Electronic Imaging, Image, and Video Comm., pp. 663–674, 2003.
15. Y. Wexler, E. Shechtman, and M. Irani, “Space-Time Video Completion,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 120–127, 2004.
16. J. Jia, T. Wu, Y. Tai, and C. Tang, “Video Repairing: Inference of Foreground and Background under Severe Occlusion,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 364–371, 2004.
17. V. Cheung, B.J. Frey, and N. Jojic, “Video Epitomes,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 42–49, 2005.
18. R. Fabian and D. Malah, “Robust Identification of Motion and Out-of-Focus Blur Parameters from Blurred and Noisy Images,” CVGIP: Graphical Models and Image Processing, vol. 53, no. 5, pp. 403–412, 1991.
19. B. Bascle, A. Blake, and A. Zisserman, “Motion Deblurring and Super-Resolution from an Image Sequence,” Proc. Fourth European Conf. Computer Vision, vol. 2, pp. 573–582, 1996.
20. P. Jansson, Deconvolution of Image and Spectra, second ed. Academic Press, 1997.
21. A. Rav-Acha and S. Peleg, “Restoration of Multiple Images with Motion Blur in Different Directions,” Proc. Fifth IEEE Workshop Application of Computer Vision, pp. 22–28, 2000.
22. Y. Yitzhaky, G. Boshusha, Y. Levy, and N. Kopeika, “Restoration of an Image Degraded by Vibrations Using Only a Single Frame,” Optical Eng., vol. 39, no. 8, pp. 2083–2091, 2000.
23. M. Ben-Ezra and S. Nayar, “Motion-Based Motion Deblurring,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 689–698, June 2004.
24. E. Adelson, “Depth-of-Focus Image Processing Method,” US Patent 4,661,986, 1987.
25. J.R. Bergen, “Method and Apparatus for Extended Depth of Field Imaging,” US Patent 6,201,899, 2001.
26. M. Ben-Ezra, A. Zomet, and S.K. Nayar, “Video Super-Resolution Using Controlled Subpixel Detector Shifts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 6, pp. 977–987, June 2005.
27. M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image Inpainting,” Proc. SIGGRAPH Conf., pp. 417–424, 2000.
28. A. Criminisi, P. Perez, and K. Toyama, “Object Removal by Exemplar-Based Inpainting,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 721–728, 2003.
29. H.-Y. Shum and R. Szeliski, “Construction of Panoramic Mosaics with Global and Local Alignment,” Int'l J. Computer Vision, vol. 36, no. 2, pp. 101–130, 2000.
30. M. Irani and S. Peleg, “Improving Resolution by Image Registration,” CVGIP: Graphical Models and Image Processing, pp. 231–239, 1991.
31. P. Anandan, “A Computational Framework and an Algorithm for the Measurement of Visual Motion,” Int'l J. Computer Vision, vol. 2, no. 3, pp. 283–310, 1989.
32. B. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” Proc. Int'l joint Conf. Artificial Intelligence, pp. 674–679, 1981.
33. J. Bouguet, “Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the Algorithm,” OpenCV Document, Intel, Microprocessor Research Labs, 2000.
34. E. Krotkov, “Focusing,” Int'l J. Computer Vision, vol. 1, no. 3, pp. 223–237, 1987.
35. N. Zhang, M. Postek, R. Larrabee, A. Vladar, W. Keery, and S. Jones, “Image Sharpness Measurement in the Scanning Electron Microscope,” The J. Scanning Microscopies, vol. 21, pp. 246–252, 1999.
36. C. Tomasi and R. Manduchi, “Bilateral Filtering for Gray and Color Images,” Proc. Int'l Conf. Computer Vision, pp. 836-846,1998.
37. J. Foley, A. van Dam, S. Feiner, and J. Hughes, Computer Graphics: Principles and Practice. Addison-Wesley, 1996.
38. J. Sethian, Level Set Methods: Evolving Interfaces in Geometry, Fluid Mechanics, Computer Vision and Materials Sciences. Cambridge Univ. Press, 1996.
39. A. Telea, “An Image Inpainting Technique Based on the Fast Marching Method,” J. Graphics Tools, vol. 9, no. 1, pp. 23–34, 2004.
40. A. Efros and T. Leung, “Texture Synthesis by Nonparametric Sampling,” Proc. Int'l Conf. Computer Vision, pp. 1033–1038, 1999.
41. R. Strzodka and C. Garbe, “Real-Time Motion Estimation and Visualization on Graphics Cards,” Proc. IEEE Visualization Conf., pp. 545–552, 2004.
42. C. Rother, V. Kolmogorov, and A. Blake, “'Grabcut’: Interactive Foreground Extraction Using Iterated Graph Cuts,” ACM Trans. Graphics, vol. 23, no. 3, pp. 309–314, 2004.
43. Y. Boykov and M. Jolly, “Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images,” Proc. Int'l Conf. Computer Vision, pp. 105–112, 2001.
44. V. Kwatra, A. Schodl, I. Essa, G. Turk, and A.F. Bobick, “Graphcut Textures: Image and Video Synthesis Using Graph Cuts,” ACM Trans. Graphics, vol. 22, no. 3, pp. 277–286, 2003.