Real-time video super-resolution with spatio-temporal networks and motion compensation

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

Volumn 2017-January, Issue , 2017, Pages 2848-2857

  Caballero, Jose ^a   Ledig, Christian ^a   Aitken, Andrew ^a   Acosta, Alejandro ^a   Totz, Johannes ^a   Wang, Zehan ^a   Shi, Wenzhe ^a  

^a TWITTER INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; CONVOLUTION; NEURAL NETWORKS; OPTICAL RESOLVING POWER;

COMPUTATIONAL COSTS; CONVOLUTIONAL NEURAL NETWORK; IMAGE SUPER RESOLUTIONS; ORDERS OF MAGNITUDE; RECONSTRUCTION ACCURACY; TEMPORAL CORRELATIONS; TEMPORAL REDUNDANCY; VIDEO SUPER-RESOLUTION;

MOTION COMPENSATION;

EID: 85030236462     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2017.304     Document Type: Conference Paper

References (40)

1. A. Ahmadi and I. Patras. Unsupervised convolutional neural networks for motion estimation. IEEE International Conference on Image Processing (ICIP), pages 1629-1633, 2016.
2. T. Brox, N. Papenberg, and J. Weickert. High accuracy optical flow estimation based on a theory for warping. European Conference on Computer Vision (ECCV), 4:25-36, 2004.
3. J. Bruna, P. Sprechmann, and Y. LeCun. Super-resolution with deep convolutional sufficient statistics. In International Conference On Learning Representations (ICLR), 2016.
4. Q. Dai, S. Yoo, A. Kappeler, and A. K. Katsaggelos. Dictionary-based multiple frame video super-resolution. IEEE International Conference on Image Processing (ICIP), pages 83-87, 2015.
5. H. Demirel and G. Anbarjafari. Discrete wavelet transformbased satellite image resolution enhancement. IEEE Transactions on Geoscience and Remote Sensing, 49(6):1997- 2004, 2011.
6. C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 38(2):295-307, 2015.
7. C. Dong, C. C. Loy, and X. Tang. Accelerating the super-resolution convolutional neural network. In European Conference on Computer Vision (ECCV), pages 391-407. Springer International Publishing, 2016.
8. A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks. arXiv preprint arXiv:1602.02644, 2016.
9. A. Dosovitskiy, P. Fischery, E. Ilg, C. Hazirbas, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox. FlowNet: Learning optical flow with convolutional networks. International Conference on Computer Vision (ICCV), pages 2758-2766, 2015.
10. G. Farneback. Two-frame motion estimation based on polynomial expansion. Scandinavian Conference on Image Analysis, pages 363-370, 2003.
11. Y. Ganin, D. Kononenko, D. Sungatullina, and V. Lempitsky. DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation. European Conference on Computer Vision (ECCV), pages 311-326, 2016.
12. D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image. In IEEE 12th International Conference on Computer Vision (ICCV), pages 349-356, 2009.
13. B. K. Gunturk, A. U. Batur, Y. Altunbasak, M. H. Hayes, and R. M. Mersereau. Eigenface-domain super-resolution for face recognition. IEEE Transactions on Image Processing, 12(5):597-606, 2003.
14. A. Handa, M. Bloesch, V. Patraucean, S. Stent, J. McCormac, and A. Davison. gvnn: Neural Network Library for Geometric Computer Vision. European Conference on Computer Vision (ECCV) Workshop on Deep Geometry, 2016.
15. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
16. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. European Conference on Computer Vision, pages 630-645, 2016.
17. Y. Huang, W. Wang, and L. Wang. Bidirectional recurrent convolutional networks for multi-frame super-resolution. In Advances in Neural Information Processing Systems (NIPS), pages 235-243, 2015.
18. ITS. Consumer Digital Video Library, accessed on 08/2016 at http://www.cdvl.org/.
19. M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial transformer networks. Advances in Neural Information Processing Systems (NIPS), pages 2017-2025, 2015.
20. J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. European Conference on Computer Vision (ECCV), pages 694-711, 2016.
21. N. Kalantari, T.Wang, and R. Ramamoorthi. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics (TOG), 35(6):193, 2016.
22. A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos. Video super-resolution with convolutional neural networks. IEEE Transactions on Computational Imaging, 2(2):109- 122, 2016.
23. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1725-1732, 2014.
24. J. Kim, J. K. Lee, and K. M. Lee. Deeply-recursive convolutional network for image super-resolution. IEEE Computer Vision and Pattern Recognition (CVPR), 2016.
25. D. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference On Learning Representations (ICLR), 2015.
26. C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint arXiv:1609.04802, 2016.
27. C. Liu, N. England, and D. Sun. A bayesian approach to adaptive video super resolution. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 209- 216, 2015.
28. S. C. Park, M. K. Park, and M. G. Kang. Super-resolution image reconstruction: A technical overview. IEEE Signal Processing Magazine, 20(3):21-36, 2003.
29. V. Patraucean, A. Handa, and R. Cipolla. Spatio-temporal video autoencoder with differentiable memory. International Conference On Learning Representations (ICLR) Workshop, 2016.
30. A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. International Conference On Learning Representations (ICLR), 2014.
31. K. Seshadrinathan, S. Member, and A. C. Bovik. Motion tuned spatio-temporal quality assessment of natural videos motion tuned spatio-temporal quality assessment of natural videos. IEEE Transactions on Image Processing, 19(2):335- 350, 2010.
32. O. Shahar, A. Faktor, and M. Irani. Space-time superresolution from a single video. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3353- 3360, 2011.
33. W. Shi, J. Caballero, H. Ferenc, T. Johannes, A. P. Aitken, R. Bishop, R. Daniel, and Z. Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1874- 1883, 2016.
34. W. Shi, J. Caballero, C. Ledig, X. Zhuang, W. Bai, K. Bhatia, A. M. S. M. De Marvao, T. Dawes, D. O'Regan, and D. Rueckert. Cardiac image super-resolution with global correspondence using multi-atlas PatchMatch. In International Conference on Medical Image Computing and Computer- Assisted Intervention (MICCAI), pages 9-16, 2013.
35. H. Takeda, S. Farsiu, and P. Milanfar. Kernel regression for image processing and reconstruction. IEEE Transactions on Image Processing, 16(2):349-366, 2007.
36. H. Takeda, P. Milanfar, M. Protter, and M. Elad. Superresolution without explicit subpixel motion estimation. IEEE Transactions on Image Processing, 18(9):1958-1975, 2009.
37. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3D convolutional networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4489-4497, 2015.
38. J. Yang, S. Member, and Z. Wang. Coupled dictionary training for image super-resolution. IEEE Transactions on Image Processing, 21(8):3467-3478, 2012.
39. J. Yang, J. Wright, T. S. Huang, and Y. Ma. Image superresolution via sparse representation. IEEE Transactions on Image Processing, 19(11):2861-2873, 2010.
40. W. Zhou, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: form error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004.