Learning Temporal Dynamics for Video Super-Resolution: A Deep Learning Approach

IEEE Transactions on Image Processing

Volumn 27, Issue 7, 2018, Pages 3432-3445

  Liu, DIng ^a   Wang, Zhaowen ^b   Fan, Yuchen ^a   Liu, Xianming ^c   Wang, Zhangyang ^d   Chang, Shiyu ^e   Wang, Xinchao ^a   Huang, Thomas S ^a  

^a University of Illinois at Urbana Champaign   (United States)

^b ADOBE RESEARCH   (United States)

^c FACEBOOK   (United States)

^d TEXAS A AND M UNIVERSITY   (United States)

^e IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

deep learning; deep neural networks; Super resolution

Indexed keywords

ADAPTIVE OPTICS; ADAPTIVE SYSTEMS; ALIGNMENT; DEEP LEARNING; DEEP NEURAL NETWORKS; DYNAMICS; IMAGE RESOLUTION; LEARNING SYSTEMS; MOTION COMPENSATION; NEURAL NETWORKS; OPTICAL RESOLVING POWER;

ADAPTATION MODELS; COMPREHENSIVE EVALUATION; OPTICAL IMAGING; SPATIAL CONSISTENCY; SUPER RESOLUTION; TEMPORAL COHERENCE; TEMPORAL RELATION; VIDEO SUPER-RESOLUTION;

COMPLEX NETWORKS;

EID: 85044722702     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2018.2820807     Document Type: Article

References (40)

1. C. Szegedy et al., "Going deeper with convolutions," in Proc. CVPR, 2015, pp. 1-9.
2. C. Dong, C. C. Loy, K. He, and X. Tang, "Learning a deep convolutional network for image super-resolution," in Proc. ECCV, 2014, pp. 184-199.
3. Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, "Deep networks for image super-resolution with sparse prior," in Proc. ICCV, 2015, pp. 370-378.
4. C. Dong, C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks," IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 295-307, Feb. 2015.
5. J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks," in Proc. CVPR, 2016, pp. 1646-1654.
6. J. Kim, J. K. Lee, and K. M. Lee, "Deeply-recursive convolutional network for image super-resolution," in Proc. CVPR, 2016, pp. 1637-1645.
7. W. Shi et al., "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network," in Proc. CVPR, 2016, pp. 1874-1883.
8. C. Dong, C. C. Loy, and X. Tang, "Accelerating the super-resolution convolutional neural network," in Proc. ECCV, 2016, pp. 391-407.
9. W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, "Deep Laplacian pyramid networks for fast and accurate super-resolution," in Proc. CVPR, 2017, pp. 5835-5843.
10. Y. Tai, J. Yang, and X. Liu, "Image super-resolution via deep recursive residual network," in Proc. CVPR, 2017, pp. 2790-2798.
11. M. Dorr, T. Martinetz, K. R. Gegenfurtner, and E. Barth, "Variability of eye movements when viewing dynamic natural scenes," J. Vis., vol. 10, no. 10, p. 28, 2010.
12. S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, "Fast and robust multiframe super resolution," IEEE Trans. Image Process., vol. 13, no. 10, pp. 1327-1344, Oct. 2004.
13. H. Takeda, P. Milanfar, M. Protter, and M. Elad, "Super-resolution without explicit subpixel motion estimation," IEEE Trans. Image Process., vol. 18, no. 9, pp. 1958-1975, Sep. 2009.
14. S. P. Belekos, N. P. Galatsanos, and A. K. Katsaggelos, "Maximum a posteriori video super-resolution using a new multichannel image prior," IEEE Trans. Image Process., vol. 19, no. 6, pp. 1451-1464, Jun. 2010.
15. C. Liu and D. Sun, "On Bayesian adaptive video super resolution," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 2, pp. 346-360, Feb. 2014.
16. Z. Ma, R. Liao, X. Tao, L. Xu, J. Jia, and E. Wu, "Handling motion blur in multi-frame super-resolution," in Proc. CVPR, 2015, pp. 5224-5232.
17. R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia, "Video super-resolution via deep draft-ensemble learning," in Proc. CVPR, 2015, pp. 531-539.
18. Q. Dai, S. Yoo, A. Kappeler, and A. K. Katsaggelos, "Sparse representation-based multiple frame video super-resolution," IEEE Trans. Image Process., vol. 26, no. 2, pp. 765-781, Feb. 2017.
19. Y. Huang, W. Wang, and L. Wang, "Bidirectional recurrent convolutional networks for multi-frame super-resolution," in Proc. NIPS, 2015, pp. 235-243.
20. A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos, "Video superresolution with convolutional neural networks," IEEE Trans. Comput. Imag., vol. 2, no. 2, pp. 109-122, Jun. 2016.
21. M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, "Spatial transformer networks," in Proc. NIPS, 2015, pp. 2017-2025.
22. D. Liu et al., "Robust video super-resolution with learned temporal dynamics," in Proc. ICCV, 2017, pp. 2526-2534.
23. K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack, "Study of subjective and objective quality assessment of video," IEEE Trans. Image Process., vol. 19, no. 6, pp. 1427-1441, Jun. 2010.
24. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Trans. Image Process., vol. 13, no. 4, pp. 600-612, Apr. 2004.
25. D. Liu, Z. Wang, B. Wen, J. Yang, W. Han, and T. S. Huang, "Robust single image super-resolution via deep networks with sparse prior," IEEE Trans. Image Process., vol. 25, no. 7, pp. 3194-3207, Jul. 2016.
26. D. Liu, Z. Wang, N. Nasrabadi, and T. Huang, "Learning a mixture of deep networks for single image super-resolution," in Proc. ACCV, 2016, pp. 145-156.
27. C. Ledig et al., "Photo-realistic single image super-resolution using a generative adversarial network," in Proc. CVPR, 2017, pp. 105-114.
28. W. Han, S. Chang, D. Liu, M. Yu, W. Michael, and T. S. Huang, "Image super-resolution via dual-state recurrent networks," in Proc. CVPR, 2018.
29. W. Yang, J. Feng, G. Xie, J. Liu, Z. Guo, and S. Yan, "Video super-resolution based on spatial-temporal recurrent residual networks," Comput. Vis. Image Understand., vol. 168, pp. 79-92, Mar. 2017.
30. J. Caballero et al., "Real-time video super-resolution with spatiotemporal networks and motion compensation," in Proc. CVPR, Jul. 2017, pp. 2848-2857.
31. X. Tao, H. Gao, R. Liao, J. Wang, and J. Jia, "Detail-revealing deep video super-resolution," in Proc. ICCV, Oct. 2017, pp. 4482-4490.
32. V. Nair and G. E. Hinton, "Rectified linear units improve restricted boltzmann machines," in Proc. ICML, 2010, pp. 807-814.
33. J. Y. Lin, R. Song, T. Liu, H. Wang, and C.-C. J. Kuo, "MCL-V: A streaming video quality assessment database," J. Vis. Commun. Image Represent., vol. 30, pp. 1-9, Jul. 2015.
34. C. Keimel, J. Habigt, T. Habigt, M. Rothbucher, and K. Diepold, "Visual quality of current coding technologies at high definition IPTV bitrates," in Proc. IEEE Int. Workshop Multimedia Signal Process. (MMSP), Oct. 2010, pp. 390-393.
35. C. Liu, "Beyond pixels: Exploring new representations and applications for motion analysis," Ph.D. dissertation, Massachusetts Inst. Technol., Cambridge, MA, USA, 2009.
36. K. Seshadrinathan and A. C. Bovik, "Motion tuned spatio-temporal quality assessment of natural videos," IEEE Trans. Image Process., vol. 19, no. 2, pp. 335-350, Feb. 2010.
37. D. Dai, Y. Wang, Y. Chen, and L. Van Gool, "Is image super-resolution helpful for other vision tasks?" in Proc. WACV, 2016, pp. 1-9.
38. Z. Wang, S. Chang, Y. Yang, D. Liu, and T. S. Huang, "Studying very low resolution recognition using deep networks," in Proc. CVPR, 2016, pp. 4792-4800.
39. L. Wolf, T. Hassner, and I. Maoz, "Face recognition in unconstrained videos with matched background similarity," in Proc. CVPR, 2011, pp. 529-534.
40. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Proc. NIPS, 2012, pp. 1097-1105.