Video super-resolution based on spatial-temporal recurrent residual networks

Computer Vision and Image Understanding

Volumn 168, Issue , 2018, Pages 79-92

  Yang, Wenhan ^a   Feng, Jiashi ^b   Xie, Guosen ^c   Liu, Jiaying ^a   Guo, Zongming ^a   Yan, Shuicheng ^d  

^a PEKING UNIVERSITY   (China)

^b NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^c INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

^d Qihoo 360 Technology Co Ltd   (China)

Author keywords

Inter frame motion context; Intra frame redundancy; Spatial residue; Temporal residue; Video super resolution

Indexed keywords

CONVOLUTIONAL NEURAL NETWORKS; OPTICAL RESOLVING POWER; REDUNDANCY; SIGNAL RECEIVERS;

INTER-FRAME; INTRA FRAMES; SPATIAL RESIDUE; TEMPORAL RESIDUE; VIDEO SUPER-RESOLUTION;

RECURRENT NEURAL NETWORKS;

EID: 85033234926     PISSN: 10773142     EISSN: 1090235X     Source Type: Journal    
DOI: 10.1016/j.cviu.2017.09.002     Document Type: Article

References (63)

1. Anon, Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38:2 (2016), 295–307.
2. Agostinelli, F., Anderson, M.R., Lee, H., Adaptive multi-column deep neural networks with application to robust image denoising. Proc. Annual Conference on Neural Information Processing Systems, 2013, 1493–1501.
3. Aguena, M.L., Mascarenhas, N.D., Multispectral image data fusion using {POCS} and super-resolution. Comput. Vision Image Understand. 102:2 (2006), 178–187.
4. Baker, S., Kanade, T., Super-resolution optical flow. CMU-RI-TR-99-36, 1999.
5. Burger, H. C., Schuler, C. J., Harmeling, S., 2012a. Image denoising with multi-layer perceptrons, part 1: comparison with existing algorithms and with bounds. ArXiv preprint, arXiv:1211.1544.
6. Burger, H. C., Schuler, C. J., Harmeling, S., 2012b. Image denoising with multi-layer perceptrons, part 2: training trade-offs and analysis of their mechanisms. ArXiv preprint, arXiv:1211.1552.
7. Chang, H., Yeung, D.-Y., Xiong, Y., Super-resolution through neighbor embedding. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 1, 2004 I–I.
8. Cui, Z., Chang, H., Shan, S., Zhong, B., Chen, X., Deep network cascade for image super-resolution. Proc. IEEE European Conf. Computer Vision, 2014.
9. Dong, C., Loy, C.C., He, K., Tang, X., Image super-resolution using deep convolutional networks. Proc. IEEE European Conf. Computer Vision, 2014.
10. Dong, W., Zhang, L., Shi, G., Centralized sparse representation for image restoration. Proc. IEEE Int'l Conf. Computer Vision, 2013, 1259–1266.
11. Farsiu, S., Robinson, M.D., Elad, M., Milanfar, P., Fast and robust multiframe super resolution. IEEE Trans. Image Process. 13:10 (2004), 1327–1344.
12. Fransens, R., Strecha, C., Gool, L.V., Optical flow based super-resolution: a probabilistic approach. Comput. Vision Image Understand. 106:1 (2007), 106–115.
13. Freeman, W.T., Jones, T.R., Pasztor, E.C., Example-based super-resolution. IEEE Comput. Graph. Appl. 22 (2002), 56–65.
14. Gatys, L.A., Ecker, A.S., Bethge, M., Image style transfer using convolutional neural networks. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2016, 2414–2423.
15. He, H., Kondi, L.P., An image super-resolution algorithm for different error levels per frame. IEEE Trans. Image Process. 15:3 (2006), 592–603.
16. Huang, Y., Wang, W., Wang, L., Bidirectional recurrent convolutional networks for multi-frame super-resolution. Proc. Annual Conference on Neural Information Processing Systems, 2015, 235–243.
17. Huang, Y., Wang, W., Wang, L., Video super-resolution via bidirectional recurrent convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell., PP(99), 2017 1–1.
18. Huhle, B., Schairer, T., Jenke, P., Straer, W., Fusion of range and color images for denoising and resolution enhancement with a non-local filter. Comput. Vision Image Understand. 114:12 (2010), 1336–1345.
19. Irani, M., Peleg, S., Improving resolution by image registration. CVGIP: Graph. Models Image Process. 53:3 (1991), 231–239.
20. Ismaeil, K.A., Aouada, D., Mirbach, B., Ottersten, B., Enhancement of dynamic depth scenes by upsampling for precise super-resolution (up-sr). Comput. Vision Image Understand. 147 (2016), 38–49 Spontaneous Facial Behaviour Analysis.
21. Jain, V., Seung, S., Natural image denoising with convolutional networks. Koller, D., Schuurmans, D., Bengio, Y., Bottou, L., (eds.) Proc. Annual Conference on Neural Information Processing Systems, 2009, 769–776.
22. Jia, Y., Shelhamer, E., Donahue, J., et al. Caffe:convolutional architecture for fast feature embedding. ACM International Conference on Multimedia, 2014, 675–678.
23. Joshi, M.V., Chaudhuri, S., Simultaneous estimation of super-resolved depth map and intensity field using photometric cue. Comput. Vision Image Understand. 101:1 (2006), 31–44.
24. Kanaev, A.V., Miller, C.W., Multi-frame super-resolution algorithm for complex motion patterns. Opt. Express 21:17 (2013), 19850–19866.
25. Kim, J., Lee, J.K., Lee, K.M., Accurate image super-resolution using very deep convolutional networks. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2016, 1646–1654.
26. Kim, J., Lee, J.K., Lee, K.M., Deeply-recursive convolutional network for image super-resolution. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2016, 1637–1645, 10.1109/CVPR.2016.181.
27. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H., Deep laplacian pyramid networks for fast and accurate super-resolution. IEEE Conf. Comput. Vis. Pattern Recognit, 2017.
28. Liao, R., Tao, X., Li, R., Ma, Z., Jia, J., Video super-resolution via deep draft-ensemble learning. Proc. IEEE Int'l Conf. Computer Vision, 2015, 531–539.
29. Liao, R., Tao, X., Li, R., Ma, Z., Jia, J., Video super-resolution via deep draft-ensemble learning. Proc. IEEE Int'l Conf. Computer Vision, 2015, 531–539.
30. Liu, C., Sun, D., On bayesian adaptive video super resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36:2 (2014), 346–360.
31. Lu, X., Li, X., Multiresolution imaging. IEEE Trans. Cybern. 44:1 (2014), 149–160.
32. Ma, C., Yang, C.-Y., Yang, X., Yang, M.-H., Learning a no-reference quality metric for single-image super-resolution. Comput. Vision Image Understand., 2017.
33. Mairal, J., Bach, F., Ponce, J., Sapiro, G., Zisserman, A., Non-local sparse models for image restoration. Proc. IEEE Int'l Conf. Computer Vision, 2009, 2272–2279.
34. Marquina, A., Osher, S., Image super-resolution by TV-regularization and bregman iteration. J. Sci. Comput. 37:3 (2008), 367–382.
35. Nguyen, K., Fookes, C., Sridharan, S., Denman, S., Feature-domain super-resolution for iris recognition. Comput. Vision Image Understand. 117:10 (2013), 1526–1535.
36. Omer, O.A., Tanaka, T., Region-based weighted-norm approach to video super-resolution with adaptive regularization. Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, 2009, 833–836.
37. Osendorfer, C., Soyer, H., van der Smagt, P., Image super-resolution with fast approximate convolutional sparse coding. Neural Information Processing, 2014.
38. Protter, M., Elad, M., Takeda, H., Milanfar, P., Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Trans. Image Process. 18:1 (2009), 36–51.
39. Rudin, L.I., Osher, S., Fatemi, E., Nonlinear total variation based noise removal algorithms. Physica D 60 (1992), 259–268.
40. Salvador, J., Prez-Pellitero, E., Naive bayes super-resolution forest. Proc. IEEE Int'l Conf. Computer Vision, 2015, 325–333.
41. Schuler, C.J., Hirsch, M., Harmeling, S., Schölkopf, B., Learning to Deblur. IEEE Trans. Pattern Anal. Mach. Intell. 38:7 (2016), 1439–1451.
42. Singh, A., Ahuja, N., Learning ramp transformation for single image super-resolution. Comput. Vision Image Understand. 135 (2015), 109–125.
43. Sun, J., Sun, J., Xu, Z., Shum, H.Y., Gradient profile prior and its applications in image super-resolution and enhancement. IEEE Trans. Image Process. 20:6 (2011), 1529–1542.
44. Takeda, H., Milanfar, P., Protter, M., Elad, M., Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. 18:9 (2009), 1958–1975.
45. Takeda, H., Milanfar, P., Protter, M., Elad, M., Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. 18:9 (2009), 1958–1975.
46. Timofte, R., De, V., Van Gool, L., Anchored neighborhood regression for fast example-based super-resolution. Proc. IEEE Int'l Conf. Computer Vision, 2013.
47. Timofte, R., De Smet, V., Van Gool, L., A+: Adjusted anchored neighborhood regression for fast super-resolution. Proc. IEEE Asia Conf. Computer Vision, 2014.
48. Timofte, R., Smet, V.D., Gool, L.V., Semantic super-resolution: when and where is it useful?. Comput. Vision Image Understand. 142 (2016), 1–12.
49. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res, 2010.
50. Wang, Z., Liu, D., Yang, J., Han, W., Huang, T., Deep networks for image super-resolution with sparse prior. Proc. IEEE Int'l Conf. Computer Vision, 2015, 370–378.
51. Wang, Z., Liu, D., Yang, J., Han, W., Huang, T., Deep networks for image super-resolution with sparse prior. Proc. IEEE Int'l Conf. Computer Vision, 2015, 370–378, 10.1109/ICCV.2015.50.
52. Wu, W., Zheng, C., Single image super-resolution using self-similarity and generalized nonlocal mean. IEEE International Conference of IEEE Region, 2013, 1–4.
53. Xie, J., Xu, L., Chen, E., Image denoising and inpainting with deep neural networks. Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q., (eds.) Proc. Annual Conference on Neural Information Processing Systems, 2012, 341–349.
54. Xu, L., Ren, J.S., Liu, C., Jia, J., Deep convolutional neural network for image deconvolution. Proc. Annual Conference on Neural Information Processing Systems, 2014, 1790–1798.
55. Yan, Z., Zhang, H., Wang, B., Paris, S., Yu, Y., Automatic photo adjustment using deep neural networks. ACM Trans. Graph. 35:2 (2016), 11:1–11:15.
56. Yang, J.C., Wright, J., Huang, T.S., Ma, Y., Image super-resolution via sparse representation. IEEE Trans. Image Process. 19:11 (2010), 2861–2873.
57. Yang, W., Feng, J., Yang, J., et al. Deep edge guided recurrent residual learning for image super-resolution. IEEE Trans. Image Process. 26:12 (2017), 5895–5907.
58. Yin, M., Gao, J., Cai, S., Image super-resolution via 2d tensor regression learning. Comput. Vision Image Understand. 132 (2015), 12–23.
59. Yuan, Q., Zhang, L., Shen, H., Regional spatially adaptive total variation super-resolution with spatial information filtering and clustering. IEEE Trans. Image Process. 22:6 (2013), 2327–2342.
60. Zeng, K., Yu, J., Wang, R., Li, C., Tao, D., Coupled deep autoencoder for single image super-resolution. IEEE Trans.Cybern. PP:99 (2016), 1–11.
61. Zhang, H., Yang, J., Zhang, Y., Huang, T.S., Image and video restorations via nonlocal kernel regression. IEEE Trans. Cybern. 43:3 (2013), 1035–1046.
62. Zhang, X., Xiong, R., Ma, S., Li, G., Gao, W., Video super-resolution with registration-reliability regulation and adaptive total variation. J. Vis. Commun. Image Represent., 2015.
63. Zhao, W., Sawhney, H., Is super-resolution with optical flow feasible?. Proc. IEEE European Conf. Computer Vision, 2002, Berlin, Heidelberg.