Video super-resolution with convolutional neural networks

IEEE Transactions on Computational Imaging

Volumn 2, Issue 2, 2016, Pages 109-122

  Kappeler, Armin ^a   Yoo, Seunghwan ^a   Dai, Qiqin ^a   Katsaggelos, Aggelos K ^a  

^a Northwestern University   (United States)

Author keywords

Convolutional Neural Networks; Deep Learning; Deep Neural Networks; Video Super Resolution

Indexed keywords

COMPUTER VISION; CONVOLUTION; DATABASE SYSTEMS; IMAGE ENHANCEMENT; IMAGE RECONSTRUCTION; OPTICAL RESOLVING POWER; RESTORATION; VIDEO RECORDING;

'CURRENT; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; IMAGE SUPER RESOLUTIONS; NEURAL NETWORK ARCHITECTURE; SPATIAL RESOLUTION; TEMPORAL DIMENSIONS; VIDEO DATABASE; VIDEO FRAME; VIDEO SUPER-RESOLUTION;

DEEP NEURAL NETWORKS;

EID: 85140805648     PISSN: 25730436     EISSN: 23339403     Source Type: Journal    
DOI: 10.1109/TCI.2016.2532323     Document Type: Article

References (43)

1. S. D. Babacan, R. Molina, and A. K. Katsaggelos, "Variational Bayesian super resolution, " IEEE Trans. Image Process., vol. 20, no. 4, pp. 984-999, Apr. 2011
2. S. P. Belekos, N. P. Galatsanos, and A. K. Katsaggelos, "Maximum a posteriori video super-resolution using a new multichannel image prior, " IEEE Trans. Image Process., vol. 19, no. 6, pp. 1451-1464, Jun. 2010
3. C. Liu and D. Sun, "On Bayesian adaptive video super resolution, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 2, pp. 346-360, Feb. 2014
4. A. K. Katsaggelos, R. Molina, and J.Mateos, "Super resolution of images and video, " Synth. Lect. Imag. Video Multimedia Process., vol. 1, no. 1, pp. 1-134, 2007
5. Z. Ma, J. Jia, and E. Wu, "Handling motion blur in multi-frame superresolution, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, vol. 1
6. J. Yang, J. Wright, T. Huang, and Y. Ma, "Image super-resolution via sparse representation, " IEEE Trans. Image Process., vol. 19, no. 11, pp. 2861-2873, Nov. 2010
7. J. Yang, Z. Wang, Z. Lin, X. Shu, and T. Huang, "Bilevel sparse coding for coupled feature spaces, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2360-2367
8. R. Zeyde, M. Elad, and M. Protter, "On single image scale-up using sparse-representations, " in Curves and Surfaces. New York, NY, USA: Springer, 2012, pp. 711-730
9. S. Wang, L. Zhang, Y. Liang, and Q. Pan, "Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2216-2223
10. R. Timofte, V. De Smet, and L. Van Gool, "A+: Adjusted anchored neighborhood regression for fast super-resolution, " in Proc. IEEE Asian Conf. Comput. Vis., 2014, pp. 1920-1927
11. S. Schulter, C. Leistner, and H. Bischof, "Fast and accurate image upscaling with super-resolution forests, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3791-3799
12. J.-B. Huang, A. Singh, and N. Ahuja, "Single image super-resolution from transformed self-exemplars, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 5197-5206
13. D. Glasner, S. Bagon, and M. Irani, "Super-resolution from a single image, " in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2009, pp. 349-356
14. B. C. Song, S. C. Jeong, and Y. Choi, "Video super-resolution algorithm using bi-directional overlapped block motion compensation and on-thefly dictionary training, " IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 3, pp. 274-285, Mar. 2011
15. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks, " in Proc. Neural Inf. Process. Syst. (NIPS), 2012, pp. 1097-1105
16. C. Szegedy et al., "Going deeper with convolutions, " in Proc. Int. Conf. Comput. Vis. Pattern Recog., 2014, pp. 1-9
17. C. Dong, C. C. Loy, K. He, and X. Tang, "Learning a deep convolutional network for image super-resolution, " in Proc. IEEE Eur. Conf. Comput. Vis., 2014, pp. 184-199
18. Z. Cui, H. Chang, S. Shan, B. Zhong, and X. Chen, "Deep network cascade for image super-resolution, " in Proc. IEEE Eur. Conf. Comput. Vis., 2014, pp. 1-16
19. Z. Wang et al., "Self-tuned deep super resolution, " in Proc. Comput. Vis. Pattern Recog. Workshops, 2015, pp. 1-8
20. M. Cheng, N. Lin, K. Hwang, and J. Jeng, "Fast video super-resolution using artificial neural networks, " in Proc. 8th Int. Symp. Commun. Syst. Netw. Digital Signal Process. (CSNDSP), 2012, pp. 1-4
21. R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia, "Video super-resolution via deep draft-ensemble learning, " in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2015, pp 531-539
22. J. Deng, W. Dong, R. Socher, and L. Li, "A large-scale hierarchical image database, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 248-255
23. A. Karpathy and G. Toderici, "Large-scale video classification with convolutional neural networks, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 1725-1732
24. S. Ji, W. Xu, M. Yang, and K. Yu, "3D convolutional neural networks for human action recognition, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 221-231, Jan. 2012
25. M. Aharon, M. Elad, and A. Bruckstein, "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation, " IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311-4322, Nov. 2006
26. L. I. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation based noise removal algorithms, " Phys. D, Nonlinear Phenom., vol. 60, nos. 1-4, pp. 259-268, 1992
27. H. Takeda, P. Milanfar, M. Protter, and M. Elad, "Super-resolution without explicit subpixel motion estimation, " IEEE Trans. Image Process., vol. 18, no. 9, pp. 1958-1975, Sep. 2009
28. S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, and R. Szeliski, "A database and evaluation methodology for optical flow, " Int. J. Comput. Vis., vol. 92, no. 1, pp. 1-31, 2011
29. D. Sun, S. Roth, and M. J. Black, "Secrets of optical flow estimation and their principles, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2010, pp. 2432-2439
30. M. Drulea and S. Nedevschi, "Total variation regularization of localglobal optical flow, " in Proc. IEEE Conf. Intell. Transp. Syst. (ITSC), 2011, pp. 318-323
31. J. Xie, L. Xu, and E. Chen, "Image denoising and inpainting with deep neural networks, " in Proc. Neural Inf. Process. Syst. (NIPS), 2012, pp. 1-9
32. H. C. Burger, C. J. Schuler, and S. Harmeling, "Image denoising: Can plain neural networks compete with BM3D?" in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2012, pp. 2392-2399
33. V. Jain and H. Seung, "Natural image denoising with convolutional networks." in Proc. Neural Inf. Process. Syst. (NIPS), 2008, pp. 1-8
34. C. J. Schuler, H. C. Burger, S. Harmeling, and B. Scholkopf, "A machine learning approach for non-blind image deconvolution, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 1067-1074
35. D. Eigen, D. Krishnan, and R. Fergus, "Restoring an image taken through a window covered with dirt or rain, " in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2013, pp. 633-640
36. J. Masci, U. Meier, D. Cirean, and J. Schmidhuber, "Stacked convolutional auto-encoders for hierarchical feature extraction, " in Artificial Neural Networks and Machine Learning-ICANN. New York, NY, USA: Springer, vol. 1, pp. 52-59, 2011
37. A. Maas, A. Hannun, and A. Ng, "Rectifier nonlinearities improve neural network acoustic models, " in Proc. ICML, 2013, vol. 28, p. 1
38. O. Russakovsky et al., "ImageNet large scale visual recognition challenge, " Int. J. Comput. Vis., vol. 115, no. 3, pp. 211-252, 2015
39. M. K. Park, M. G. Kang, and A. K. Katsaggelos, "Regularized highresolution image reconstruction considering inaccurate motion information, " Opt. Eng., vol. 46, no. 11, p. 117004, 2007
40. Myanmar 60p, Harmonic Inc. (2014). [Online]. Available: http://www.harmonicinc.com/resources/videos/4k-video-clip-center
41. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, "Caffe: Convolutional architecture for fast feature embedding, " arXiv preprint arXiv: 1408.5093, 2014
42. Infognition. (2010). Video Enhancer [Online]. Available: http://www. infognition.com/videoenhancer
43. M. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks, " in Proc. Comput. Vis. (ECCV14), 2014, vol. 8689, pp. 818-833