Learning-based view synthesis for light field cameras

ACM Transactions on Graphics

Volumn 35, Issue 6, 2016, Pages

  Kalantari, Nima Khademi ^a   Wang, Ting Chun ^b   Ramamoorthi, Ravi ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Convolutional neural network; Disparity estimation; Light field; View synthesis

Indexed keywords

CAMERAS; CONVOLUTION; ECONOMIC AND SOCIAL EFFECTS; IMAGE RESOLUTION; LEARNING ALGORITHMS; LEARNING SYSTEMS; NEURAL NETWORKS;

ANGULAR RESOLUTION; CONVOLUTIONAL NEURAL NETWORK; DISPARITY ESTIMATIONS; HIGH QUALITY IMAGES; LEARNING-BASED APPROACH; LIGHT FIELDS; STATE-OF-THE-ART TECHNIQUES; VIEW SYNTHESIS;

IMAGE PROCESSING;

EID: 85046260289     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/2980179.2980251     Document Type: Article

References (51)

1. ADELSON, E. H., AND WANG, J. Y. A. 1992. Single lens stereo with a plenoptic camera. IEEE PAMI 14, 2, 99-106.
2. BISHOP, T. E., ZANETTI, S., AND FAVARO, P. 2009. Light field superresolution. In IEEE ICCP, 1-9.
3. BURGER, H. C., SCHULER, C. J., AND HARMELING, S. 2012. Image denoising: Can plain neural networks compete with BM3D? In IEEE CVPR, 2392-2399.
4. CHAURASIA, G., SORKINE, O., AND DRETTAKIS, G. 2011. Silhouette-aware warping for image-based rendering. In EGSR, 1223-1232.
5. CHAURASIA, G., DUCHENE, S., SORKINE-HORNUNG, O., AND DRETTAKIS, G. 2013. Depth synthesis and local warps for plausible image-based navigation. ACM TOG 32, 3, 30:1-30:12.
6. CHO, D., LEE, M., KIM, S., AND TAI, Y.-W. 2013. Modeling the calibration pipeline of the lytro camera for high quality light-field image reconstruction. In IEEE ICCV, 3280-3287.
7. DONG, C., LOY, C. C., HE, K., AND TANG, X. 2014. Learning a deep convolutional network for image super-resolution. In ECCV, 184-199.
8. DOSOVITSKIY, A., SPRINGENBERG, J. T., AND BROX, T. 2015. Learning to generate chairs with convolutional neural networks. In IEEE CVPR, 1538-1546.
9. EISEMANN, M., DE DECKER, B., MAGNOR, M., BEKAERT, P., DE AGUIAR, E., AHMED, N., THEOBALT, C., AND SELLENT, A. 2008. Floating textures. CGF 27, 2, 409-418.
10. FITZGIBBON, A., WEXLER, Y., AND ZISSERMAN, A. 2003. Image-based rendering using image-based priors. In IEEEICCV, 1176-1183 vol.2.
11. FLYNN, J., NEULANDER, I., PHILBIN, J., AND SNAVELY, N. 2016. Deepstereo: Learning to predict new views from the worlds imagery. In IEEE CVPR, 5515-5524.
12. FURUKAWA, Y., AND PONCE, J. 2010. Accurate, dense, and robust multiview stereopsis. IEEE PAMI 32, 8, 1362-1376.
13. GEORGIEV, T., ZHENG, K. C., CURLESS, B., SALESIN, D., NA-YAR, S., AND INTWALA, C. 2006. Spatio-angular resolution tradeoffs in integral photography. In EGSR, 263-272.
14. GIROD, B., CHANG, C.-L., RAMANATHAN, P., AND ZHU, X. 2003. Light field compression using disparity-compensated lifting. In IEEE ICME, vol. 1, I-373-6 vol.1.
15. GLOROT, X., AND BENGIO, Y. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, vol. 9, 249-256.
16. GOESELE, M., ACKERMANN, J., FUHRMANN, S., HAUBOLD, C., KLOWSKY, R., STEEDLY, D., AND SZELISKI, R. 2010. Ambient point clouds for view interpolation. ACM TOG 29, 4, 95.
17. HEBER, S., AND POCK, T. 2016. Convolutional networks for shape from light field. In IEEE CVPR.
18. JEON, H. G., PARK, J., CHOE, G., PARK, J., BOK, Y., TAI, Y. W., AND KWEON, I. S. 2015. Accurate depth map estimation from a lenslet light field camera. In IEEE CVPR, 1547-1555.
19. KHOLGADE, N., SIMON, T., EFROS, A., AND SHEIKH, Y. 2014. 3D object manipulation in a single photograph using stock 3D models. ACM TOG 33, 4, 127.
20. KINGMA, D., AND BA, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
21. LEVIN, A., AND DURAND, F. 2010. Linear view synthesis using a dimensionality gap light field prior. In IEEE CVPR, 1831-1838.
22. LEVOY, M., AND HANRAHAN, P. 1996. Light field rendering. In ACM SIGGRAPH, 31-42.
23. LYTRO, 2016. https://www.lytro.com/.
24. MAHAJAN, D., HUANG, F.-C., MATUSIK, W., RAMAMOORTHI, R., AND BELHUMEUR, P. 2009. Moving gradients: a path-based method for plausible image interpolation. ACM TOG 28, 3, 42.
25. MARWAH, K., WETZSTEIN, G., BANDO, Y., AND RASKAR, R. 2013. Compressive light field photography using overcomplete dictionaries and optimized projections. ACM TOG 32, 4, 46:1-46:12.
26. MITRA, K., AND VEERARAGHAVAN, A. 2012. Light field denoising, light field superresolution and stereo camera based re-focussing using a GMM light field patch prior. In IEEE CVPRW, 22-28.
27. NG, R., LEVOY, M., BRÉDIF, M., DUVAL, G., HOROWITZ, M., AND HANRAHAN, P. 2005. Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR 2, 11, 1-11.
28. PELICAN IMAGING, 2016. Capture life in 3D. http://www.pelicanimaging.com/.
29. RAJ, A., LOWNEY, M., SHAH, R., AND WETZSTEIN, G., 2016. Stanford lytro light field archive. http://lightfields.stanford.edu/.
30. RAYTRIX, 2016. 3D light field camera technology. https://www.raytrix.de/.
31. RUMELHART, D. E., HINTON, G. E., AND WILLIAMS, R. J. 1986. Learning representations by back-propagating errors. Nature 323, 533-536.
32. SCHEDL, D. C., BIRKLBAUER, C., AND BIMBER, O. 2015. Directional super-resolution by means of coded sampling and guided upsampling. In IEEE ICCP, 1-10.
33. SHECHTMAN, E., RAV-ACHA, A., IRANI, M., AND SEITZ, S. 2010. Regenerative morphing. In IEEE CVPR, 615-622.
34. SHI, L., HASSANIEH, H., DAVIS, A., KATABI, D., AND DURAND, F. 2014. Light field reconstruction using sparsity in the continuous fourier domain. ACM TOG 34, 1, 12:1-12:13.
35. SU, H., WANG, F., YI, L., AND GUIBAS, L. 2014. 3D-assisted image feature synthesis for novel views of an object. arXiv preprint arXiv:1412.0003.
36. SUN, J., CAO, W., XU, Z., AND PONCE, J. 2015. Learning a convolutional neural network for non-uniform motion blur removal. In IEEE CVPR, 769-777.
37. TAO, M. W., HADAP, S., MALIK, J., AND RAMAMOORTHI, R. 2013. Depth from combining defocus and correspondence using light-field cameras. In IEEE ICCV, 673-680.
38. TAO, M. W., SRINIVASAN, P. P., MALIK, J., RUSINKIEWICZ, S., AND RAMAMOORTHI, R. 2015. Depth from shading, defocus, and correspondence using light-field angular coherence. In IEEE CVPR, 1940-1948.
39. TATARCHENKO, M., DOSOVITSKIY, A., AND BROX, T. 2015. Single-view to multi-view: Reconstructing unseen views with a convolutional network. CoRR abs/1511.06702.
40. TONG, X., AND GRAY, R. M. 2003. Interactive rendering from compressed light fields. IEEE TCSVT 13, 11 (Nov), 1080-1091.
41. VEDALDI, A., AND LENC, K. 2015. MatConvNet: Convolutional neural networks for Matlab. In ACMMM, 689-692.
42. WANG, Z., BOVIK, A., SHEIKH, H., AND SIMONCELLI, E. 2004. Image quality assessment: from error visibility to structural similarity. IEEE TIP 13, 4 (April), 600-612.
43. WANG, T. C., EFROS, A. A., AND RAMAMOORTHI, R. 2015. Occlusion-aware depth estimation using light-field cameras. In IEEE ICCV, 3487-3495.
44. WANNER, S., AND GOLDLUECKE, B. 2012. Globally consistent depth labeling of 4D light fields. In IEEE CVPR, 41-48.
45. WANNER, S., AND GOLDLUECKE, B. 2014. Variational light field analysis for disparity estimation and super-resolution. IEEE PAMI 36, 3, 606-619.
46. WILBURN, B., JOSHI, N., VAISH, V., TALVALA, E.-V., AN-TUNEZ, E., BARTH, A., ADAMS, A., HOROWITZ, M., AND LEVOY, M. 2005. High performance imaging using large camera arrays. ACM TOG 24, 3, 765-776.
47. YANG, J., REED, S. E., YANG, M.-H., AND LEE, H. 2015. Weakly-supervised disentangling with recurrent transformations for 3D view synthesis. In NIPS, 1099-1107.
48. YOON, Y., JEON, H. G., YOO, D., LEE, J. Y., AND KWEON, I. S. 2015. Learning a deep convolutional network for light-field image super-resolution. In IEEE ICCV Workshop, 57-65.
49. ZHANG, Z., LIU, Y., AND DAI, Q. 2015. Light field from micro-baseline image pair. In IEEE CVPR, 3800-3809.
50. ZHANG, F. L., WANG, J., SHECHTMAN, E., ZHOU, Z. Y., SHI, J. X., AND HU, S. M. 2016. PlenoPatch: Patch-based plenoptic image manipulation. IEEE TVCG PP, 99, 1-1.
51. ZHOU, T., TULSIANI, S., SUN, W., MALIK, J., AND EFROS, A. A. 2016. View synthesis by appearance flow. CoRR abs/1605.03557.