Unsupervised learning for physical interaction through video prediction

Advances in Neural Information Processing Systems

Volumn , Issue , 2016, Pages 64-72

  Finn, Chelsea ^a   Goodfellow, Ian ^b   Levine, Sergey ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b OpenAI LLC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

PIXELS; STATISTICAL TESTS;

ACCURATE PREDICTION; INTERACTIVE AGENTS; OBJECT APPEARANCE; OBJECT INFORMATION; PHYSICAL INTERACTIONS; PHYSICAL OBJECTS; ROBOT INTERACTIONS; VIDEO PREDICTION;

FORECASTING;

EID: 85018936904     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, et al. Tensorflow: Large-scale machine learning on heterogeneous systems, 2015. Software from tensorflow.org, 2015.
2. P. W. Battaglia, J. B. Hamrick, and J. B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45), 2013.
3. S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pages 1171-1179, 2015.
4. B. Boots, A. Byravan, and D. Fox. Learning predictive models of a depth camera & manipulator from raw execution traces. In International Conference on Robotics and Automation (ICRA), 2014.
5. M. A. Brubaker, L. Sigal, and D. J. Fleet. Estimating contact dynamics. In International Conference on Computer Vision (ICCV), 2009.
6. B. De Brabandere, X. Jia, T. Tuytelaars, and L. Van Gool. Dynamic filter networks. In Neural Information Processing Systems (NIPS). 2016.
7. S. Eslami, N. Heess, T. Weber, Y. Tassa, K. Kavukcuoglu, and G. E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. Neural Information Processing Systems (NIPS), 2016.
8. A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The KITTI dataset. International Journal of Robotics Research (IJRR), 2013.
9. D.-A. Huang and K. M. Kitani. Action-reaction: Forecasting the dynamics of human interaction. In European Conference on Computer Vision (ECCV). Springer, 2014.
10. C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. PAMI, 36(7), 2014.
11. M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In Neural Information Processing Systems (NIPS), 2015.
12. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In Compuer Vision and Pattern Recognition (CVPR), 2014.
13. D. Kinga and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
14. S. Lange, M. Riedmiller, and A. Voigtlander. Autonomous reinforcement learning on raw visual input data in a real world application. In International Joint Conference on Neural Networks (IJCNN), 2012.
15. A. Lerer, S. Gross, and R. Fergus. Learning physical intuition of block towers by example. International Conference on Machine Learning (ICML), 2016.
16. W. Lotter, G. Kreiman, and D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.08104, 2016.
17. M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. International Conference on Learning Representations (ICLR), 2016.
18. R. Mottaghi, H. Bagherinezhad, M. Rastegari, and A. Farhadi. Newtonian image understanding: Unfolding the dynamics of objects in static images. Computer Vision and Pattern Recognition (CVPR), 2015.
19. R. Mottaghi, M. Rastegari, A. Gupta, and A. Farhadi. "What happens if⋯" learning to predict the effect of forces in images. European Conference on Computer Vision (ECCV), 2016.
20. J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh. Action-conditional video prediction using deep networks in atari games. In Neural Information Processing Systems (NIPS), 2015.
21. M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv:1412.6604, 2014.
22. N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using lstms. International Conference on Machine Learning (ICML), 2015.
23. C. Vondrick, H. Pirsiavash, and A. Torralba. Anticipating the future by watching unlabeled video. CoRR, abs/1504.08023, 2015.
24. C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In Neural Information Processing Systems (NIPS). 2016.
25. J. Walker, C. Doersch, A. Gupta, and M. Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference on Computer Vision (ECCV), 2016.
26. J. Walker, A. Gupta, and M. Hebert. Patch to the future: Unsupervised visual prediction. In Computer Vision and Pattern Recognition (CVPR), 2014.
27. M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Neural Information Processing Systems (NIPS), 2015.
28. S. Xingjian, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Neural Information Processing Systems, 2015.
29. T. Xue, J. Wu, K. L. Bouman, and W. T. Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Neural Information Processing Systems (NIPS). 2016.
30. J. Yuen and A. Torralba. A data-driven approach for event prediction. In European Conference on Computer Vision (ECCV), 2010.