MoCoGAN: Decomposing Motion and Content for Video Generation

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn , Issue , 2018, Pages 1526-1535

  Tulyakov, Sergey ^a   Liu, Ming Yu ^b   Yang, Xiaodong ^b   Kautz, Jan ^b  

^a Snap Research   (United States)

^b NVIDIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

RANDOM PROCESSES; STOCHASTIC SYSTEMS;

ADVERSARIAL LEARNING; ADVERSARIAL NETWORKS; QUANTITATIVE COMPARISON; RANDOM VECTORS; STATE-OF-THE-ART APPROACH; VIDEO FRAME; VIDEO GENERATION; VISUAL SIGNALS;

COMPUTER VISION;

EID: 85057313381     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2018.00165     Document Type: Conference Paper

References (45)

1. N. Aifanti, C. Papachristou, and A. Delopoulos. The mug facial expression database. In Image Analysis for Multimedia Interactive Services (WIAMIS), 2010 11th International Workshop on, pages 1-4, 2010.
2. B. Amos, B. Ludwiczuk, and M. Satyanarayanan. Openface: A general-purpose face recognition library with mobile applications. Technical report, CMU-CS-16-118, CMU School of Computer Science, 2016.
3. M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.
4. Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Realtime multiperson 2d pose estimation using part affinity fields. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
5. X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2016.
6. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In Advances in Neural Information Processing Systems (NIPS) Workshop, 2014.
7. E. Denton and V. Birodkar. Unsupervised learning of disentangled representations from video. Advances in Neural Information Processing Systems (NIPS), 2017.
8. E. L. Denton, S. Chintala, R. Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. In Advances in Neural Information Processing Systems (NIPS), 2015.
9. G. Doretto, A. Chiuso, Y. N. Wu, and S. Soatto. Dynamic textures. International Journal of Computer Vision (IJCV), 2003.
10. C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. In Advances in Neural Information Processing Systems (NIPS), 2016.
11. I. Goodfellow. Nips 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160, 2016.
12. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D.Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014.
13. L. Gorelick, M. Blank, E. Shechtman, M. Irani, and R. Basri. Actions as space-time shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 29(12):2247-2253, 2007.
14. K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning (ICML), 2015.
15. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 1997.
16. D. J. Im, C. D. Kim, H. Jiang, and R. Memisevic. Generating images with recurrent adversarial networks. arXiv preprint arXiv:1602.05110, 2016.
17. N. Kalchbrenner, A. v. d. Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and K. Kavukcuoglu. Video pixel networks. arXiv preprint arXiv:1610.00527, 2016.
18. T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
19. D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
20. D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
21. M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-toimage translation networks. In Advances in Neural Information Processing Systems (NIPS), 2017.
22. M.-Y. Liu and O. Tuzel. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS), 2016.
23. X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley. Least squares generative adversarial networks. IEEE International Conference on Computer Vision (ICCV), 2017.
24. M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. In International Conference on Learning Representations (ICLR), 2016.
25. A. Odena, C. Olah, and J. Shlens. Conditional image synthesis with auxiliary classifier gans. In International Conference on Machine Learning (ICML), 2017.
26. J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh. Actionconditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems (NIPS), 2015.
27. Y. Pan, Z. Qiu, T. Yao, H. Li, and T. Mei. To create what you tell: Generating videos from captions. In ACM Conference on Multimedia, 2017.
28. A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations (ICLR), 2016.
29. D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and variational inference in deep latent Gaussian models. In International Conference on Machine Learning (ICML), 2014.
30. K. Roth, A. Lucchi, S. Nowozin, and T. Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in Neural Information Processing Systems (NIPS), 2017.
31. M. Saito, E. Matsumoto, and S. Saito. Temporal generative adversarial nets with singular value clipping. In IEEE International Conference on Computer Vision (ICCV), 2017.
32. T. Salimans, I. Goodfellow,W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NIPS), 2016.
33. K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
34. N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using lstms. In International Conference on Machine Learning (ICML), 2015.
35. M. Szummer and R. W. Picard. Temporal texture modeling. In International Conference on Image Processing (ICIP), 1996.
36. L. Theis, A. v. d. Oord, and M. Bethge. A note on the evaluation of generative models. In International Conference on Learning Representations (ICLR), 2016.
37. S. Tulyakov, A. Fitzgibbon, and S. Nowozin. Hybrid vae: Improving deep generative models using partial observations. Advances in Neural Information Processing Systems (NIPS) Workshop, 2017.
38. J. van Amersfoort, A. Kannan, M. Ranzato, A. Szlam, D. Tran, and S. Chintala. Transformation-based models of video sequences. arXiv preprint arXiv:1701.08435, 2017.
39. A. van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing Systems (NIPS), 2016.
40. R. Villegas, J. Yang, S. Hong, X. Lin, and H. Lee. Decomposing motion and content for natural video sequence prediction. In International Conference on Learning Representations (ICLR), 2017.
41. C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In Advances in Neural Information Processing Systems (NIPS), 2016.
42. L.-Y. Wei and M. Levoy. Fast texture synthesis using treestructured vector quantization. In ACM SIGGRAPH, 2000.
43. J. Xie, S.-C. Zhu, and Y. N. Wu. Synthesizing dynamic patterns by spatial-temporal generative convnet. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
44. T. Xue, J. Wu, K. Bouman, and B. Freeman. Probabilistic modeling of future frames from a single image. In Advances in Neural Information Processing Systems (NIPS), 2016.
45. H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. IEEE International Conference on Computer Vision (ICCV), 2016.