End-to-End Learning of Action Detection from Frame Glimpses in Videos

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn 2016-December, Issue , 2016, Pages 2678-2687

  Yeung, Serena ^a   Russakovsky, Olga ^a,b   Mori, Greg ^c   Fei Fei, Li ^a  

^a STANFORD UNIVERSITY   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

^c SIMON FRASER UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; RECURRENT NEURAL NETWORKS;

DECISION POLICY; END TO END; NON-DIFFERENTIABLE; STATE OF THE ART; VIDEO FRAME;

PATTERN RECOGNITION;

EID: 84986253505     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2016.293     Document Type: Conference Paper

References (50)

1. J. Ba, V. Mnih, and K. Kavukcuoglu. Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755, 2014.
2. M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri. Actions as space-time shapes. In Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on, volume 2, pages 1395-1402. IEEE, 2005.
3. F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970, 2015.
4. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. arXiv preprint arXiv:1411.4389, 2014.
5. D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov. Scalable object detection using deep neural networks. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 2155-2162. IEEE, 2014.
6. R. Girshick. Fast r-cnn. arXiv preprint arXiv:1504.08083, 2015.
7. G. Gkioxari and J. Malik. Finding action tubes. arXiv preprint arXiv:1411.6031, 2014.
8. A. Gupta, P. Srinivasan, J. Shi, and L. S. Davis. Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 2012-2019. IEEE, 2009.
9. M. Jain, J. Van Gemert, H. Jégou, P. Bouthemy, and C. G. Snoek. Action localization with tubelets from motion. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 740-747. IEEE, 2014.
10. H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black. Towards understanding action recognition. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 3192-3199. IEEE, 2013.
11. Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. THUMOS challenge: Action recognition with a large number of classes. http: //crcv.ucf.edu/THUMOS14/, 2014.
12. V. Kantorov and I. Laptev. Efficient feature extraction, encoding, and classification for action recognition. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 2593-2600. IEEE, 2014.
13. S. Karaman, L. Seidenari, and A. Del Bimbo. Fast saliency based pooling of fisher encoded dense trajectories.
14. Y. Ke, R. Sukthankar, and M. Hebert. Event detection in crowded videos. In ICCV, 2007.
15. S. Kwak, B. Han, and J. H. Han. Multi-agent event detection: Localization and role assignment. In CVPR, 2013.
16. T. Lan, Y. Wang, and G. Mori. Discriminative figure-centric models for joint action localization and recognition. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2003-2010. IEEE, 2011.
17. I. Laptev, M. Marsza?ek, C. Schmid, and B. Rozenfeld. Learning realistic human actions from movies. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1-8. IEEE, 2008.
18. C. C. Loy, T. Xiang, and S. Gong. Incremental activity modeling in multiple disjoint cameras. TPAMI, 34(9):1799-1813, 2012.
19. V. Mnih, N. Heess, A. Graves, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pages 2204-2212, 2014.
20. D. Moore and I. Essa. Recognizing multitasked activities from video using stochastic context-free grammar. 2002.
21. B. Ni, V. R. Paramathayalan, and P. Moulin. Multiple granularity analysis for fine-grained action detection. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 756-763. IEEE, 2014.
22. D. Oneata, J. Verbeek, and C. Schmid. The lear submission at thumos 2014. 2014.
23. H. Pirsiavash and D. Ramanan. Parsing videos of actions with segmental grammars. In Computer Vision and Pattern Recognition (CVPR), 2014.
24. R. Poppe. A survey on vision-based human action recognition. IVC, 28:976-990, 2010.
25. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. arXiv preprint arXiv:1506.02640, 2015.
26. S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497, 2015.
27. M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele. A database for fine grained activity detection of cooking activities. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 1194-1201. IEEE, 2012.
28. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, pages 1-42, 2014.
29. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013.
30. P. Sermanet, A. Frome, and E. Real. Attention for finegrained categorization. arXiv preprint arXiv:1412.7054, 2014.
31. Y. Shi, A. Bobick, and I. Essa. Learning temporal sequence model from partially labeled data. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, volume 2, pages 1631-1638. IEEE, 2006.
32. T. Shu, D. Xie, B. Rothrock, S. Todorovic, and S.-C. Zhu. Joint inference of groups, events and human roles in aerial videos. In CVPR, 2015.
33. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
34. C. Sun, S. Shetty, R. Sukthankar, and R. Nevatia. Temporal localization of fine-grained actions in videos by domain transfer from web images. arXiv preprint arXiv:1504.00983, 2015.
35. C. Szegedy, S. Reed, D. Erhan, and D. Anguelov. Scalable, high-quality object detection. arXiv preprint arXiv:1412.1441, 2014.
36. Y. Tian, R. Sukthankar, and M. Shah. Spatiotemporal deformable part models for action detection. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 2642-2649. IEEE, 2013.
37. T. Tieleman and G. E. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude., 2012.
38. H. Wang and C. Schmid. Action recognition with improved trajectories. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 3551-3558. IEEE, 2013.
39. L. Wang, Y. Qiao, and X. Tang. Action recognition and detection by combining motion and appearance features.
40. D. Weinland, R. Ronfard, and E. Boyer. A survey of visionbased methods for action representation, segmentation and recognition. In Computer Vision and Image Understanding, Vol. 115, Issues 2, pp. 224,241, 2010.
41. P. Weinzaepfel, Z. Harchaoui, and C. Schmid. Learning to track for spatio-temporal action localization. arXiv preprint arXiv:1506.01929, 2015.
42. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
43. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2015.
44. A. Yao, J. Gall, and L. Van Gool. A hough transform-based voting framework for action recognition. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 2061-2068. IEEE, 2010.
45. G. Yu and J. Yuan. Fast action proposals for human action detection and search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1302-1311, 2015.
46. J. Yuan, Y. Pei, B. Ni, P. Moulin, and A. Kassim. Adsc submission at thumos challenge 2015.
47. W. Zaremba and I. Sutskever. Reinforcement learning neural turing machines. arXiv preprint arXiv:1505.00521, 2015.
48. S. Zha, F. Luisier, W. Andrews, N. Srivastava, and R. Salakhutdinov. Exploiting image-trained cnn architectures for unconstrained video classification. arXiv preprint arXiv:1503.04144, 2015.
49. H. Zhong, J. Shi, and M. Visontai. Detecting unusual activity in video. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II-819. IEEE, 2004.
50. Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv preprint arXiv:1506.06724, 2015.