Semantic Pooling for Complex Event Analysis in Untrimmed Videos

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 39, Issue 8, 2017, Pages 1617-1632

  Chang, Xiaojun ^a   Yu, Yao Liang ^b   Yang, Yi ^a   Xing, Eric P ^b  

^a UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

^b Carnegie Mellon University ^*  (United States)

Author keywords

Complex event detection; event recognition; event recounting; nearly isotonic SVM; semantic saliency

Indexed keywords

IMAGE RETRIEVAL; SUPPORT VECTOR MACHINES;

COMPLEX EVENT DETECTION; DISCRIMINATIVE POWER; EFFICIENT IMPLEMENTATION; EVENT RECOGNITION; EVENT RECOUNTING; NEARLY-ISOTONIC SVM; SUPPORT VECTOR MACHINE CLASSIFIERS; VIDEO REPRESENTATIONS;

SEMANTICS;

AUTOMATED PATTERN RECOGNITION; CLASSIFICATION; FACTUAL DATABASE; HUMAN; HUMAN ACTIVITIES; IMAGE PROCESSING; PROCEDURES; SEMANTICS; SUPPORT VECTOR MACHINE; VIDEORECORDING;

DATABASES, FACTUAL; HUMAN ACTIVITIES; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; PATTERN RECOGNITION, AUTOMATED; SEMANTICS; SUPPORT VECTOR MACHINE; VIDEO RECORDING;

EID: 85015732763     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2608901     Document Type: Article

References (75)

1. Z. Ma, Y. Yang, N. Sebe, K. Zheng, and A. G. Hauptmann, "Multimedia event detection using a classifier-specific intermedi ate representation, " IEEE Trans. Multimedia, vol. 15, no. 7, pp. 1628-1637, Nov. 2013.
2. A. Tamrakar, et al., "Evaluation of low-level features and their combinations for complex event detection in open source vid-eos, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 3681-3688.
3. Z. Ma, Y. Yang, N. Sebe, and A. G. Hauptmann, "Knowledge adaptation with partially shared features for event detection using few exemplars, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 9, pp. 1789-1802, Sep. 2014.
4. Y. Yang, F. Nie, D. Xu, J. Luo, Y. Zhuang, and Y. Pan, "A multimedia retrieval framework based on semi-supervised ranking and relevance feedback, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 4, pp. 723-742, Apr. 2012.
5. R. Aly, et al., "The AXES submissions at TrecVid 2013, " in TREC-VID 2013.
6. S.-I. Yu, et al., "Informedia@TRECVID 2014 MED and MER, " in TRECVID 2014.
7. L. Cao, Y. Mu, A. Natsev, S.-F. Chang, G. Hua, and J. R. Smith, "Scene aligned pooling for complex video recognition, " in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 688-701.
8. W. Li, Q. Yu, A. Divakaran, and N. Vasconcelos, "Dynamic pooling for complex event recognition, " presented at the IEEE Int. Conf. Comput. Vis., Sydney, Australia, 2013.
9. C. Koch and S. Ullman, "Shifts in selective visual attention: Towards the underlying neural circuitry, " Human Neurobiology, vol. 4, pp. 219-227, 1985.
10. E. Rahtu, J. Kannala, M. Salo, and J. Heikkila, "Segmenting salient objects from images and videos, " in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 366-379.
11. Y. J. Lee, J. Ghosh, and K. Grauman, "Discovering important peo ple and objects for egocentric video summarization, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 1346-1353.
12. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and F.-F. Li, "Large-scale video classification with convolutional neu-ral networks, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 1725-1732.
13. K. Soomro, A. R. Zamir, and M. Shah, "UCF101: A dataset of 101 human actions classes from videos in the wild, " Univ. Central Florida, Orlando, FL, USA, Tech. Rep. CRCV-TR-12-01, 2012.
14. B. Thomee, et al., "YFCC100M: The new data in multimedia research, " Comm. ACM, vol. 59, no. 2, pp. 64-73, 2016.
15. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality, " in Proc. Advances Neural Inf. Process. Syst., 2013, pp. 3111-3119.
16. J. Bolte, S. Sabach, and M. Teboulle, "Proximal alternating linearized minimization for nonconvex and nonsmooth problems, " Math. Program. Series A, vol. 146, pp. 459-494, 2014.
17. X. Chang, Y. Yang, E. P. Xing, and Y. Yu, "Complex event detection using semantic saliency and nearly-isotonic SVM, " in Proc. 32nd Int. Conf. Mach. Learn., 2015, pp. 1348-1357.
18. L. Duan, D. Xu, I. W. Tsang, and J. Luo, "Visual event recognition in videos by learning from Web data, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 9, pp. 1667-1680, Sep. 2012.
19. F. Wu, Y. Liu, and Y. Zhuang, "Tensor-based transductive learning for multimodality video semantic concept detection, " IEEE Trans. Multimedia, vol. 11, no. 5, pp. 868-878, Aug. 2009.
20. D. G. Lowe, "Distinctive image features from scale-invariant key-points, " Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
21. I. Laptev, B. Caputo, C. Schuldt, and T. Lindeberg, "Local veloc-ity-adapted motion events for spatio-temporal recognition, " Comput. Vis. Image Understanding, vol. 108, no. 3, pp. 207-229, 2007.
22. H. Wang and C. Schmid, "Action recognition with improved trajectories, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 3551-3558.
23. K. Simonyan and A. Zisserman, "Two-stream convolutional net-works for action recognition in videos, " in Proc. Advances Neural Inf. Process. Syst., 2014, pp. 568-576.
24. D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, and M. Paluri, "Learning spatiotemporal features with 3D convolutional networks, " in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 4489-4497.
25. Z. Xu, Y. Yang, and A. G Hauptmann, "A discriminative CNN video representation for event detection, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1798-1807.
26. S. Zha, F. Luisier, W. Andrews, N. Srivastava, and R. Salakhutdi-nov, "Exploiting image-trained CNN architectures for unconstrained video classification, " in Proc. 26th British Mach. Vis. Conf., 2015, pp. 60.1-60.13.
27. Z. Wu, X. Wang, Y. Jiang, H. Ye, and X. Xue, "Modeling spatial-temporal clues in a hybrid deep learning framework for video classification, " in Proc. 23rd ACM Conf. Multimedia Conf., 2015, pp. 461-470.
28. M. Nagel, T. Mensink, and C. G. Snoek, "Event Fisher vectors: Robust encoding visual diversity of visual streams, " in Proc. 26th British Mach. Vis. Conf., 2015, pp. 178.1-178.12.
29. D. Oneata, J. Verbeek, and C. Schmid, "Action and event recognition with Fisher vectors on a compact feature set, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1817-1824.
30. C. Sun and R. Nevatia, "DISCOVER: Discovering important segments for classification of video events and recounting, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 2569-2576.
31. P. Mettes, J. C. van Gemert, S. Cappallo, T. Mensink, and C. G M. Snoek, "Bag-of-fragments: Selecting and encoding video frag-ments for event detection and recounting, " in Proc. 5th ACM Int. Conf. Multimedia Retrieval, 2015, pp. 427-434.
32. P. Natarajan, et al., "Multimodal feature fusion for robust event detection in Web videos, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 1298-1305.
33. K. Tang, B. Yao, F.-F. Li, and D. Koller, "Combining the right fea-tures for complex event recognition, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 2696-2703.
34. J. Liu, S. McCloskey, and Y. Liu, "Local expert forest of score fusion for video event classification, " in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 397-410.
35. D. Liu, K.-T. Lai, G Ye, M.-S. Chen, and S.-F. Chang, "Sample-specific late fusion for visual category recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 803-810.
36. A. Vahdat, K. Cannons, G. Mori, S. Oh, and I. Kim, "Compositional models for video event detection: A multiple ker-nel learning latent variable approach, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1185-1192.
37. K.-T. Lai, D. Liu, M.-S. Chen, and S.-F. Chang, "Recognizing com-plex events in videos by learning key static-dynamic evidences, " in Proc. 13th Eur. Conf. Comput. Vis., 2014, pp. 675-688.
38. K. Tang, F.-F. Li, and D. Koller, "Learning latent temporal struc-ture for complex event detection, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 1250-1257.
39. K. Xu, et al., "Show, attend and tell: Neural image caption genera-tion with visual attention, " in Proc. 32nd Int. Conf. Mach. Learn., 2015, pp. 2048-2057.
40. S. Sharma, R. Kiros, and R. Salakhutdinov, "Action recognition using visual attention, " arXiv:1511.04119, 2016.
41. L. Li and F. Li, "What, where and who? Classifying events by scene and object recognition, " in Proc. IEEE 11th Int. Conf. Comput. Vis., 2007, pp. 1-8.
42. J. J. McAuley and J. Leskovec, "Image labeling on a network: Using social-network metadata for image classification, " in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 828-841.
43. L. Bossard, M. Guillaumin, and L. V. Gool, "Event recognition in photo collections with a stopwatch HMM, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1193-1200.
44. H. Izadinia and M. Shah, "Recognizing complex events using large margin joint low-level event model, " in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 430-444.
45. X. Zhang, et al., "Enhancing video event recognition using automatically constructed semantic-visual knowledge base, " IEEE Trans. Multimedia, vol. 17, no. 9, pp. 1562-1575, Sep. 2015.
46. J. Liu, et al., "Video event recognition using concept attributes, " in Proc. IEEE Workshop Appl. Comput. Vis., 2013, pp. 339-346.
47. C. Sun, et al., "ISOMER: Informative segment observations for multimedia event recounting, " in Proc. Int. Conf. Multimedia Retrieval, 2014, Art. no. 241.
48. C. C. Tan, Y. Jiang, and C. Ngo, "Towards textually describing complex video contents with audio-visual concept classifiers, " in Proc. 19th ACM Int. Conf. Multimedia, 2011, pp. 655-658.
49. A. Gupta, P. Srinivasan, J. Shi, and L. S. Davis, "Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 2012-2019.
50. X. Chang, Y.-L. Yu, Y. Yang, and A. G Hauptmann, "Searching persuasively: Joint event detection and evidence recounting with limited supervision, " in Proc. 23rd ACM Int. Conf. Multimedia, 2015, pp. 581-590.
51. J. Yuan, et al., "A formal study of shot boundary detection, " IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 2, pp. 168-186, Feb. 2007.
52. K. Simonyan and A. Zisserman, "Very deep convolutional net-works for large-scale image recognition, " in Proc. Int. Conf. Learn. Representations, 2015, pp. 1245-1258.
53. S. Bird, "NLTK: The natural language toolkit, " in Proc. COLING/ACL Interactive Presentation Sessions, 2006, pp. 69-72.
54. J. Sanchez, F. Perronnin, T. Mensink, and J. J. Verbeek, "Image classification with the Fisher vector: Theory and practice, " Int. J. Comput. Vis., vol. 105, no. 3, pp. 222-245, 2013.
55. S. Clinchant and F. Perronnin, "Textual similarity with a bag-of-embedded-words model, " in Proc. Conf. Theory Inf. Retrieval, 2013, Art. no. 25.
56. A. Vedaldi and B. Fulkerson, "VLFeat: An open and portable library of computer vision algorithms, " in Proc. 18th ACM Int. Conf. Multimedia, 2010, pp. 1469-1472.
57. C. H. Lampert, H. Nickisch, and S. Harmeling, "Learning to detect unseen object classes by between-class attribute transfer, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 951-958.
58. A. Habibian, T. Mensink, and C. G Snoek, "Videostory embeddings recognize events when examples are scarce, " arXiv:1511.02492, 2015.
59. R. E. Barlow, D. J. Bartholomew, J. M. Bremner, and H. D. Brunk, Statistical Inference Under Order Restrictions: The Theory and Applica-tion of Isotonic Regression. Hoboken, NJ, USA: Wiley, 1972.
60. R. J. Tibshirani, H. Hoefling, and R. Tibshirani, "Nearly-isotonic regression, " Technometrics, vol. 53, no. 1, pp. 54-61, 2011.
61. S. Yang, L. Yuan, Y.-C. Lai, X. Shen, P. Wonka, and J. Ye, "Feature grouping and selection over an undirected graph, " in Proc. 18th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2012, pp. 922-930.
62. L. I. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation based noise removal algorithms, " Physica D, vol. 60, pp. 259-268, 1992.
63. D. D. Lee and H. S. Seung, "Learning the parts of objects by non-negative matrix factorization, " Nature, vol. 401, pp. 788-791, 1999.
64. Y. Yu, H. Cheng, D. Schuurmans, and C. Szepesvari, "Characterizing the representer theorem, " in Proc. Int. Conf. Mach. Learn., 2013, pp. 570-578.
65. A. Rahimi and B. Recht, "Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning, " in Proc. Advances Neural Inf. Process. Syst., 2006, pp. 1313-1320.
66. M. Fukushima and H. Mine, "A generalized proximal point algorithm for certain non-convex minimization problems, " Int. J. Syst. Sci., vol. 12, no. 8, pp. 989-1000, 1981.
67. Y. Yu, "On decomposing the proximal map, " in Proc. Advances Neural Inf. Process. Syst., 2013, pp. 91-99.
68. P. L. Davies and A. Kovac, "Local extremes, runs, strings and multiresolution, " Ann. Statist., vol. 29, no. 1, pp. 1-65, 2001.
69. K. Crammer and Y. Singer, "On the algorithmic implementation of multiclass kernel-based vector machines, " J. Mach. Learn. Res., vol. 2, pp. 265-292, 2001.
70. Y. Jiang, G. Ye, S. Chang, D. P. W. Ellis, and A. C. Loui, "Consumer video understanding: A benchmark database and an evaluation of human and machine performance, " in Proc. 1st ACM Int. Conf. Multimedia Retrieval, 2011, Art. no. 29.
71. F. Perronnin, J. Sanchez, and T. Mensink, "Improving the Fisher kernel for large-scale image classification, " in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 143-156.
72. H. P. Graf, E. Cosatto, L. Bottou, I. Durdanovic, and V. Vapnik, "Parallel support vector machines: The cascade SVM, " in Proc. Advances Neural Inf. Process. Syst., 2004, pp. 521-528.
73. Z. Lan, M. Lin, X. Li, A. G Hauptmann, and B. Raj, "Beyond Gaussian pyramid: Multi-skip feature stacking for action recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 204-212.
74. T. Joachims, "Training linear SVMs in linear time, " in Proc. 12th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2006, pp. 217-226.
75. B. Ni, P. Moulin, and S. Yan, "Order preserving sparse coding, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 8, pp. 1615-1628, Aug. 2015.