Higher-Order Occurrence Pooling for Bags-of-Words: Visual Concept Detection

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 39, Issue 2, 2017, Pages 313-326

  Koniusz, Piotr ^a,b   Yan, Fei ^a   Gosselin, Philippe Henri ^c   Mikolajczyk, Krystian ^d  

^a UNIVERSITY OF SURREY   (United Kingdom)

^b CSIRO   (Australia)

^c ENSEA   (France)

^d IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

Bag of words; co occurrence; first order; mid level features; pooling operator; second order; sparse coding

Indexed keywords

AGGREGATES; COMPUTER VISION; OBJECT RECOGNITION;

BAG OF WORDS; CO-OCCURRENCE; FIRST ORDER; MID-LEVEL FEATURES; POOLING OPERATOR; SECOND ORDERS; SPARSE CODING;

INFORMATION RETRIEVAL;

EID: 85009910420     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2545667     Document Type: Article

References (64)

1. J. Sivic and A. Zisserman, "Video Google: A text retrieval approach to object matching in videos, " in Proc. 9th IEEE Int. Conf. Comput. Vis., 2003, pp. 1470-1477.
2. G. Csurka, C. R. Dance, L. Fan, J. Willamowski, and C. Bray, "Visual categorization with bags of keypoints, " in Proc. Workshop Statistical Learn. Comput. Vis., 2004, pp. 1-22.
3. D. G. Lowe, "Object recognition from local scale-invariant features, " in Proc. 7th IEEE Int. Conf. Comput. Vis., 1999, pp. 1150-1157.
4. K. Mikolajczyk and C. Schmid, "A performance evaluation of local descriptors, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 10, pp. 1615-1630, Oct. 2005.
5. K. E. A. van de Sande, T. Gevers, and C. G. M. Snoek, "A comparison of color features for visual concept classification, " in Proc. Int. Conf. Content-Based Image Video Retrieval, Jul. 2008, pp. 141-149.
6. J. C. van Gemert, J. M. Geusebroek, C. J. Veenman, and A. W. M. Smeulders, "Kernel codebooks for scene categorization, " in Proc. 10th Eur. Conf. Comput. Vis.: Part III, 2008, pp. 696-709.
7. J. C. van Gemert, C. J. Veenman, A. W. M. Smeulders, and J. M. Geusebroek, "Visual word ambiguity, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 7, pp. 1271-1283, Jul. 2010.
8. J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman, "Lost in quantization: Improving particular object retrieval in large scale image databases, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2008, p. 8.
9. P. Koniusz and K. Mikolajczyk, "Soft assignment of visual words as linear coordinate coding and optimisation of its reconstruction error, " inProc. 18th IEEE Int. Conf. Image Process., 2011, pp. 2413-2416.
10. L. Lingqiao, L. Wang, and X. Liu, "In defence of soft-assignment coding, " in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 2486-2493.
11. P. Koniusz, F. Yan, and K. Mikolajczyk, "Comparison of mid-level feature coding approaches and pooling strategies in visual con cept detection, " in Comput. Vis. Image Und., vol. 117, no. 5, pp. 479-492, 2012.
12. H. Lee, A. Battle, R. Raina, and A. Y. Ng, "Efficient sparse coding algorithms, " in Proc. Adv. Neural Inform. Process. Syst., 2007, pp. 801-808.
13. J. Yang, K. Yu, Y. Gong, and T. S. Huang, "Linear spatial pyramid matching using sparse coding for image classification, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 1794-1801.
14. K. Yu, T. Zhang, and Y. Gong, "Nonlinear learning using local coordinate coding, " NIPS, pp. 2223-2231, 2009.
15. J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, "Locality-constrained linear coding for image classification, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 3360-3367.
16. X. Zhou, K. Yu, T. Zhang, and T. S. Huang, "Image classification using super-vector coding of local image descriptors, " in Proc. 11th Eur. Conf. Comput. Vis.: Part V, 2010, pp. 141-154.
17. H. Jegou, M. Douze, C. Schmid, and P. Perez, "Aggregating local descriptors into a compact image representation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 3304-3311.
18. F. Perronnin and C. Dance, "Fisher kernels on visual vocabularies for image categorization, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007, pp. 1-8, 2007.
19. F. Perronnin, J. Sanchez, and T. Mensink, "Improving the Fisher kernel for large-scale image classification, " in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 143-156.
20. R. Negrel, D. Picard, and P.-H. Gosselin, "Compact tensor based image representation for similarity search, " in Proc. 19th IEEE Int. Conf. Image Process., 2012, pp. 2425-2428.
21. S. Boughorbel, J.-P. Tarel, and N. Boujemaa, "Generalized histo-gram intersection kernel for image recognition, " in Proc. IEEE Int. Conf. Image Process., 2005, pp. 161-164.
22. H. Jegou, M. Douze, and C. Schmid, "On the burstiness of visual elements, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 1169-1176.
23. J. Yang, Y.-G Jiang, A. G Hauptmann, and C.-W. Ngo, "Evaluating bag-of-visual-words representations in scene classi-fication, " in Proc. Int. Workshop Multimedia Inform. Retrieval, 2007, pp. 197-206.
24. K. Chatfield, V. Lempitsky, A. Vedaldi, and A. Zisserman, "The devil is in the details: An evaluation of recent feature encoding methods, " in Proc. Brit. Mach. Vis. Conf., 2011, pp. 76.1-76.12.
25. Y. Boureau, F. Bach, Y. LeCun, and J. Ponce, "Learning mid-level features for recognition, " in IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 2559-2566.
26. Y. Boureau, J. Ponce, and Y. LeCun, "A theoretical analysis of feature pooling in vision algorithms, " in Proc. Int. Conf. Mach. learn., 2010, pp. 111-118.
27. L. D. Lathauwer, B. D. Moor, and J. Vandewalle, "A multilinear singular value decomposition, " SIAM J. Matrix Anal. Appl., vol. 21, pp. 1253-1278, 2000.
28. T. G Kolda and B. W. Bader, "Tensor decompositions and applications, " SIAM Rev., vol. 51, no. 3, pp. 455-500, 2009.
29. J. Sanchez, F. Perronnin, and T. E. de Campos, "Modeling the spa-tial layout of images beyond spatial pyramids, " Pattern Recog. Lett., vol. 33, no. 6, pp. 2216-2223, 2012.
30. S. Lazebnik, C. Schmid, and J. Ponce, "Beyond bags of features: Spatial pyramid matching for recognizing natural scene catego-ries, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2006, pp. 2169-2178.
31. J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu, "Semantic segmentation with second-order pooling, " in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 430-443.
32. X. YU and Y.-J. ZHANG, "A 2-D histogram representation of images for pooling, " SPIE, 2011.
33. A. Krizhevsky, I. Sutskever, and G E. Hinton, "ImageNet classifi-cation with deep convolutional neural networks, " in Proc. Adv. Neural Inform. Process. Syst., 2012, pp. 1097-1105.
34. J. Bilmes, "A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models, " Int. Comput. Sci. Inst., Berkeley, CL, Tech. Rep. TR-97-021, 1997.
35. P. Koniusz and K. Mikolajczyk, "On a quest for image descriptors based on unsupervised segmentation maps, " in Proc. 20th Int. Conf. Pattern Recog., 2010, pp. 762-765.
36. M.-E. Nilsback and A. Zisserman, "Automated flower classification over a large number of classes, " in Proc. 6th Indian Conf. Comput. Vis. Graphics Image Process., Dec. 2008, pp. 722-729.
37. J. Yang, Y. Tian, L.-Y. Duan, T. Huang, and W. Gao, "Group-sensitive multiple kernel learning for object recognition, " IEEE Trans. Image Process., vol. 21, no. 5, pp. 2838-2852, May 2012.
38. F. Yan, K. Mikolajczyk, M. Barnard, H. Cai, and J. Kittler, "LP norm multiple kernel Fisher discriminant analysis for object and image categorisation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 3626-3632.
39. S. Nowak, K. Nagel, and J. Liebetra, "The CLEF 2011 photo annotation and concept-based retrieval tasks, " CLEF, 2011.
40. M. J. Huiskes and M. S. Lew, "The MIR Flickr retrieval eval-uation, " in Proc. 1st ACM Int. Conf. Multimedia Inform. Retrieval, 2008, pp. 39-43.
41. M. A. Tahir, F. Yan, M. Barnard, M. Awais, K. Mikolajczyk, and J. Kittler, "The University of Surrey visual concept detection system at ImageCLEF 2010: Working notes, " in Proc. ICPR Contests Recog. Patterns Signals, Speech, Images Videos, 2010, pp. 162-170.
42. A. Binder, W. Samek, and M. Kawanabe, "The joint submission of the TU Berlin and Fraunhofer FIRST (TUBFI) to the ImageCLEF 2011 photo annotation task: Working notes, " CLEF, 2011.
43. P. Koniusz and K. Mikolajczyk, "Spatial coordinate coding to reduce histogram representations, dominant angle and colour pyramid match, " in Proc. 18th IEEE Int. Conf. Image Process., 2011, pp. 661-664.
44. J. Yang, Z. Wang, Z. Lin, X. Shu, and T. Huang, "Bilevel sparse coding for coupled feature spaces, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2360-2367.
45. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. (2012). The PASCAL visual object classes challenge 2007-2012 results [Online]. Available: http://pascallin.ecs.soton. ac.uk/challenges/VOC
46. K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition, " in Proc. 13th Eur. Conf. Comput. Vis., 2014, pp. 346-361.
47. L. Fei-fei, R. Fergus, and P. Perona, "Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories, " in Proc. Conf. Comput. Vis. Pattern Recog. Workshop Generative-Model Based Vis., 2004, p. 178.
48. M. Awais, F. Yan, K. Mikolajczyk, and J. Kittler, "Novel fusion methods for pattern recognition, " in Proc. Eur. Conf. Mach. Learn. Knowl. Discovery Databases, 2011, pp. 140-155.
49. H. Azizpour, A. S. Razavian, J. Sullivan, A. Maki, and S. Carlsson, "From generic to specific deep representations for visual recog-nition, " 2014.
50. S. Gao, I. W. Tsang, L. Chia, and P. Zhao, "Local features are not lonely-Laplacian sparse coding for image classification, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 3555-3561.
51. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, "Learning deep features for scene recognition using places data-base, " in Proc. Adv. Neural Inform. Process. Syst., 2014, pp. 487-495.
52. B. Yao, X. Jiang, A. Khosla, A. L. Lin, L. J. Guibas, and L. Fei-Fei, "Action recognition by learning bases of action attributes and parts, " in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 1331-1338.
53. F. Khan, J. van de Weijer, R. M. Anwer, M. Felsberg, and C. Gatta, "Semantic pyramids for gender and action recognition, " IEEE Trans. Image Process., vol. 23, no. 8, pp. 3633-3645, Aug. 2014.
54. A. Quattoni and A. Torralba, "Recognizing indoor scenes, " in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog., 2009, pp. 413-420.
55. B. Yao, A. Khosla, and L. Fei-Fei, "Combining randomization and discrimination for fine-grained image categorization, " in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog., 2011, pp. 1577-1584.
56. J. Xiao, K. A. Ehinger, J. Hays, A. Torralba, and A. Oliva, "SUN database: Exploring a large collection of scene categories, " Proc. Int. J. Comput. Vis., 2014.
57. J. Sanchez, F. Perronnin, T. Mensink, and J. Verbeek, "Image classification with the Fisher vector: Theory and practice, " Int. J. Comput. Vis., vol. 105, no. 3, pp. 222-245, 2013.
58. J. Mairal, F. Bach, J. Ponce, and G Sapiro, "Online learning for matrix factorization and sparse coding, " J. Mach. Learn. Res., vol. 11, pp. 19-60, 2010.
59. M. Tahir, J. Kittler, K. Mikolajczyk, F. Yan, K. van de Sande, and T. Gevers, "Visual category recognition using spectral regression and kernel discriminant analysis, " in Proc. IEEE 12th Int. Conf. Comput. Vis. Workshop Subspace Methods, 2009, pp. 178-185.
60. P. Koniusz, "Novel image representations for visual categorisa-tion with bag-of-words, " Ph.D. dissertation, Centre Vis., Speech Signal Process, Univ. Surrey, Surrey GU2 7XH, United Kingdom, March 2013.
61. P. Koniusz, F. Yan, P. Gosselin, and K. Mikolajczyk, "Higherorder Occurrence Pooling on Mid-and Low-level Features: Visual Concept Detection, " http://claret.wikidot.com, 2013.
62. J. Mairal, P. Koniusz, Z. Harchaoui, and C. Schmid, "Convolutional kernel networks, " NIPS, 2014.
63. F. Yan and K. Mikolajczyk, "Leveraging high level visual information for matching images and captions, " in Proc. 12th Asian Conf. Comput. Vis., 2014, pp. 613-627.
64. F. Yan and K. Mikolajczyk, "Deep correlation for matching images and text, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3441-3450.