Associating neural word embeddings with deep image representations using Fisher Vectors

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn 07-12-June-2015, Issue , 2015, Pages 4437-4446

  Klein, Benjamin ^a   Lev, Guy ^a   Sadeh, Gil ^a   Wolf, Lior ^a  

^a TEL AVIV UNIVERSITY   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; GAUSSIAN DISTRIBUTION; IMAGE ANALYSIS; IMAGE RETRIEVAL; IMAGE SEGMENTATION; LAPLACE TRANSFORMS; MAXIMUM PRINCIPLE; PATTERN RECOGNITION;

EXPECTATION - MAXIMIZATIONS; GAUSSIAN MIXTURE MODEL; IMAGE ANNOTATION; IMAGE REPRESENTATIONS; LAPLACIAN DISTRIBUTION; LAPLACIAN MIXTURE MODELS; LOG LIKELIHOOD; STATE OF THE ART;

VECTORS;

EID: 84959196607     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2015.7299073     Document Type: Conference Paper

References (48)

1. Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. A neural probabilistic language model. J. Mach. Learn. Res., 3:1137-1155, Mar. 2003
2. K. Chatfield, V. Lempitsky, A. Vedaldi, and A. Zisserman. The devil is in the details: an evaluation of recent feature encoding methods. In British Machine Vision Conference, 2011
3. S. Clinchant and F. Perronnin. Textual similarity with a bag-of-embedded-words model. In Proceedings of the 2013 Conference on the Theory of Information Retrieval, page 25. ACM, 2013
4. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological), pages 1-38, 1977
5. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. arXiv preprint arXiv:1411. 4389v2, 2014
6. T. Eltoft, T. Kim, and T.-W. Lee. On the multivariate laplace distribution. Signal Processing Letters, IEEE, 13(5):300-303, 2006
7. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In Computer Vision-ECCV 2010, pages 15-29. Springer, 2010
8. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv preprint arXiv:1311. 2524, 2013
9. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207. 0580, 2012
10. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Intell. Res. (JAIR), 47:853-899, 2013
11. P. Y. A. L. M. Hodosh and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014
12. H. Hotelling. Relations between two sets of variates. Biometrika, pages 321-377, 1936
13. Y. Jia and T. Darrell. Heavy-tailed distances for gradient based image descriptors. In Advances in Neural Information Processing Systems, pages 397-405, 2011
14. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. Technical report, Computer Science Department, Stanford University, 2014
15. A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. arXiv preprint arXiv:1406. 5679, 2014
16. Y. Ke and R. Sukthankar. Pca-sift: A more distinctive representation for local image descriptors. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II-506. IEEE, 2004
17. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411. 2539, 2014
18. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Understanding and generating simple image descriptions. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1601-1608. IEEE, 2011
19. P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. Collective generation of natural image descriptions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 359-368. Association for Computational Linguistics, 2012
20. Q. V. Le and T. Mikolov. Distributed representations of sentences and documents. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Proceedings, pages 1188-1196. JMLR. org, 2014
21. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998
22. T.-W. Lee. Independent component analysis: theory and applications [book review]. IEEE Transactions on Neural Networks, 10(4):982-982, 1999
23. S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Composing simple image descriptions using web-scale ngrams. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pages 220-228. Association for Computational Linguistics, 2011
24. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollr, and C. Zitnick. Microsoft coco: Common objects in context. In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, Computer Vision ECCV 2014, volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer International Publishing, 2014
25. D. G. Lowe. Object recognition from local scale-invariant features. In Computer vision, 1999. The proceedings of the seventh IEEE international conference on, volume 2, pages 1150-1157. Ieee, 1999
26. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. arXiv preprint arXiv:1410. 1090, 2014
27. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. CoRR, abs/1301. 3781, 2013
28. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111-3119, 2013
29. M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé III. Midge: Generating image descriptions from computer vision detections. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 747-756. Association for Computational Linguistics, 2012
30. A. Mnih and G. Hinton. Three new graphical models for statistical language modelling. In Proceedings of the 24th International Conference on Machine Learning, ICML '07, pages 641-648, New York, NY, USA, 2007. ACM
31. R. Neal. Annealed importance sampling. Statistics and Computing, 11(2):125-139, 2001
32. X. Peng, C. Zou, Y. Qiao, and Q. Peng. Action recognition with stacked fisher vectors. In Computer Vision-ECCV 2014, pages 581-595. Springer, 2014
33. F. Perronnin and C. Dance. Fisher kernels on visual vocabularies for image categorization. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on, pages 1-8. IEEE, 2007
34. F. Perronnin, Y. Liu, J. Sánchez, and H. Poirier. Large-scale image retrieval with compressed fisher vectors. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 3384-3391. IEEE, 2010
35. F. Perronnin, J. Sánchez, and T. Mensink. Improving the fisher kernel for large-scale image classification. In Computer Vision-ECCV 2010, pages 143-156. Springer, 2010
36. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier. Collecting image annotations using amazon's mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 139-147. Association for Computational Linguistics, 2010
37. J. Sánchez, F. Perronnin, T. Mensink, and J. Verbeek. Image classification with the fisher vector: Theory and practice. International journal of computer vision, 105(3):222-245, 2013
38. K. Simonyan, O. M. Parkhi, A. Vedaldi, and A. Zisserman. Fisher vector faces in the wild. In Proc. BMVC, volume 1, page 7, 2013
39. K. Simonyan, A. Vedaldi, and A. Zisserman. Deep fisher networks for large-scale image classification. In Advances in neural information processing systems, pages 163-171, 2013
40. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409. 1556, 2014
41. J. Sivic, B. C. Russell, A. A. Efros, A. Zisserman, and W. T. Freeman. Discovering objects and their location in images. In Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on, volume 1, pages 370-377. IEEE, 2005
42. R. Socher and L. Fei-Fei. Connecting modalities: Semisupervised segmentation and annotation of images using unaligned text corpora. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 966-973. IEEE, 2010
43. R. Socher, Q. Le, C. Manning, and A. Ng. Grounded compositional semantics for finding and describing images with sentences. In NIPS Deep Learning Workshop, 2013
44. R. Socher, C. C. Lin, A. Y. Ng, and C. D. Manning. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In Proceedings of the 26th International Conference on Machine Learning (ICML), 2011
45. V. Sydorov, M. Sakurada, and C. H. Lampert. Deep fisher kernels-end to end learning of the fisher kernel gmm parameters. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2014
46. H. Vinod. Canonical ridge and econometrics of joint production. Journal of Econometrics, 4(2):147-166, 1976
47. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411. 4555, 2014
48. C. L. Zitnick, D. Parikh, and L. Vanderwende. Learning the visual interpretation of sentences. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 1681-1688. IEEE, 2013.