Describing images using inferred visual dependency representations

ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference

Volumn 1, Issue , 2015, Pages 42-52

  Elliott, Desmond ^a   De Vries, Arjen P ^a  

^a CWI   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

ARTS COMPUTING; COMPUTATIONAL LINGUISTICS; DEEP NEURAL NETWORKS; OBJECT DETECTION;

EXPLICIT MODELING; HUMAN SUPERVISION; MULTI-MODAL; OBJECT DETECTORS; SPATIAL RELATIONSHIPS; STATE OF THE ART; TEMPLATE-BASED; TRAINING DATA;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 84943812736     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/v1/p15-1005     Document Type: Conference Paper

References (49)

1. Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2014. 2D Human Pose Estimation: New Benchmark and State of the Art Analysis. In CVPR 14, pages 3686-3693, Columbus, OH, US.
2. Moshe Bar and Shimon Ullman. 1996. Spatial Context in Recognition. Perception, 25(3):343-52.
3. Irving Biederman, Robert J Mezzanotte, and Jan C Rabinowitz. 1982. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive Psychology, 14(2):143-177.
4. JH Clark, Chris Dyer, Alon Lavie, and NA Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In ACL-HTL 11, pages 176-181, Portland, OR, U.S.A.
5. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In SMT at EMNLP 11, Edinburgh, Scotland, U.K.
6. Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Longterm Recurrent Convolutional Networks for Visual Recognition and Description. In CVPR 15, Boston, MA, U.S.A.
7. David Eigen, Christian Puhrsch, and Rob Fergus. 2014. Depth Map Prediction from a Single Image using a Multi-Scale Deep Network. In NIPS 27, Lake Tahoe, CA, U.S.A, June.
8. Desmond Elliott and Frank Keller. 2013. Image Description using Visual Dependency Representations. In EMNLP 13, pages 1292-1302, Seattle, WA, U.S.A.
9. Desmond Elliott and Frank Keller. 2014. Comparing Automatic Evaluation Measures for Image Description. In ACL 14, pages 452-457, Baltimore, MD, U.S.A.
10. Desmond Elliott, Victor Lavrenko, and Frank Keller. 2014. Query-by-Example Image Retrieval using Visual Dependency Representations. In COLING 14, pages 109-120, Dublin, Ireland.
11. Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. 2010. The PASCAL Visual Object Classes Challenge. IJCV, 88(2):303-338.
12. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivastava, Li Deng, Piotr Dolĺar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. 2015. From Captions to Visual Concepts and Back. In CVPR 15, Boston, MA, U.S.A.
13. Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: generating sentences from images. In ECCV 10, pages 15-29, Heraklion, Crete, Greece.
14. Yansong Feng and Mirella Lapata. 2008. Automatic Image Annotation Using Auxiliary Text Information. In ACL 08, pages 272-280, Colombus, Ohio.
15. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. CoRR, abs/1311.2.
16. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics. JAIR, 47:853-899.
17. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross B. Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. In MM 14, pages 675-678, Orlando, FL, U.S.A.
18. Andrej Karpathy and Li Fei-Fei. 2015. Deep Visual-Semantic Alignments for Generating Image Descriptions. In CVPR 15, Boston, MA, U.S.A.
19. Andrej Karpathy, Armand Joulin, and Li Fei-Fei. 2014. Deep Fragment Embeddings for Bidirectional Image Sentence Mapping. In NIPS 28, Montreal, Quebec, Canada.
20. Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. 2011. Baby talk: Understanding and generating simple image descriptions. In CVPR 11, pages 1601-1608, Colorado Springs, CO, U.S.A.
21. Polina Kuznetsova, Vicente Ordonez, Alexander C. Berg, Tamara L. Berg, and Yejin Choi. 2012. Collective Generation of Natural Image Descriptions. In ACL 12, pages 359-368, Jeju Island, South Korea.
22. Dieu-Thu Le, Jasper Uijlings, and Raffaella Bernardi. 2014. TUHOI : Trento Universal Human Object Interaction Dataset. In WVL at COLING 14, pages 17-24, Dublin, Ireland.
23. Remi Lebret, Pedro O. Pinheiro, and Ronan Collobert. 2015. Phrase-based Image Captioning. In ICML 15, Lille, France, February.
24. Siming Li, Girish Kulkarni, Tamara L. Berg, Alexander C. Berg, and Yejin Choi. 2011. Composing simple image descriptions using web-scale n-grams. In CoNLL 11, pages 220-228, Portland, OR, U.S.A.
25. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In ACL 04, pages 605-612, Barcelona, Spain.
26. Min Lin, Qiang Chen, and Shuicheng Yan. 2014a. Network In Network. In ICLR 14, volume abs/1312.4, Banff, Canada.
27. Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dolĺar. 2014b. Microsoft COCO: Common Objects in Context. In ECCV 14, pages 740-755, Zurich, Switzerland.
28. GD Logan and DD Sadler. 1996. A computational analysis of the apprehension of spatial relations. In Paul Bloom, Mary A. Peterson, Lynn Nadel, and Merrill F. Garrett, editors, Language and Space, pages 492-592. MIT Press.
29. Junhua Mao, Wei Xu, Yi Yang, Yiang Wang, and Alan L. Yuille. 2015. Deep captioning with multimodal recurrent neural networks (m-rnn). In ICLR 15, volume abs/1412.6632, San Diego, CA, U.S.A.
30. Emily E. Marsh and Marilyn Domas White. 2003. A taxonomy of relationships between images and text. Journal of Documentation, 59(6):647-672.
31. Guido Minnen, John Carroll, and Darren Pearce. 2001. Applied morphological processing of English. Natural Language Engineering, 7(3):207-223.
32. Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Alyssa Mensch, Alex Berg, Tamara Berg, and Hal Daum. 2012. Midge : Generating Image Descriptions From Computer Vision Detections. In EACL 12, pages 747-756, Avignon, France.
33. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülsen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. MaltParser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2):1.
34. Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. 2014. Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks. In CVPR 14, pages 1717-1724, Columbus, OH, US.
35. Luis M. G. Ortiz, Clemens Wolff, and Mirella Lapata. 2015. Learning to Interpret and Describe Abstract Scenes. In NAACL 15, Denver, CO, U.S.A.
36. Erwin Panofsky. 1939. Studies in Iconology. Oxford University Press.
37. Kishore Papineni, Salim Roukos, Todd Ward, and WJ Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL 02, pages 311-318, Philadelphia, PA, U.S.A.
38. DA Randell, Z Cui, and AG Cohn. 1992. A spatial logic based on regions and connection. In Principles of Knowledge Representation and Reasoning, pages 165-176.
39. Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. 2010. Collecting image annotations using Amazons Mechanical Turk. In AMT at NAACL 10, pages 139-147, Los Angeles, CA, U.S.A.
40. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. 2014. ImageNet Large Scale Visual Recognition Challenge.
41. Mohammad A Sadeghi and Ali Farhadi. 2011. Recognition Using Visual Phrases. In CVPR 11, pages 1745-1752, Colorado Springs, CO, U.S.A.
42. Sara Shatford. 1986. Analysing the Subject of a Picture: A Theoretical Approach. Cataloging &Classification Quarterly, 6(3):39-62.
43. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR 15, volume abs/1409.1, San Diego, CA, U.S.A.
44. Richard Socher, Andrej Karpathy, Q Le, C Manning, and A Ng. 2014. Grounded Compositional Semantics for Finding and Describing Images with Sentences. TACL, 2:207-218.
45. Kristina Toutanova, Dan Klein, and Christopher D Manning. 2003. Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. In HLTNAACL 03, pages 173-180, Edmonton, Canada.
46. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In CVPR 15, Boston, MA, U.S.A.
47. Yezhou Yang, Ching Lik Teo, Hal Daumé, and Yiannis Aloimonos. 2011. Corpus-Guided Sentence Generation of Natural Images. In EMNLP 11, pages 444-454, Edinburgh, Scotland, UK.
48. Mark Yatskar, Michel Galley, L Vanderwende, and L Zettlemoyer. 2014. See No Evil, Say No Evil: Description Generation from Densely Labeled Images. In SEM, pages 110-120, Dublin, Ireland.
49. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78