Paying Attention to Descriptions Generated by Image Captioning Models

Proceedings of the IEEE International Conference on Computer Vision

Volumn 2017-October, Issue , 2017, Pages 2506-2515

  Tavakoliy, Hamed R ^a   Shetty, Rakshith ^b   Borji, Ali ^c   Laaksonen, Jorma ^a  

^a AALTO UNIVERSITY   (Finland)

^b MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

^c University of Central Florida   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BEHAVIORAL RESEARCH;

BOTTOM-UP SALIENCIES; HUMAN DESCRIPTIONS; IMAGE CAPTIONING; LOW-LEVEL CUES; SALIENT OBJECTS; SCENE DESCRIPTION; VISUAL ATTENTION; WRITTEN DESCRIPTION;

COMPUTER VISION;

EID: 85041928364     PISSN: 15505499     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICCV.2017.272     Document Type: Conference Paper

References (68)

1. Microsoft COCO image captioning challenge. http: //competitions. codalab. org/competitions/ 3221. Accessed: 2016-03-01.
2. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual question answering. In ICCV, 2015.
3. H. Azizpour, A. S. Razavian, J. Sullivan, A. Maki, and S. Carlsson. From generic to specific deep representation for visual recognition. In CVPR Workshops, 2015.
4. A. C. Berg, T. L. Berg, H. Daum, J. Dodge, A. Goyal, X. Han, A. Mensch, M. Mitchell, A. Sood, K. Stratos, and K. Yamaguchi. Understanding and predicting importance in images. In CVPR, 2012.
5. R. Bernardi, R. Cakici, D. Elliott, A. Erdem, E. Erdem, N. Ikizler-Cinbis, F. Keller, A. Muscat, and B. Plank. Automatic description generation from images: A survey. J. Artif. Intell. Res., 55 (1), 2016.
6. K. Bock, D. Irwin, D. Davidson, andW. Levelt. Minding the clock. J. Mem. Lang., 48, 2003.
7. A. Borji and J. Tanner. Reconciling saliency and object center-bias hypotheses in explaining free-viewing fixations. IEEE Trans Neural Netw Learn Syst., 27 (6), 2016.
8. A. Borji, H. R. Tavakoli, D. N. Sihite, and L. Itti. Analysis of scores, datasets, and models in visual saliency prediction. In ICCV, 2013.
9. X. Chen, T.-Y. L. Hao Fang, R. Vedantam, S. Gupta, P. Dollr, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504. 00325, 2015.
10. M. Cimpoi, S. Maji, and A. Vedaldi. Deep filter banks for texture recognition and segmentation. In CVPR, 2015.
11. A. D. F. Clarke, M. Elsner, and H. Rohde. Giving good directions: order of mention reflects visual salience. Front. Psychol., 6, 2015.
12. M. Denkowski and A. Lavie. Meteor universal: Language specific translation evaluation for any target language. In EACL, 2014.
13. J. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. He, G. Zweig, and M. Mitchell. Language models for image captioning: The quirks and what work. In ACL, 2015.
14. J. Donahue, L. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
15. D. Elliott and A. P. de Vries. Describing images using inferred visual dependency representations. In ACL, 2015.
16. D. Elliott and F. Keller. Image description using visual dependency representations. In EMNLP, 2013.
17. M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 111 (1), 2015.
18. H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollar, J. Gao, X. He, M. Mitchell, J. Platt, L. Zitnick, and G. Zweig. From captions to visual concepts and back. In CVPR, 2015.
19. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In ECCV, 2010.
20. M. R. Greene. Statistics of high-level scene context. Front. Psychol., 4, 2013.
21. Z. Griffin and K. Bock. What the eyes say about speaking. Psychol Sci., 11 (4), 2000.
22. Z. M. Griffin and D. H. Spieler. Observing the what and when of language production for different age groups by monitoring speakers eye movements. Brain and Language, 99 (3):272-288, 2006. Language Comprehension across the Life Span.
23. J. Harel, C. Koch, and P. Perona. Graph-based visual saliency. In NIPS, 2006.
24. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
25. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Intell. Res., 47, 2013.
26. J. Holsanova. How we focus attention in picture viewing, picture description, and during mental imagery, pages 291-313. Bilder-sehen-denken : zum Verhltnis von begrifflichphilosophischen und empirisch-psychologischen Anstzen in der bildwissenschaftlichen Forschung. von Halem, 2011.
27. G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew. Extereme learning machine: Theory and applicatons. Neurocomput., 70, 2006.
28. L. Itti and M. A. Arbib. Action to Language via the Mirror Neuron System, chapter Attention and the Minimal Subscene. Combridge Press, 2006.
29. M. Jiang, S. Huang, J. Duan, and Q. Zhao. SALICON: Saliency in context. In CVPR, 2015.
30. T. Judd, K. Ehinger, F. Durand, and A. Torralba. Learning to predict where humans look. In ICCV, 2009.
31. M. k. Tanenhaus, C. Chambers, and J. E. Hanna. Referential domains in spoken language comprehension: Using eye movements to bridge the product and action traditions. In The interface of language, vision, and action: Eye movements and visual world. Psychology Press, 2004.
32. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.
33. A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. In NIPS, 2014.
34. S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Composing simple image descriptions using web-scale n-grams. In CoNLL, 2011.
35. C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL Workshop, 2004.
36. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
37. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 2015.
38. C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky. The Stanford CoreNLP natural language processing toolkit. In ACL, 2014.
39. S. Mathe and C. Sminchisescu. Actions in the eye: Dynamic gaze datasets and learnt saliency models for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell., 2015.
40. A. S. Meyer. The use of eye tracking in studies of sentence generation. In The interface of language, vision, and action: Eye movements and the visual world. Psychology Press, 2004.
41. A. S. Meyer, A. M. Sleiderink, andW. J. Levelt. Viewing and naming objects: eye movements during noun phrase production. Cognition, 66 (2), 1998.
42. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. ICLR, 2013.
43. R. Mottaghi, X. Chen, X. Liu, N. G. Cho, S. W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014.
44. V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In NIPS, 2011.
45. K. Papineni, S. Roukos, T. Ward, and W. jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
46. M. Pedersoli, T. Lucas, C. Schmid, and J. Verbeek. Areas of attention for image captioning. In CVPR, 2017.
47. R. J. Peters, A. Iyer, L. Itti, and C. Koch. Components of bottom-up gaze allocation in natural images. Vision Research, 45, 2005.
48. F. Pulvermüller, M. Hrle, and F. Hummel. Walking or talking: Behavioral and neurophysiological correlates of action verb processing. Brain and Language, 78 (2), 2001.
49. V. Ramanishka, A. Das, J. Zhang, and K. Saenko. Top-down visual saliency guided by captions. In CVPR, 2017.
50. C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier. Collecting image annotations using amazon's mechanical turk. In NAACL HLT, 2010.
51. A. Rohrbach, M. Rohrbach, and B. Schiele. The Long-Short Story of Movie Description. In GCPR, 2015.
52. R. Shetty and J. Laaksonen. Video captioning with recurrent networks based on frame-and video-level features and visual content classification. In CVPR Workshops, 2015.
53. R. Shetty, H. R-Tavakoli, and J. Laaksonen. Exploiting scene context for image captioning. In ACMMM Vision and Language Integration Meets Multimedia Fusion Workshop, 2016.
54. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409. 1556, 2014.
55. D. D. Sleator and D. Temperley. Parsing english with a link grammar. In Third International Workshop on Parsing Technologies, 1991.
56. S. N. Sridhar. Cognition and Sentence Production: A Cross-Linguistic Study, chapter Models of Sentence Production, pages 7-19. Springer New York, 1988.
57. Y. Sugano and A. Bulling. Seeing with humans: Gazeassisted neural image captioning. CoRR, abs/1608. 05203, 2016.
58. H. R. Tavakoli, F. Ahmad, A. Borji, and J. Laaksonen. Saliency revisited: Analysis of mouse movements versus fixations. In CVPR, 2017.
59. H. R. Tavakoli, A. Borji, J. Laaksonen, and E. Rahtu. Exploiting inter-image similarity and ensemble of extreme learners for fixation prediction using deep features. Neurocomput., 244, 2017.
60. R. Vedantam, C. L. Zitnick, and D. Parikh. CIDEr: Consensus-based image description evaluation. In CVPR, 2015.
61. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In CVPR, 2015.
62. K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2048-2057, Lille, France, 07-09 Jul 2015. PMLR.
63. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2, 2014.
64. K. Yun, Y. Peng, D. Samaras, G. Zelinsky, and T. Berg. Exploring the role of gaze behavior and object detection in scene understanding. Front. Psychol., 4, 2013.
65. K. Yun, Y. Peng, D. Samaras, G. J. Zelinsky, and T. L. Berg. Studying relationships between human gaze, description, and computer vision. In CVPR, 2013.
66. Q. Zhao and C. Koch. Learning saliency-based visual attention: A review. Signal Processing, 93, 2013.
67. C. L. Zitnick, R. Vedantam, and D. Parikh. Adopting abstract images for semantic scene understanding. IEEE Trans. Pattern Anal. Mach. Intell., 38 (4), 2016.
68. B. M. t Hart, H. C. E. F. Schmidt, C. Roth, andW. Einhauser. Fixations on objects in natural scenes: dissociating importance from salience. Front. Psychol., 4, 2013.