Zero-resource machine translation by multimodal encoder–decoder network with multimedia pivot

Machine Translation

Volumn 31, Issue 1-2, 2017, Pages 49-64

  Nakayama, Hideki ^a   Nishida, Noriki ^a  

^a UNIVERSITY OF TOKYO   (Japan)

Author keywords

Machine translation; Multimedia pivot; Multimodal embedding; Neural network; Zero resource learning

Indexed keywords

BENCHMARKING; COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; DECODING; NETWORK CODING; NEURAL NETWORKS; SEMANTICS; SIGNAL ENCODING; TRANSLATION (LANGUAGES);

EMBEDDED REPRESENTATION; END-TO-END MODELS; MACHINE TRANSLATION SYSTEMS; MACHINE TRANSLATIONS; MULTI-MODAL; MULTIMEDIA INFORMATION; MULTIMEDIA PIVOT; ZERO-RESOURCE LEARNING;

MULTIMEDIA SYSTEMS;

EID: 85019896469     PISSN: 09226567     EISSN: 15730573     Source Type: Journal    
DOI: 10.1007/s10590-017-9197-z     Document Type: Article

References (39)

1. Bergsma S, Van Durme B (2011) Learning bilingual lexicons using the visual similarity of labeled web images. In: Proc. IJCAI, pp 1764–1769
2. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: a Large-scale Hierarchical Image Database. In: Proc. IEEE CVPR
3. Elliott D, Frank S, Sima’an K, Specia L (2016) Multi30K: multilingual English–German image descriptions. In: Proceedings of the 5th ACL Workshop on Vision and Language, pp 70–74
4. Firat O, Sankaran B, Al-Onaizan Y, Vural FTY, Cho K (2016) Zero-resource translation with multi-lingual neural machine translation. In: Proc. EMNLP, pp 268–277
5. Frome A, Corrado G, Shlens J (2013) Devise: a deep visual-semantic embedding model. In: Proc. NIPS, pp 1–11
6. Funaki R, Nakayama H (2015) Image-mediated learning for zero-shot cross-lingual document retrieval. In: Proc. EMNLP, pp 585–590
7. Grübinger M, Clough P, Müller H, Deselaers T (2006) The IAPR TC-12 benchmark: a new evaluation resource for visual information systems. In: Proc. LREC, pp 13–23
8. Hardoon DR, Szedmak S, Shawe-taylor J (2004) Canonical correlation analysis: an overview with application to learning methods. Neural Comput 16(12):2639–2664
9. Hitschler J, Riezler S (2016) Multimodal pivots for image caption translation. In: Proc. ACL, pp 2399–2409
10. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1–32
11. Hotelling H (1936) Relations between two sets of variants. Biometrika 28:321–377
12. Huang PY, Liu F, Shiang SR, Oh J, Dyer C (2016) Attention-based multimodal neural machine translation. In: Proc. the first conference on machine translation (WMT), vol 2, pp 639–645
13. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. In: ACM conference on multimedia, pp 675–678
14. Johnson J, Karpathy A, Fei-Fei L (2016) DenseCap: fully convolutional localization networks for dense captioning. In: Proc. IEEE CVPR
15. Kiela D, Vulic I, Clark S (2015) Visual bilingual lexicon induction with transferred ConvNet features. In: Proc. EMNLP, pp 148–158
16. Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: Proc. ICLR
17. Kiros R, Salakhutdinov R, Zemel RS (2015) Unifying visual-semantic embeddings with multimodal neural language models. Transactions of the Association for Computational Linguistics (TACL)
18. Koehn P (2005) Europarl: a parallel corpus for statistical machine translation. Proc Mach Transl Summit 11:79–86
19. Koehn P (2009) Statistical machine translation. Cambridge University Press, Cambridge
20. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Proc. NIPS, pp 1097–1105
21. Lin CY, Och FJ (2004) Orange: a method for evaluating automatic evaluation metrics for machine translation. In: Proc. COLING, pp 501–507
22. Luong MT, Le QV, Sutskever I, Vinyals O, Kaiser L (2016) Multi-task sequence to sequence learning. In: Proc. ICLR
23. Oard D (1999) Issues in cross-language retrieval from document image collections. In: Proceedings of symposium on document image understanding technology, pp 229–234
24. Papineni K, Roukos S, Ward T, Zhu Wj (2002) BLEU: a method for automatic evaluation of machine translation. In: Proc. ACL, pp 311–318
25. Rajendran J, Khapra MM, Chandar S, Ravindran B (2016) Bridge correlational neural networks for multilingual multimodal representation learning. In: Proc. NAACL-HLT, pp 171–181
26. Riesa J, Marcu D (2012) Automatic parallel fragment extraction from noisy data. In: Proc. NAACL, pp 538–542
27. Saha A, Khapra MM, Chandar S, Rajendran J, Cho K (2016) A correlational encoder decoder architecture for pivot based sequence generation. In: Proc. COLING
28. Shen S, Cheng Y, He Z, He W, Wu H, Sun M, Liu Y (2016) Minimum risk training for neural machine translation. In: Proc. ACL, pp 1683–1692
29. Silberer C, Lapata M (2014) Learning grounded meaning representations with autoencoders. In: Proc. ACL, pp 721–732
30. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recoginition. In: Proc. ICLR
31. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Proc. NIPS, pp 3104–3112
32. Taeger W (2011) The sentence-aligned European Patent Corpus. In: Proc. EAMT, pp 177–184
33. Udupa R, Khapra MM (2010) Improving the multilingual user experience of wikipedia using cross-language name search. In: Proc. NAACL, pp 492–500
34. Uszkoreit J, Ponte J, Popat AC, Dubiner M (2010) Large scale parallel document mining for machine translation. In: Proc. COLING, pp 1101–1109
35. Vinyals O, Toshev A, Bengio S, Erhan D (2015) Show and tell: a neural image caption generator. In: Proc. IEEE CVPR
36. Vuli I, Kiela D, Clark S, Moens MF (2016) Multi-modal representations for improved bilingual lexicon learning. In: Proc. ACL, pp 188–194
37. Wu H, Wang H (2007) Pivot language approach for phrase-based statistical machine translation. Mach Transl 21(3):165–181
38. Wu H, Wang H (2009) Revisiting pivot language approach for machine translation. In: Proc. IJCNLP-ACL, pp 154–162
39. Young P, Lai A, Hodosh M, Hockenmaier J (2014) From Image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Trans ACL 2:67–78