ΔbLEU: A discriminative metric for generation tasks with intrinsically diverse targets

ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference

Volumn 2, Issue , 2015, Pages 445-450

  Galley, Michel ^a   Brockett, Chris ^a   Sordoni, Alessandro ^b   Ji, Yangfeng ^c   Aim, Michael ^d   Quirk, Chris ^a   Mitchell, Margaret ^a   Gao, Jianfeng ^a   Dolan, Bill ^a  

^a MICROSOFT RESEARCH   (United States)

^b UNIVERSITÉ DE MONTRÉAL   (Canada)

^c GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^d FACEBOOK AI RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; LINGUISTICS; SCALES (WEIGHING INSTRUMENTS);

DIVERSE RANGE; HUMAN JUDGMENTS; KENDALL'S TAU; MULTI REFERENCE; SENTENCE LEVEL; SPEARMAN'S RHO;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 84944036795     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. Satanjeev Banerjee and Alon Lavie. 2005. meteor: An automatic metric for MT evaluation with im-proved correlation with human judgments. In Proc. of ACL Workshop on Intrinsic and Extrinsic Evalua-tion Measures for Machine Translation and/or Sum-marization, pages 65-72
2. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of bleu in machine translation research. In EACL, pages 249-256
3. Deborah Coughlin. 2003. Correlating automated and human assessments of machine translation quality. In Proc. ofMT Summit IX, pages 63-70
4. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. of HLT, pages 138-145
5. Markus Dreyer and Daniel Marcu. 2012. HyTER: Meaning-equivalent semantics for translation evalu-ation. In Proc. of HLT-NAACL, pages 162-171
6. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proc. of EMNLP, pages 172-176
7. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level ma-chine translation metrics. In Proc. of NAACL-HLT, pages 1183-1191
8. Micah Hodosh, Peter Young, and Mia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Int. Res., 47(1):853-899
9. Po-Sen Huang, Xiaodong He, Lianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proc. of the 22nd ACM Inter-national Conference on Information & Knowledge Management, pages 2333-2338
10. Alon Lavie and Abhaya Agarwal. 2007. meteor: An automatic metric for mt evaluation with high lev-els of correlation with human judgments. In Proc. of the Workshop on Statistical Machine Translation (StatMT), pages 228-231
11. Preslav Nakov, Francisco Guzman, and Stephan Vo-gel. 2012. Optimizing for Sentence-Level BLEU+1 Yields Short Translations. In Proc. of COLING
12. Franz Losef Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of ACL, pages 160-167
13. Kishore Papineni, Salim Roukos, Todd Ward, and Weiling Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL, pages 311-318
14. M. Przybocki, K. Peterson, and S. Bronsart. 2008. Official results of the NIST 2008 "Metrics for MA-chine TRanslation" challenge (MetricsMATR08). http://nist.gov/speech/tests/metricsmatr/2008/results/
15. Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proc. of EMNLP, pages 583-593
16. Stephen E Robertson, Steve Walker, Susan lones, et al. 1995. Okapi at TREC-3. In TREC
17. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Li, Meg Mitchell, Lian-Yun Nie, Lianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proc. of NAACL-HLT
18. Hong Sun and Ming Zhou. 2012. loint learning of a dual SMT system for paraphrase generation. In ACL, pages 38-12
19. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In CVPR