Beer: Better evaluation as ranking

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2014, Pages 414-419

  Stanojevic, Milos ^a   Simaan, Khalil ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS;

HUMAN-SYSTEMS; LINEAR MODELING; SENTENCE LEVEL; SYSTEM RANKINGS;

BEER;

EID: 84957860313     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Alexandra Birch and Miles Osborne. 2010. LRscore for Evaluating Lexical and Reordering Quality in MT. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 327?332, Uppsala, Sweden, July. Association for Computational Linguistics.
2. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) Evaluation of Machine Translation. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT 07, pages 136?158, Stroudsburg, PA, USA. Association for Computational Linguistics.
3. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 Workshop on Statistical Machine Translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10?51, Montreal, Canada, June. Association for Computational Linguistics.
4. Michael Collins, Philipp Koehn, and Ivona Kucerova. 2005. Clause Restructuring for Statistical Machine Translation. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL 05, pages 531?540, Stroudsburg, PA, USA. Association for Computational Linguistics.
5. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation.
6. Kevin Duh. 2008. Ranking vs. Regression in Machine Translation Evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, StatMT 08, pages 191?194, Stroudsburg, PA, USA. Association for Computational Linguistics.
7. Mark Hopkins and Jonathan May. 2011. Tuning as Ranking. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1352?1362, Edinburgh, Scotland, UK., July. Association for Computational Linguistics.
8. Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic Evaluation of Translation Quality for Distant Language Pairs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP 10, pages 944?952, Stroudsburg, PA, USA. Association for Computational Linguistics.
9. Alon Lavie and Abhaya Agarwal. 2008. METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In Proceedings of the ACL 2008 Workshop on Statistical Machine Translation.
10. Hang Li. 2011. Learning to Rank for Information Retrieval and Natural Language Processing. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
11. Matous Machacek and Ondrej Bojar. 2013. Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45?51, Sofia, Bulgaria, August. Association for Computational Linguistics.
12. Sebastian Pado, Michel Galley, Dan Jurafsky, and Christopher D. Manning. 2009. Textual Entailment Features for Machine Translation Evaluation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, StatMT 09, pages 37? 41, Stroudsburg, PA, USA. Association for Computational Linguistics.
13. Kishore Papineni, Salim Roukos, ToddWard, andWei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL 02, pages 311?318, Stroudsburg, PA, USA. Association for Computational Linguistics.
14. Xingyi Song and Trevor Cohn. 2011. Regression and Ranking based Optimisation for Sentence Level MT Evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 123?129, Edinburgh, Scotland, July. Association for Computational Linguistics.
15. Lucia Specia and Jesus Gimenez. 2010. Combining Confidence Estimation and Reference-based Metrics for Segment-level MT Evaluation. In Ninth Conference of the Association for Machine Translation in the Americas, AMTA-2010, Denver, Colorado.
16. Milos Stanojevic and Khalil Simaan. 2013. Evaluating Long Range Reordering with Permutation-Forests. In ILLC Prepublication Series, PP-2013-14. University of Amsterdam.
17. Mengqiu Wang and Christopher D. Manning. 2012. SPEDE: Probabilistic Edit Distance Metrics for MT Evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, WMT 12, pages 76?83, Stroudsburg, PA, USA. Association for Computational Linguistics.
18. Dekai Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational linguistics, 23(3):377?403.
19. Muyun Yang, Junguo Zhu, Sheng Li, and Tiejun Zhao. 2013. Fusion of Word and Letter Based Metrics for Automatic MT Evaluation. In Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, IJCAI13, pages 2204?2210. AAAI Press.
20. Yang Ye, Ming Zhou, and Chin-Yew Lin. 2007. Sentence Level Machine Translation Evaluation As a Ranking Problem: One Step Aside from BLEU. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT 07, pages 240?247, Stroudsburg, PA, USA. Association for Computational Linguistics.
21. Hao Zhang and Daniel Gildea. 2007. Factorization of synchronous context-free grammars in linear time. In In NAACL Workshop on Syntax and Structure in Statistical Translation (SSST.