Accurate evaluation of segment-level machine translation metrics

NAACL HLT 2015 - 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

Volumn , Issue , 2015, Pages 1183-1191

  Graham, Yvette ^a,b   Mathur, Nitika ^a   Baldwin, Timothy ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

^b TRINITY COLLEGE DUBLIN   (Ireland)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION;

EFFECTIVE MECHANISMS; HUMAN ASSESSMENT; HUMAN EVALUATION; LANGUAGE PAIRS; MACHINE TRANSLATIONS; SIGNIFICANCE TESTING; STATE-OF-THE-ART PERFORMANCE;

QUALITY CONTROL;

EID: 84944091060     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/v1/n15-1124     Document Type: Conference Paper

References (17)

1. W. Aziz, S. Castilho, and L. Specia. 2012. PET: a tool for post-editing and assessing machine translation. In Proc. of the 8th International Conference on Language Resources and Evaluation (LREC 2012), pages 3982-3987, Istanbul, Turkey.
2. O. Bojar, C. Buck, C. Callison-Burch, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia. 2013a. Findings of the 2013Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1-44, Sofia, Bulgaria.
3. O. Bojar, C. Buck, C. Callison-Burch, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia. 2013b. Findings of the 2013 Workshop on Statistical Machine Translation. In Proc. 8th Wkshp. Statistical Machine Translation, pages 1-44, Sofia, Bulgaria.
4. O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, R. Soricut, L. Specia, and A. Tamchyna. 2014. Findings of the 2014 Workshop on Statistical Machine Translation. In Proc. 9th Wkshp. Statistical Machine Translation, pages 12-58, Baltimore, USA.
5. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2007. (Meta-) evaluation of machine translation. In Proc. 2nd Wkshp. Statistical Machine Translation, pages 136-158, Prague, Czech Republic.
6. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2008. Further meta-evaluation of machine translation. In Proc. III Wkshp. Statistical Machine Translation, pages 70-106, Columbus, USA.
7. C. Callison-Burch, P. Koehn, C. Monz, and J. Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In Proc. 4th Wkshp. Statistical Machine Translation, pages 1-28, Athens, Greece.
8. M. Denkowski and A. Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proc. 6th Wkshp. Statistical Machine Translation, pages 85-91, Edinburgh, Scotland.
9. Y. Graham and T. Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 172-176, Doha, Qatar.
10. Y. Graham, T. Baldwin, A. Moffat, and J. Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proc. 7th Linguistic Annotation Wkshp. & Interoperability with Discourse, pages 33-41, Sofia, Bulgaria.
11. Y. Graham, T. Baldwin, A. Moffat, and J. Zobel. 2014a. Is machine translation getting better over time? In Proceedings of the European Chapter of the Association of Computational Linguistics, pages 443-451, Gothenburg, Sweden.
12. Y. Graham, N. Mathur, and T. Baldwin. 2014b. Randomized significance tests in machine translation. In Proc. Ninth ACL Wkshp. Statistical Machine Translation, pages 266-74, Baltimore, MD.
13. A.L. Han, D.F. Wong, L.S. Chao, Y. Lu, L. He, Y. Wang, and J. Zhou. 2013. A description of tunable machine translation evaluation systems in WMT13 metrics task. In Proceedings of the Eight Workshop on Statistical Machine Translation, pages 414-421, Sofia, Bulgaria.
14. P. Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain.
15. M. Machacek and O. Bojar. 2014. Results of the wmt14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293-301, Baltimore, USA.
16. J.H. Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245.
17. E.J. Williams. 1959. Regression analysis, volume 14. Wiley, New York, USA.