The NIST 2008 metrics for machine translation challenge-overview, methodology, metrics, and results

Machine Translation

Volumn 23, Issue 2-3, 2009, Pages 71-103

  Przybocki, Mark ^a   Peterson, Kay ^a   Bronsart, Sébastien ^a   Sanders, Gregory ^a  

^a NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

Author keywords

Automated MT metrics; Metric evaluation; MetricsMATR

Indexed keywords

CORRELATION STATISTICS; GENERAL CLASS; MACHINE TRANSLATIONS; METRIC EVALUATION; SYSTEM LEVELS; TEST DATA;

CLUSTER ANALYSIS; INFORMATION THEORY; REGRESSION ANALYSIS; SPEECH TRANSMISSION; TEST FACILITIES;

AUTOMATION;

EID: 77954761816     PISSN: 09226567     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10590-009-9065-6     Document Type: Article

References (31)

1. Babych B, Hartley A (2004) Extending the BLEU MT evaluation method with frequency weightings. Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL'04). Association for Computational Linguistics, Barcelona, Spain
2. Callison-Burch C, Fordyce C, Koehn P, Monz C, Schroeder J (2007) (Meta-) evaluation of machine translation. Proceedings ofthe Second WorkshoponStatistical Machine Translation. Prague Czech Republic, Association for Computational Linguistics
3. Callison-Burch C, Fordyce C, Koehn P, Monz C, Schroeder J (2008) Further meta-evaluation of machine translation. Proceedings of the Third Workshop on Statistical Machine Translation (WMT08). Association for Computational Linguistics, Columbus OH
4. Callison-Burch C, Osborne M, Koehn P (2006) Re-evaluating the role of BLEU in machine translation research. Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Trento, Italy
5. Chiang D, Knight K, Wang W (2009) 11, 001 New features for statistical machine translation. Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2009). Association for Computational Linguistics, Boulder, CO
6. Chiang D, DeNeefe S, Chan YS, Ng HT (2008) Decomposability of translation metrics for improved evaluation and efficient algorithms. Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii
7. Chinchor N, Hirschman L, Lewis DD (1993) Evaluating message understanding systems: an analysis of the third message understanding conference (MUC-3). Computational Linguistics 19 (3)
8. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 37-46
9. Condon S, Sanders GA, Parvaz D, Rubenstein A, Doran C, Aberdeen J, Oshika B (2009) Normalization for automated metrics: English and Arabic speech translation. Proceedings of MT Summit XII. Association for Machine Translation in the Americas, Ottawa, ON, Canada
10. Coughlin D (2003) Correlating automated and human assessments of machine translation quality. Proceedings of MT Summit IX. Association for Machine Translation in the Americas, New Orleans, LA
11. Doddington G (2002) Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. Proceedings of the Second International Conference on Human Language Technology Research, Morgan Kaufmann, San Diego, CA
12. Fellbaum C (1998) Wordnet: an electronic lexical database. Bradford Books
13. Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76
14. Fleiss JL, Cohen J, Everitt BS (1969) Large sample standard errors of Kappa and weighted Kappa. Psychol Bull 72
15. Hodges JL, Lehmann EL (1963) Estimates of location based on tank tests. Ann Math Stat 34
16. Jones D, Herzog M, Ibrahim H, Jairam A, Shen W, Gibson E, Emonts M (2007) ILR-based MT comprehension test with multi-level questions. Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2007). Association for Computational Linguistics, Rochester, NY
17. Kendall MG (1938) A new measure of rank correlation. Biometrika 30
18. Lavie A, Agarwal A (2007) METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. Workshop on statistical machine translation at the 45th annual meeting of the association of computational linguistics (ACL-2007). Prague
19. Neter J, Kutner M, Nachtsheim C, Wasserman W (1996) Applied linear statistical models. McGraw-Hill/Irwin
20. Noreen EW (1989) Computer intensive methods for testing hypotheses. An introduction. Wiley, New York
21. Och FJ (2003) Minimum error rate training in statistical machine translation. Association for Computational Linguistics, Sapporo, Japan
22. Papineni K, Roukos S, Ward T, Zhu W-J (2001) BLEU: a method for automatic evaluation of machine translation. Technical Report, Yorktown Heights NY. IBM Research Division
23. Paul M (2006) Overview of the IWSLT 2006 evaluation campaign. Proceedings of the international workshop on spoken language translation. Kyoto, Japan
24. Pearson K (1900) On a criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can reasonably be supposed to have arisen in random sampling. Philos Mag 50
25. Porter M (2009) Snowball: a language for stemming algorithms. 2001. http://snowball.tartarus.org/texts/introduction.html. Accessed 28 Oct. 2009)
26. Przybocki M, Peterson K, Bronsart S (2008) NIST metrics for machine translation challenge (MetricsMATR). NIST Multimodal Information Group. National Institute of Standards and Technol ogy. http://www.nist.gov/speech/tests/ metricsmatr/2008/doc/mm08-evalplan-v1.1pdf. Accessed 28 Oct. 2009)
27. Riezler J, Maxwell JT (2005) On some pitfalls in automatic evaluation and significance testing for MT. ACL-05 workshop on intrinsic and extrinsic evaluation measures for MT and/or summarization. Association for Computational Linguistics, Ann Arbor, MI
28. Sanders GA, Bronsart S, Condon S, Schlenoff C (2008) Odds of successful transfer of low-level concepts: a key metric for bidirectional speech-to-speech machine translation in DARPA's TRANSTAC program. Proceedings of the 6th international conference on language resources and evaluation (LREC'08). European Language Resources Association (ELRA), Marrakech, Morocco
29. Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul J (2006) A study of translation edit rate with targeted human annotation. Proceedings of association for machine translation in the Americas. Cambridge, MA
30. Spearman CE (1904) The proof and measurement of association between two things. Am J Psychol 15
31. Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics 1