A dataset for assessing machine translation evaluation metrics

Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010

Volumn , Issue , 2010, Pages 3375-3378

  Specia, Lucia ^a   Cancedda, Nicola ^b   Dymetman, Marc ^b  

^a UNIVERSITY OF WOLVERHAMPTON   (United Kingdom)

^b XEROX RESEARCH CENTRE EUROPE   (France)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; LEARNING SYSTEMS; STATISTICAL TESTS;

CORRELATION ANALYSIS; MACHINE TRANSLATION EVALUATIONS; MACHINE TRANSLATION SYSTEMS; MT EVALUATIONS;

TRANSLATION (LANGUAGES);

EID: 85037329553     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. J. Blatz, E. Fitzgerald, G. Foster, S. Gandrabur, C. Goutte, A. Kulesza, A. Sanchis, and N. Ueffing. 2004. Confidence Estimation for Machine Translation. In 20th Coling, pages 315-321, Geneva.
2. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2007. (Meta-) Evaluation of Machine Translation. In 2nd Workshop on Statistical Machine Translation, pages 136-158, Prague.
3. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2008. Further Meta-Evaluation of Machine Translation. In 3rd Workshop on Statistical Machine Translation, pages 70-106, Columbus.
4. C. Callison-Burch, P. Koehn, C. Monz, and J. Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In 4th Workshop on Statistical Machine Translation, pages 1-28.
5. J. Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46, April.
6. G. Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co-occurrence Statistics. In 2nd Conference on Human Language Technology Research, pages 138-145, San Diego.
7. J. Gimenez and L. Marquez. 2008. A Smorgasbord of Features for Automatic MT Evaluation. In 3rd Workshop on Statistical Machine Translation, pages 195-198, Columbus, Ohio.
8. H. Johnson, F. Sadat, G. Foster, R. Kuhn, M. Simard, E. Joanis, and S. Larkin. 2006. Portage with Smoothed Phrase Tables and Segment Choice Models. In Workshop on Statistical Machine Translation, pages 134-137, New York.
9. M. Kääriäinen. 2009. Sinuhe - Statistical Machine Translation using a Globally Trained Conditional Exponential Family Translation Model. In Conference on Empirical Methods in Natural Language Processing, pages 1027-1036, Singapore.
10. P. Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. In Conference on Empirical Methods in Natural Language Processing, Barcelona.
11. P. Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In MT Summit.
12. R. J. Landis and G. G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1):159-174.
13. A. Lavie and A. Agarwal. 2007. METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In 2nd Workshop on Statistical Machine Translation, pages 228-231, Prague.
14. C. Y. Lin and F. J. Och. 2004. ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In Coling-2004, pages 501-507, Geneva.
15. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Morris-town.
16. C. B. Quirk. 2004. Training a Sentence-Level Machine Translation Confidence Measure. In 4th Conference on Language Resources and Evaluation, pages 825-828, Lisbon.
17. C. Saunders. 2008. Application of Markov Approaches to Statistical Machine Translation. Technical report, SMART Project Deliverable 2.2.
18. M. Simard, N. Cancedda, B. Cavestro, M. Dymetman, E. Gaussier, C. Goutte, and K. Yamada. 2005. Translating with Non-contiguous Phrases. In Conference on Empirical Methods in Natural Language, pages 755-762, Vancouver.
19. M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In 7th Conference of the Association for Machine Translation in the America, pages 223-231, Cambridge, MA.
20. L. Specia, M. Turchi, N. Cancedda, M. Dymetman, and N. Cristianini. 2009. Estimating the Sentence-Level Quality of Machine Translation Systems. In 13th Conference of the European Association for Machine Translation, pages 28-37, Barcelona.
21. S. Wold, A. Ruhe, H. Wold, and W. J. Dunn. 1984. The Co-variance Problem in Linear Regression. The Partial Least Squares (PLS) Approach to Generalized Inverses. SIAM Journal on Scientific Computing, 5:735-743.