Combining confidence estimation and reference-based metrics for segment-level MT evaluation

AMTA 2010 - 9th Conference of the Association for Machine Translation in the Americas

Volumn , Issue , 2010, Pages

  Specia, Lucia ^a   Giménez, Jesús ^b  

^a UNIVERSITY OF WOLVERHAMPTON   (United Kingdom)

^b UNIVERSITAT POLITÈCNICA DE CATALUNYA   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

CONFIDENCE ESTIMATION; END USERS; GOOD CORRELATIONS; HUMAN ANNOTATIONS; LEARNING MECHANISM; MACHINE TRANSLATIONS; MT EVALUATIONS; MULTIPLE SYSTEMS; SYSTEM OUTPUT;

LEARNING ALGORITHMS; SEMANTICS; SOFTWARE AGENTS;

INFORMATION THEORY;

EID: 84857530287     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (38)

1. Joshua Albrecht and Rebecca Hwa. 2007a. A re-examination of machine learning approaches for sentence-level mt evaluation. In 45th Meeting of the Association for Computational Linguistics, pages 880-887, Prague.
2. Joshua Albrecht and Rebecca Hwa. 2007b. Regression for sentence-level mt evaluation with pseudo references. In 45th Meeting of the Association for Computational Linguistics, pages 296-303, Prague.
3. Joshua Albrecht and Rebecca Hwa. 2008. The role of pseudo references in MT evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 187-190.
4. J. Blatz, E. Fitzgerald, G. Foster, S. Gandrabur, C. Goutte, A. Kulesza, A. Sanchis, and N. Ueffing. 2004. Confidence Estimation for Machine Translation. In 20th Coling, pages 315-321, Geneva.
5. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In 3rd Workshop on Statistical Machine Translation, pages 70-106, Columbus.
6. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In 4th Workshop on Statistical Machine Translation, pages 1-28.
7. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint FifthWorkshop on Statistical Machine Translation and MetricsMATR, pages 17-53.
8. Chih-Chung Chang and Chih-Jen Lin, 2001. LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm.
9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46, April.
10. Simon Corston-Oliver, Michael Gamon, and Chris Brockett. 2001. A Machine Learning Approach to the Automatic Evaluation of Machine Translation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (ACL), pages 140-147.
11. Michael Denkowski and Alon Lavie. 2010. Meteor-next and the meteor paraphrase tables: Improved evaluation support for five target languages. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 339-342, July.
12. George Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Cooccurrence Statistics. In 2nd Conference on Human Language Technology Research, pages 138-145, San Diego.
13. Michael Gamon, Anthony Aue, and Martine Smets. 2005. Sentence-level MT evaluation without reference translations: beyond language modeling. In 10th Meeting of the European Association for Machine Translation, Budapest.
14. Jesús Giménez and Lluís Mà rquez. 2007. Linguistic Features for Automatic Evaluation of Heterogeneous MT Systems. In Proceedings of the Workshop on Statistical Machine Translation, pages 256-264.
15. Jesús Giménez and Lluís Màrquez. 2010a. ASIYA: An Open Toolkit for Automatic Machine Translation Evaluation. To Appear in The Prague Bulletin of Mathematical Linguistics.
16. Jesús Giménez and Lluís Màrquez. 2010b. Linguistic Features for Automatic MT Evaluation. To Appear in Machine Translation.
17. Howard Johnson, F. Sadat, George Foster, Roland Kuhn, Michael Simard, Eric Joanis, and S. Larkin. 2006. Portage with Smoothed Phrase Tables and Segment Choice Models. In Workshop on Statistical Machine Translation, pages 134-137, New York.
18. Matti Kääriäinen. 2009. Sinuhe - Statistical Machine Translation using a Globally Trained Conditional Exponential Family Translation Model. In Conference on Empirical Methods in Natural Language Processing, pages 1027-1036, Singapore.
19. Alex Kulesza and Stuart Shieber. 2004. A learning approach to improving sentence-level MT evaluation. In 10th International Conference on Theoretical and Methodological Issues in Machine Translation, Baltimore.
20. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In 2nd Workshop on Statistical Machine Translation, pages 228-231, Prague.
21. Chin-Yew Lin and Franz Josef Och. 2004a. Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL).
22. Chin-Yew Lin and Franz Josef Och. 2004b. ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In Proceedings of the 20th International Conference on Computational Linguistics (COLING).
23. Ding Liu and Daniel Gildea. 2005. Syntactic Features for Evaluation of Machine Translation. In Proceedings ofWorkshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pages 25-32.
24. Ding Liu and Daniel Gildea. 2007. Source-Language Features and Maximum Correlation Training for Machine Translation Evaluation. In Proceedings of the 2007 Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 41-48.
25. I. Dan Melamed, Ryan Green, and Joseph P. Turian. 2003. Precision and Recall of Machine Translation. In Proceedings of the Joint Conference on Human Language Technology and the North American Chapter of the Association for Computational Linguistics (HLTNAACL).
26. Sonja Nießen, Franz Josef Och, Gregor Leusch, and Hermann Ney. 2000. An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation (LREC).
27. Franz Josef Och and Herman Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29(1):19-51. (Pubitemid 37049767)
28. Sebastian Padó, Michel Galley, Dan Jurafsky, and Christopher D. Manning. 2009. Textual entailment features for machine translation evaluation. In 4th Workshop on Statistical Machine Translation, pages 37-41, Athens.
29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Morristown.
30. Chris Quirk. 2004. Training a Sentence-Level Machine Translation Confidence Measure. In 4th Conference on Language Resources and Evaluation, pages 825-828, Lisbon.
31. Craig Saunders. 2008. Application of Markov Approaches to Statistical Machine Translation. Technical report, SMART Project Deliverable 2.2.
32. Michel Simard, Nicola Cancedda, Bruno Cavestro, Marc Dymetman, Eric Gaussier, Cyril Goutte, Kenji Yamada, Philippe Langlais, and Arne Mauser. 2005. Translating with Non-contiguous Phrases. In Conference on Empirical Methods in Natural Language, pages 755-762, Vancouver.
33. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006a. A Study of Translation Edit Rate with Targeted Human Annotation. In 7th Conference of the Association for Machine Translation in the America, pages 223-231, Cambridge, MA.
34. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006b. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA), pages 223-231.
35. Lucia Specia, Marco Turchi, Nicola Cancedda, Marc Dymetman, and Nello Cristianini. 2009. Estimating the Sentence-Level Quality of Machine Translation Systems. In 13th Conference of the European Association for Machine Translation, pages 28-37, Barcelona.
36. Lucia Specia, Nicola Cancedda, and Marc Dymetman. 2010a. A dataset for assessing machine translation evaluation metrics. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10), Valletta, Malta.
37. Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010b. Machine translation evaluation versus quality estimation. Machine Translation, pages 1-12.
38. Christoph Tillmann, Stefan Vogel, Hermann Ney, A. Zubiaga, and H. Sawaf. 1997. Accelerated DP based Search for Statistical Translation. In Proceedings of European Conference on Speech Communication and Technology.