Re-assessing the WMT2013 human evaluation with professional translators trainees

EAMT 2015 - Proceedings of the 18th Annual Conference of the European Association for Machine Translation

Volumn , Issue , 2015, Pages 161-168

  Vela, Mihaela ^a   van Genabith, Josef ^b  

^a SAARLAND UNIVERSITY   (Germany)

^b GERMAN RESEARCH CENTER FOR ARTIFICIAL INTELLIGENCE DFKI   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER AIDED LANGUAGE TRANSLATION;

DEVELOPMENT TEAMS; HUMAN EVALUATION; MACHINE TRANSLATIONS; SYSTEM RANKINGS;

COMPUTATIONAL LINGUISTICS;

EID: 85037163883     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (15)

1. Bojar, Ondřej, Christian Buck, Chris Callison-Burch, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Hervé Saint-Amand, Radu Soricut, and Lucia Specia, editors. 2013. Proceedings of the 8th Workshop on SMT. ACL.
2. Cohen, Jacob. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46, April.
3. Comelles, Elisabet and Jordi Atserias. 2014. Verta participation in the wmt14 metrics task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 368–375, Baltimore, Maryland, USA, June. Association for Computational Linguistics.
4. Denkowski, Michael and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
5. Doddington, George. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the 2nd International Conference on HLT, pages 138–145.
6. Federmann, Christian. 2012. Appraise: An open-source toolkit for manual evaluation of machine translation output. PBML, 98:25–35, 9.
7. Gonzàlez, Meritxell, Alberto Barrón-Cedeño, and Lluís Màrquez. 2014. Ipa and stout: Leveraging linguistic and source-based features for machine translation evaluation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 394–401, Baltimore, Maryland, USA, June. Association for Computational Linguistics.
8. Koehn, Philipp. 2012. Simulating human judgment in machine translation evaluation campaigns. In IWSLT, pages 179–184.
9. Levenshtein, Vladimir Iosifovich. 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10(8):707–710.
10. Lo, Chi-Kiu and Dekai Wu. 2011. MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles. In Proceedings of the 49th Annual Meeting of the ACL, pages 220–229.
11. Lo, Chi-kiu, Anand Karthik Tumuluru, and Dekai Wu. 2012. Fully automatic semantic MT evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 243–252, Montréal, Canada, June. Association for Computational Linguistics.
12. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the ACL, pages 311–318.
13. Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of AMTA, pages 223–231.
14. Snover, Matthew, Nitin Madnani, Bonnie Dorr, and Richard Schwartz. 2009. Fluency, adequacy, or HTER? Exploring different human judgments with a tunable MT metric. In Proceedings of the 4th Workshop on SMT, pages 259–268.
15. Tillmann, Christoph, Stephan Vogel, Hermann Ney, Alexander Zubiaga, and Hassan Sawaf. 1997. Accelerated DP based search for statistical translation. In Proceedings of the EUROSPEECH, pages 2667–2670.