Further meta-evaluation of machine translation

3rd Workshop on Statistical Machine Translation, WMT 2008 at the Annual Meeting of the Association for Computational Linguistics, ACL 2008

Volumn , Issue , 2008, Pages 70-106

  Callison Burch, Chris ^a   Fordyce, Cameron ^d   Koehn, Philipp ^b   Monz, Christof ^c   Schroeder, Josh ^b  

^a JOHNS HOPKINS UNIVERSITY   (United States)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

^c UNIVERSITY OF LONDON   (United Kingdom)

^d NONE

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; MACHINE TRANSLATION; NATURAL LANGUAGE PROCESSING SYSTEMS;

AUTOMATIC EVALUATION; HUMAN JUDGMENTS; HUNGARIANS; LANGUAGE PAIRS; LARGE-SCALES; MACHINE TRANSLATION SYSTEMS; MACHINE TRANSLATIONS; PAPER ANALYSIS; TRANSLATION QUALITY; TRANSLATION SYSTEMS;

QUALITY CONTROL;

EID: 85018089028     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (48)

1. Abhaya Agarwal and Alon Lavie. 2008. Meteor, M-BLEU and M-TER: Evaluation metrics for high-correlation with human rankings of machine translation output. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 115-118, Columbus, Ohio, June. Association for Computational Linguistics.
2. Joshua Albrecht and Rebecca Hwa. 2007a. A re-examination of machine learning approaches for sentence-level mt evaluation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL-2007), Prague, Czech Republic.
3. Joshua Albrecht and Rebecca Hwa. 2007b. Regression for sentence-level mt evaluation with pseudo references. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL-2007), Prague, Czech Republic.
4. Joshua Albrecht and Rebecca Hwa. 2008. The role of pseudo references in MT evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 187-190, Columbus, Ohio, June. Association for Computational Linguistics.
5. Abhishek Arun and Frank Keller. 2005. Lexicalization in crosslinguistic probabilistic parsing: The case of French. In Proceedings of ACL.
6. Amittai Axelrod, Mei Yang, Kevin Duh, and Katrin Kirchhoff. 2008. The University of Washington machine translation system for ACL WMT 2008. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 123-126, Columbus, Ohio, June. Association for Computational Linguistics.
7. Nguyen Bach, Qin Gao, and Stephan Vogel. 2008. Improving word alignment with language model based confidence scores. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 151-154, Columbus, Ohio, June. Association for Computational Linguistics.
8. Eckhard Bick. 2006. A constraint grammar-based parser for Spanish. In Proceedings of the 4th Workshop on Information and Human Language Technology (TIL-2006), Ribeiro Preto, Brazil.
9. Dan Bikel. 2002. Design of a multi-lingual, parallel-processing statistical parsing engine. In Proceedings of Second International Conference on Human Language Technology Research (HLT-02), San Diego, California.
10. Graeme Blackwood, Adrià de Gispert, Jamie Brunning, and William Byrne. 2008. European language translation with weighted finite state transducers: The CUED MT system for the 2008 ACL workshop on SMT. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 131-134, Columbus, Ohio, June. Association for Computational Linguistics.
11. Ondřej Bojar and Jan Hajič. 2008. Phrase-based and deep syntactic English-to-Czech statistical machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 143-146, Columbus, Ohio, June. Association for Computational Linguistics.
12. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL-2006), Trento, Italy.
13. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136-158, Prague, Czech Republic, June. Association for Computational Linguistics.
14. Jean Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254.
15. Daniel Déchelotte, Gilles Adda, Alexandre Allauzen, Hélène Bonneau-Maynard, Olivier Galibert, Jean-Luc Gauvain, Philippe Langlais, and François Yvon. 2008. Limsi's statistical translation systems for WMT'08. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 107-110, Columbus, Ohio, June. Association for Computational Linguistics.
16. Amit Dubey. 2005. What to do when lexicalization fails: parsing German with suffix analysis and smoothing. In Proceedings of ACL.
17. Loïc Dugast, Jean Senellart, and Philipp Koehn. 2008. Can we relearn an RBMT system? In Proceedings of the Third Workshop on Statistical Machine Translation, pages 175-178, Columbus, Ohio, June. Association for Computational Linguistics.
18. Kevin Duh. 2008. Ranking vs. regression in machine translation evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 191-194, Columbus, Ohio, June. Association for Computational Linguistics.
19. Christopher J. Dyer. 2007. The 'noisier channel': translation from morphologically complex languages. In Proceedings of the ACL-2007 Workshop on Statistcal Machine Translation (WMT-07), Prague, Czech Republic.
20. Andreas Eisele, Christian Federmann, Hervé Saint-Amand, Michael Jellinghaus, Teresa Herrmann, and Yu Chen. 2008. Using moses to integrate multiple rule-based machine translation engines into a hybrid system. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 179-182, Columbus, Ohio, June. Association for Computational Linguistics.
21. Jesus Gimenez and Lluis Marquez. 2008. A smorgasbord of features for automatic MT evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 195-198, Columbus, Ohio, June. Association for Computational Linguistics.
22. Kevin Gimpel and Noah A. Smith. 2008. Rich source-side context for statistical machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 9-17, Columbus, Ohio, June. Association for Computational Linguistics.
23. Greg Hanneman, Edmund Huber, Abhaya Agarwal, Vamshi Ambati, Alok Parlikar, Erik Peterson, and Alon Lavie. 2008. Statistical transfer systems for French-English and German-English machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 163-166, Columbus, Ohio, June. Association for Computational Linguistics.
24. Shyamsundar Jayaraman and Alon Lavie. 2005. Multiengine machine translation guided by explicit word matching. In Proceedings of the 10th Annual Conference of the European Association for Machine Translation, pages 143-152, Budapest, Hungary, May.
25. Maxim Khalilov, Adolfo Hernández H., Marta R. Costajussà, Josep M. Crego, Carlos A. Henríquez Q., Patrik Lambert, José A. R. Fonollosa, José B. Mariño, and Rafael E. Banchs. 2008. The TALP-UPC Ngrambased statistical machine translation system for ACLWMT 2008. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 127-130, Columbus, Ohio, June. Association for Computational Linguistics.
26. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings of NAACL 2006 Workshop on Statistical Machine Translation, New York, New York.
27. Philipp Koehn and Josh Schroeder. 2007. Experiments in domain adaptation for statistical machine translation. In Proceedings of the ACL-2007 Workshop on Statistcal Machine Translation (WMT-07), Prague, Czech Republic.
28. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the Human Language Technology Conference of the North American chapter of the Association for Computational Linguistics (HLT/NAACL-2003), Edmonton, Alberta.
29. Philipp Koehn, Nicola Bertoldi, Ondrej Bojar, Chris Callison-Burch, Alexandra Constantin, Brooke Cowan, Chris Dyer, Marcello Federico, Evan Herbst, Hieu Hoang, Christine Moran, Wade Shen, and Richard Zens. 2007. Open source toolkit for statistical machine translation: Factored translation models and confusion network decoding. CLSP Summer Workshop Final Report WS-2006, Johns Hopkins University.
30. Philipp Koehn, Abhishek Arun, and Hieu Hoang. 2008. Towards better machine translation quality for the German-English language pairs. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 139-142, Columbus, Ohio, June. Association for Computational Linguistics.
31. Alex Kulesza and Stuart M. Shieber. 2004. A learning approach to improving sentence-level MT evaluation. In Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation, Baltimore, MD, October 4-6.
32. J. R. Landis and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159-174.
33. LDC. 2005. Linguistic data annotation specification: Assessment of fluency and adequacy in translations. Revision 1.5.
34. Preslav Nakov. 2008. Improving English-Spanish statistical machine translation: Experiments in domain adaptation, sentence paraphrasing, tokenization, and recasing. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 147-150, Columbus, Ohio, June. Association for Computational Linguistics.
35. Vassilina Nikoulina and Marc Dymetman. 2008. Using syntactic coupling features for discriminating phrase-based translations (wmt-08 shared translation task). In Proceedings of the Third Workshop on Statistical Machine Translation, pages 159-162, Columbus, Ohio, June. Association for Computational Linguistics.
36. Attila Novák, László Tihanyi, and Gábor Prószéky. 2008. The MetaMorpho translation system. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 111-114, Columbus, Ohio, June. Association for Computational Linguistics.
37. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19-51, March.
38. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL-2003), Sapporo, Japan.
39. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-2002), Philadelphia, Pennsylvania.
40. Maja Popovic and Hermann Ney. 2007. Word error rates: Decomposition over POS classes and applications for error analysis. In Proceedings of ACL Workshop on Machine Translation, Prague, Czech Republic.
41. Mark Przybocki and Kay Peterson, editors. 2008. Proceedings of the 2008 NIST Open Machine Translation Evaluation Workshop. Arlington, Virginia, March.
42. Antti-Veikko Rosti, Spyros Matsoukas, and Richard Schwartz. 2007. Improved word-level system combination for machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL-2007), Prague, Czech Republic.
43. Antti-Veikko Rosti, Bing Zhang, Spyros Matsoukas, and Richard Schwartz. 2008. Incremental hypothesis alignment for building confusion networks with application to machine translation system combination. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 183-186, Columbus, Ohio, June. Association for Computational Linguistics.
44. Holger Schwenk, Jean-Baptiste Fouet, and Jean Senellart. 2008. First steps towards a general purpose French/English statistical machine translation system. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 119-122, Columbus, Ohio, June. Association for Computational Linguistics.
45. Sara Stymne, Maria Holmqvist, and Lars Ahrenberg. 2008. Effects of morphological analysis in translation between German and English. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 135-138, Columbus, Ohio, June. Association for Computational Linguistics.
46. John Tinsley, Yanjun Ma, Sylwia Ozdowska, and Andy Way. 2008. MaTrEx: The DCU MT system for WMT 2008. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 171-174, Columbus, Ohio, June. Association for Computational Linguistics.
47. Zhuoran Wang and John Shawe-Taylor. 2008. Kernel regression framework for machine translation: UCL system description for WMT 2008 shared translation task. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 155-158, Columbus, Ohio, June. Association for Computational Linguistics.
48. Zdenek Zabokrtsky, Jan Ptacek, and Petr Pajas. 2008. TectoMT: Highly modular MT system with tectogrammatics used as transfer layer. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 167-170, Columbus, Ohio, June. Association for Computational Linguistics.