The Feasibility of HMEANT as a Human MT Evaluation Metric

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2013, Pages 52-61

  Birch, Alexandra ^a   Haddow, Barry ^a   Germann, Ulrich ^a   Nadejde, Maria ^a   Buck, Christian ^a   Koehn, Philipp ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

MACHINE TRANSLATION;

AUTOMATIC METRICS; EVALUATION METRICS; HUMAN EVALUATION; INTRINSIC PROPERTY; LANGUAGE PAIRS; MACHINE TRANSLATIONS; MT EVALUATIONS; SEMANTIC ROLES;

SEMANTICS;

EID: 85037071347     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Abend, Omri and Ari Rappoport. 2013. “Universal Conceptual Cognitive Annotation (UCCA).” Proceedings of ACL.
2. Bojar, Ondrej, Miloš Ercegovčević, Martin Popel, and Omar Zaidan. 2011. “A Grain of Salt for the WMT Manual Evaluation.” Proceedings of the Sixth Workshop on Statistical Machine Translation, 1-11. Edinburgh, Scotland.
3. Bojar, Ondrej and Dekai Wu. 2012. “Towards a Predicate-Argument Evaluation for MT.” Proceedings of SSST, 30-38.
4. Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. “(Meta-) evaluation of machine translation.” Proceedings of the Second Workshop on Statistical Machine Translation, 136-158. Prague, Czech Republic.
5. Callison-Burch, Chris, Philipp Koehn, Christof Monz, and Omar F Zaidan. 2011. “Findings of the 2011 workshop on statistical machine translation.” Proceedings of the Sixth Workshop on Statistical Machine Translation, 22-64.
6. Dreyer, Markus and Daniel Marcu. 2012. “Hyter: Meaning-equivalent semantics for translation evaluation.” Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 162-171. Montréal, Canada.
7. Germann, Ulrich. 2008. “Yawat: Yet Another Word Alignment Tool.” Proceedings of the ACL-08: HLT Demo Session, 20-23. Columbus, Ohio.
8. Giménez, Jesús and Lluís Màrquez. 2007. “Linguistic features for automatic evaluation of heterogenous mt systems.” Proceedings of the Second Workshop on Statistical Machine Translation, StatMT'07, 256-264. Stroudsburg, PA, USA.
9. Hutchins, W. J. and H. L. Somers. 1992. An introduction to machine translation. Academic Press New York.
10. Jones, Bevan, Jacob Andreas, Daniel Bauer, Karl Moritz Hermann, and Kevin Knight. 2012. “Semantics-based machine translation with hyperedge replacement grammars.” Proceedings of COLING.
11. Koponen, Maarit. 2012. “Comparing human perceptions of post-editing effort with post-editing operations.” Proceedings of the Seventh Workshop on Statistical Machine Translation, 181-190. Montréal, Canada.
12. Lavie, Alon and Michael Denkowski. 2009. “The METEOR metric for automatic evaluation of machine translation.” Machine Translation.
13. Linguistic Data Consortium. 2002. “Linguistic data annotation specification: Assessment of fluency and adequacy in Chinese-English translation.” http://projects.ldc.upenn.edu/TIDES/Translation/TranAssessSpec.pdf.
14. Lo, Chi-kiu, Anand Karthik Tumuluru, and Dekai Wu. 2012. “Fully automatic semantic MT evaluation.” Proceedings of WMT, 243-252.
15. Lo, Chi-kiu and Dekai Wu. 2010. “Evaluating machine translation utility via semantic role labels.” Proceedings of LREC, 2873-2877.
16. Lo, Chi-kiu and Dekai Wu. 2011a. “MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via semantic frames.” Proceedings of ACL, 220-229.
17. Lo, Chi-kiu and Dekai Wu. 2011b. “Structured vs. flat semantic role representations for machine translation evaluation.” Proceedings of SSST, 10-20.
18. Lo, Chi-kiu and Dekai Wu. 2012. “Unsupervised vs. supervised weight estimation for semantic MT evaluation metrics.” Proceedings of SSST, 49-56.
19. Lopez, Adam. 2012. “Putting human assessments of machine translation systems in order.” Proceedings of WMT, 1-9.
20. NIST. 2005. “The 2005 NIST machine translation evaluation plan (MT-05).” http://www.itl.nist.gov/iad/mig/tests/mt/2005/doc/mt05_evalplan.v1.1.pdf.
21. Palmer, Martha, Daniel Gildea, and Paul Kings-bury. 2005. “The proposition bank: An annotated corpus of semantic roles.” Computational Linguistics, 31(1):71-106.
22. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. “BLEU: a method for automatic evaluation of machine translation.” Proceedings of the Association for Computational Linguistics, 311-318. Philadelphia, USA.
23. Przybocki, Mark, Kay Peterson, Sébastien Bronsart, and Gregory Sanders. 2009. “The NIST 2008 metrics for machine translation challengeoverview, methodology, metrics, and results.” Machine Translation, 23(2):71-103.
24. Snover, Matthew, Nitin Madnani, Bonnie Dorr, and Richard Schwartz. 2009a. “Fluency, adequacy, or HTER? exploring different human judgments with a tunable MT metric.” Proceedings of the Workshop on Statistical Machine Translation at the Meeting of the European Chapter of the Association for Computational Linguistics (EACL-2009). Athens, Greece.
25. Snover, Matthew, Nitin Madnani, Bonnie Dorr, and Richard Schwartz. 2009b. “TER-plus: paraphrase, semantic, and alignment enhancements to translation edit rate.” Machine Translation.
26. Tumuluru, Anand Karthik, Chi-kiu Lo, and Dekai Wu. 2012. “Accuracy and robustness in measuring the lexical similarity of semantic role fillers for automatic semantic MT evaluation.” Proceedings of PACLIC, 574-581.
27. Weaver, Warren. 1955. “Translation.” William N. Locke and Andrew D. Booth (eds.), Machine Translation of Languages; Fourteen Essays, 15-23. Cambridge, MA: MIT Press. Reprint of a memorandum written in 1949.