MEANT at WMT 2013: A tunable, accurate yet inexpensive semantic frame based MT evaluation metric

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2013, Pages 422-428

  Lo, Chi Kiu ^a   Wu, Dekai ^a  

^a HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS;

EVALUATION METRICS; FRAME-BASED; METRIC EVALUATION; MT EVALUATIONS; SEMANTIC FRAMES; SEMANTIC ROLES; SIMPLE++; SUPERVISED METHODS; TUNABLES; UNSUPERVISED APPROACHES;

SEMANTICS;

EID: 85122648956     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan, June 2005.
2. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. Re-evaluating the role of BLEU in Machine Translation Research. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL-06), pages 249-256, 2006.
3. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. (Meta-) evaluation of Machine Translation. In Proceedings of the 2nd Workshop on Statistical Machine Translation, pages 136-158, 2007.
4. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. Further Meta-evaluation of Machine Translation. In Proceedings of the 3rd Workshop on Statistical Machine Translation, pages 70-106, 2008.
5. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. Findings of the 2012 Workshop on Statistical Machine Translation. In Proceedings of the 7th Workshop on Statistical Machine Translation (WMT 2012), pages 10-51, 2012.
6. Julio Castillo and Paula Estrella. Semantic Textual Similarity for MT evaluation. In Proceedings of the 7th Workshop on Statistical Machine Translation (WMT 2012), pages 52-58, 2012.
7. Boxing Chen, Roland Kuhn, and George Foster. Improving AMBER, an MT Evaluation Metric. In Proceedings of the 7th Workshop on Statistical Machine Translation (WMT 2012), pages 59-63, 2012.
8. George Doddington. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the 2nd International Conference on Human Language Technology Research, pages 138-145, San Diego, California, 2002.
9. Jesús Giménez and Lluís Màrquez. Linguistic features for automatic evaluation of heterogenous MT systems. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 256-264, Prague, Czech Republic, June 2007.
10. Jesús Giménez and Lluís Màrquez. A smorgasbord of features for automatic MT evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 195-198, Columbus, Ohio, June 2008.
11. Philipp Koehn and Christof Monz. Manual and Automatic Evaluation of Machine Translation between European Languages. In Proceedings of the Workshop on Statistical Machine Translation (WMT-06), pages 102-121, 2006.
12. Patrik Lambert, Jesús Giménez, Marta R Costajussá, Enrique Amigó, Rafael E Banchs, Lluıs Márquez, and JAR Fonollosa. Machine Translation system development based on human likeness. In Spoken Language Technology Workshop, 2006. IEEE, pages 246-249. IEEE, 2006.
13. Gregor Leusch, Nicola Ueffing, and Hermann Ney. CDer: Efficient MT Evaluation Using Block Movements. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL-06), 2006.
14. Chi-kiu Lo and Dekai Wu. MEANT: An Inexpensive, High-Accuracy, Semi-Automatic Metric for Evaluating Translation Utility based on Semantic Roles. In Proceedings of the Joint conference of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT-11), 2011.
15. Chi-kiu Lo and Dekai Wu. SMT vs. AI redux: How semantic frames evaluate MT more accurately. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI-11), 2011.
16. Chi-kiu Lo and Dekai Wu. Unsupervised vs. supervised weight estimation for semantic MT evaluation metrics. In Proceedings of the 6th Workshop on Syntax and Structure in Statistical Translation (SSST-6), 2012.
17. Chi-kiu Lo and Dekai Wu. Can informal genres be better translated by tuning on automatic semantic metrics? In Proceedings of the 14th Machine Translation Summit (MTSummit-XIV), 2013.
18. Chi-kiu Lo, Anand Karthik Tumuluru, and Dekai Wu. Fully Automatic Semantic MT Evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation (WMT2012), 2012.
19. Chi-kiu Lo, Karteek Addanki, Markus Saers, and Dekai Wu. Improving machine translation by training against an automatic semantic frame based evaluation metric. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL-13), 2013.
20. Rada Mihalcea, Courtney Corley, and Carlo Strapparava. Corpus-based and knowledge-based measures of text semantic similarity. In Proceedings of the national conference on artificial intelligence, volume 21, page 775. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2006.
21. Sonja Nießen, Franz Josef Och, Gregor Leusch, and Hermann Ney. A Evaluation Tool for Machine Translation: Fast Evaluation for MT Research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation (LREC-2000), 2000.
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, July 2002.
23. Sameer Pradhan, Wayne Ward, Kadri Hacioglu, James H. Martin, and Dan Jurafsky. Shallow Semantic Parsing Using Support Vector Machines. In Proceedings of the 2004 Conference on Human Language Technology and the North American Chapter of the Association for Computational Linguistics (HLT-NAACL-04), 2004.
24. Miguel Rios, Wilker Aziz, and Lucia Specia. Tine: A metric to assess MT adequacy. In Proceedings of the Sixth Workshop on Statistical Machine Translation (WMT-2011), pages 116-122, 2011.
25. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA-06), pages 223-231, Cambridge, Massachusetts, August 2006.
26. Xingyi Song and Trevor Cohn. Regression and Ranking based Optimisation for Sentence Level Machine Translation Evaluation. In Proceedings of the 6th Workshop on Statistical Machine Translation (WMT 2011), pages 123-129, 2011.
27. Anand Karthik Tumuluru, Chi-kiu Lo, and Dekai Wu. Accuracy and robustness in measuring the lexical similarity of semantic role fillers for automatic semantic mt evaluation. In Proceeding of the 26th Pacific Asia Conference on Language, Information, and Computation (PACLIC-26), 2012.
28. Mengqiu Wang and Christopher D. Manning. SPEDE: Probabilistic Edit Distance Metrics for MT Evaluation. In Proceedings of the 7th Workshop on Statistical Machine Translation (WMT 2012), pages 76-83, 2012.