PORT: A precision-order-recall MT evaluation metric for tuning

50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference

Volumn 1, Issue , 2012, Pages 930-939

  Chen, Boxing ^a   Kuhn, Roland ^a   Larkin, Samuel ^a  

^a NATIONAL RESEARCH COUNCIL CANADA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

BETTER PERFORMANCE; EVALUATION METRICS; EXPERIMENTAL CONDITIONS; EXTERNAL RESOURCES; HUMAN EVALUATION; HUMAN JUDGMENTS; LINGUISTIC RESOURCES; MACHINE TRANSLATIONS;

COMPUTATIONAL LINGUISTICS;

EID: 84875707768     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. S. Banerjee and A. Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of ACL Workshop on Intrinsic & Extrinsic Evaluation Measures for Machine Translation and/or Summarization.
2. A. Birch and M. Osborne. 2011. Reordering Metrics for MT. In Proceedings of ACL.
3. C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz and J. Schroeder. 2008. Further Meta-Evaluation of Machine Translation. In Proceedings of WMT.
4. C. Callison-Burch, M. Osborne, and P. Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In Proceedings of EACL.
5. C. Callison-Burch, P. Koehn, C. Monz, K. Peterson, M. Przybocki and O. Zaidan. 2010. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In Proceedings of WMT.
6. C. Callison-Burch, P. Koehn, C. Monz and O. Zaidan. 2011. Findings of the 2011 Workshop on Statistical Machine Translation. In Proceedings of WMT.
7. D. Cer, D. Jurafsky and C. Manning. 2010. The Best Lexical Metric for Phrase-Based Statistical MT System Optimization. In Proceedings of NAACL.
8. Y. S. Chan and H. T. Ng. 2008. MAXSIM: A maximum similarity metric for machine translation evaluation. In Proceedings of ACL.
9. B. Chen and R. Kuhn. 2011. AMBER: A Modified BLEU, Enhanced Ranking Metric. In: Proceedings of WMT. Edinburgh, UK. July.
10. D. Chiang, S. DeNeefe, Y. S. Chan, and H. T. Ng. 2008. Decomposability of translation metrics for improved evaluation and efficient algorithms. In Proceedings of EMNLP, pages 610-619.
11. M. Denkowski and A. Lavie. 2010. Meteor-next and the meteor paraphrase tables: Improved evaluation support for five target languages. In Proceedings of the Joint Fifth Workshop on SMT and MetricsMATR, pages 314-317.
12. G. Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of HLT.
13. J. L. Fleiss. 1971. Measuring nominal scale agreement among many raters. In Psychological Bulletin, Vol. 76, No. 5 pp. 378-382.
14. Y. He, J. Du, A. Way and J. van Genabith. 2010. The DCU dependency-based metric in WMTMetricsMATR 2010. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 324-328.
15. H. Isozaki, T. Hirao, K. Duh, K. Sudoh, H. Tsukada. 2010. Automatic Evaluation of Translation Quality for Distant Language Pairs. In Proceedings of EMNLP.
16. M. Kendall. 1938. A New Measure of Rank Correlation. In Biometrika, 30 (1-2), pp. 81-89.
17. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin and E. Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of ACL, pp. 177-180, Prague, Czech Republic.
18. A. Lavie and M. J. Denkowski. 2009. The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23.
19. C. Liu, D. Dahlmeier, and H. T. Ng. 2010. TESLA: Translation evaluation of sentences with linearprogramming- based analysis. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 329-334.
20. C. Liu, D. Dahlmeier, and H. T. Ng. 2011. Better evaluation metrics lead to better machine translation. In Proceedings of EMNLP.
21. C. Lo and D. Wu. 2011. MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles. In Proceedings of ACL.
22. F. J. Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of ACL-2003. Sapporo, Japan.
23. F. J. Och and H. Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. In Computational Linguistics, 29, pp. 19-51.
24. S. Pado, M. Galley, D. Jurafsky, and C.D. Manning. 2009. Robust machine translation evaluation with entailment features. In Proceedings of ACL-IJCNLP.
25. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL.
26. K. Parton, J. Tetreault, N. Madnani and M. Chodorow. 2011. E-rating Machine Translation. In Proceedings of WMT.
27. M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas.
28. M. Snover, N. Madnani, B. Dorr, and R. Schwartz. 2009. Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece.
29. C. Spearman. 1904. The proof and measurement of association between two things. In American Journal of Psychology, 15, pp. 72-101.
30. S. Vogel, H. Ney, and C. Tillmann. 1996. HMM based word alignment in statistical translation. In Proceedings of COLING.