Randomized significance tests in machine translation

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2014, Pages 266-274

  Graham, Yvette ^a   Mathur, Nitika ^a   Baldwin, Timothy ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; NATURAL LANGUAGE PROCESSING SYSTEMS;

BOOTSTRAP RESAMPLING; GOLD STANDARDS; HUMAN EVALUATION; LANGUAGE PAIRS; LARGE-SCALES; MACHINE TRANSLATIONS; RANDOMISATION; RANDOMIZED METHOD; SIGNIFICANCE TEST; SIGNIFICANCE TESTING;

MACHINE TRANSLATION;

EID: 85083537618     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. S. Banerjee and A. Lavie. 2005. METEOR: An automatic metric for mt evaluation with improved correlation with human judgements. In Proc. Wkshp.
2. Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65? 73, Ann Arbor, MI. ACL.
3. C. Callison-Burch, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia. 2012. Findings of the 2012 Workshop on Statistical Machine Translation. In Proc. 7th Wkshp. Statistical Machine Translation, pages 10?51, Montreal, Canada. ACL.
4. J. H. Clark, C. Dyer, A. Lavie, and N. A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proc. of the 49th Annual Meeting of the Assoc. Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 176? 181, Portland, OR. ACL.
5. B. Efron and R. J. Tibshirani. 1993. An Introduction to the Bootstrap. Chapman & Hall, New York City, NY.
6. M. Galley and C. D. Manning. 2008. A simple and effective hierarchical phrase reordering model. In Proc. of the Conference on Empirical Methods in Natural Language Processing, pages 848?856, Edinburgh, Scotland. ACL.
7. U. Germann. 2003. Greedy decoding for statistical machine translation in almost linear time. In Proc. of the 2003 Conference of the North American Chapter of the Assoc. Computational Linguistics on Human Language Technology-Volume 1, pages 1?8, Edmonton, Canada. ACL.
8. Y. Graham, T. Baldwin, A. Moffat, and J. Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proc. 7th Linguistic Annotation Wkshp. & Interoperability with Discourse, pages 33?41, Sofia, Bulgaria. ACL.
9. S. Green, M. Galley, and C. D. Manning. 2010. Improved models of distortion cost for statistical machine translation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Assoc. Computational Linguistics, pages 867?875, Los Angeles, CA. ACL.
10. P. Koehn and C. Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings of the Workshop on Statistical Machine Translation, pages 102?121, New York City, NY. ACL.
11. P. Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of Empirical Methods in Natural Language Processing, pages 388?395, Barcelona, Spain. ACL.
12. S. Kumar and W. J. Byrne. 2004. Minimum Bayesrisk decoding for statistical machine translation. In HLT-NAACL, pages 169?176, Boston, MA. ACL.
13. C. Monz. 2011. Statistical machine translation with local language models. In Proc. of the Conference on Empirical Methods in Natural Language Processing, pages 869?879, Edniburgh, Scotland. ACL.
14. NIST. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics. Technical report.
15. E. W. Noreen. 1989. Computer intensive methods for testing hypotheses. Wiley, New York City, NY.
16. F. J. Och. 2003. Minimum error rate training in statistical machine translation. In Proc. 41st Ann. Meeting of the Assoc. Computational Linguistics, pages 160?167, Sapporo, Japan. ACL
17. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. 2002. A method for automatic evaluation of machine translation. In Proc. 40th Ann. Meeting of the Assoc. Computational Linguistics, pages 311?318, Philadelphia, PA. ACL.
18. S. Riezler and J. T. Maxwell. 2005. On some pitfalls in automatic evaluation and significance testing for mt. In Proc. of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 57?64, Ann Arbor, MI. ACL.
19. S. Riezler and J. T. Maxwell. 2006. Grammatical machine translation. In Proc. of the Main Conference on Human Language Technology Conference of the North American Chapter of the Assoc. Computational Linguistics, pages 248?255, New York City, NY. ACL.
20. M. Smucker, J. Allan, and B. Carterette. 2007. A comparison of statistical significance tests for information retrieval evaluation. In Proc. of the Sixteenth ACM Conference on Information and Knowledge Management (CIKM 2007), pages 623?632, Lisbon, Portugal. ACM.
21. M. Snover, B. Dorr, R. Scwartz, J. Makhoul, and L. Micciula. 2006. A study of translation error rate with targeted human annotation. In Proc. 7th Biennial Conf. of the Assoc. Machine Translaiton in the Americas, pages 223?231, Boston, MA. ACL.