On some pitfalls in automatic evaluation and significance testing for MT

Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Proceedings of the Workshop ACL 2005

Volumn , Issue , 2005, Pages 57-64

  Riezler, Stefan ^a   Maxwell, John T ^a  

^a PALO ALTO RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC EVALUATION; DISCRIMINATIVE RERANKING; DISCRIMINATORY POWER; EVALUATION METRICS; EXPERIMENTAL COMPARISON; F-SCORE; MT EVALUATIONS; RANDOMISATION; SIGNIFICANCE TESTING; STATISTICAL SIGNIFICANCE TEST;

RANDOM PROCESSES;

EID: 84886375939     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. Nancy Chinchor, Lynette Hirschman, and David D. Lewis. 1993. Evaluating message understanding systems: An analysis of the third message understanding conference (MUC-3). Computational Linguistics, 19(3):409–449.
2. Paul R. Cohen. 1995. Empirical Methods for Artificial Intelligence. The MIT Press, Cambridge, MA.
3. Richard Crouch, Ronald M. Kaplan, Tracy H. King, and Stefan Riezler. 2002. A comparison of evaluation metrics for a broad-coverage stochastic parser. In Proceedings of the”Beyond PARSEVAL” Workshop at the 3rd International Conference on Language Resources and Evaluation (LREC’02), Las Palmas, Spain.
4. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the ARPA Workshop on Human Language Technology.
5. Bradley Efron and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. Chapman and Hall, New York.
6. Kevin Knight and Yaser Al-Onaizan. 1999. A primer on finite-state software for natural language processing. Technical report, USC Information Sciences Institute, Marina del Rey, CA.
7. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the Human Language Technology Conference and the 3rd Meeting of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL’03), Edmonton, Cananda.
8. Philipp Koehn. 2002. Europarl: A multilingual corpus for evaluation of machine translation. Technical report, USC Information Sciences Institute, Marina del Rey, CA.
9. Philipp Koehn. 2004a. PHARAOH. a beam search decoder for phrase-based statistical machine translation models. user manual. Technical report, USC Information Sciences Institute, Marina del Rey, CA.
10. Philipp Koehn. 2004b. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP’04), Barcelona, Spain.
11. Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT/NAACL’04), Boston, MA.
12. Eric W. Noreen. 1989. Computer Intensive Methods for Testing Hypotheses. An Introduction. Wiley, New York.
13. Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL’02), Philadelphia, PA.
14. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51.
15. Franz Josef Och, Christoph Tillmann, and Hermann Ney. 1999. Improved alignment models for statistical machine translation. In Proceedings of the 1999 Conference on Empirical Methods in Natural Language Processing (EMNLP’99).
16. Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alex Fraser, Shankar Kumar, Libin Shen, David Smith, Ketherine Eng, Viren Jain, Zhen Jin, and Dragomir Radev. 2004. A smorgasbord of features for statistical machine translation. In Proceedings of the Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT/NAACL’04), Boston, MA.
17. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the Human Language Technology Conference and the 3rd Meeting of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL’03), Edmonton, Cananda.
18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. Technical Report IBM Research Division Technical Report, RC22176 (W0190-022), Yorktown Heights, N.Y.
19. Stefan Riezler and Alexander Vasserman. 2004. Incremental feature selection and `1 regularization for relaxed maximum-entropy modeling. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP’04), Barcelona, Spain.
20. Stefan Riezler, Tracy H. King, Richard Crouch, and Annie Zaenen. 2003. Statistical sentence condensation using ambiguity packing and stochastic disambiguation methods for lexical-functional grammar. In Proceedings of the Human Language Technology Conference and the 3rd Meeting of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL’03), Edmonton, Cananda.
21. Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In Proceedings of the Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT/NAACL’04), Boston, MA.
22. Sidney Siegel. 1988. Nonparametric Statistics for the Behavioral Sciences. Second Edition. MacGraw-Hill, Boston, MA.
23. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, Denver, CO.
24. Ying Zhang, Stephan Vogel, and Alex Waibel. 2004. Interpreting BLEU/NIST scores: How much improvement do we need to have a better system? In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal.