Interpreting BLEU/NIST scores: How much improvement do we need to have a better system?

Proceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004

Volumn , Issue , 2004, Pages 2051-2054

  Zhang, Ying ^a   Vogel, Stephan ^a   Waibel, Alex ^a  

^a Carnegie Mellon University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC EVALUATION; CONFIDENCE INTERVAL; MACHINE TRANSLATION SYSTEMS;

EID: 84921700653     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (8)

1. Culy, Christopher & Riehemann, Susanne Z. (2003). The Limits of N-Gram Translation Evaluation Metrics. In Proceedings of the Ninth Machine Translation Summit. New Orleans, Louisiana, USA.
2. Efron, B. and R. Tibshirani (1986). Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy, Statistical Science 1, p. 54-77.
3. Efron, Bradley and Rob, Tibshirani (1993). An Introduction to Bootstrap. Chapman & Hall, New York.
4. NIST Report (2002) Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics. http://www.nist.gov/speech/tests/mt/doc/ngramstudy.pdf
5. Papineni, Kishore & Roukos, Salim et al. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 20th Annual Meeting of the Association for Computational Linguistics.
6. Pepescu-Belis, Andrei (2003). An Experiment in Comparative Evaluation: Humans vs. Computers. In Proceedings of the Ninth Machine Translation Summit. New Orleans, Louisiana, USA.
7. Van Slype, Georges. (1979). Critical Study of Methods for Evaluating the Quality of Machine Translation, European Commission/Directorate for General Scientific and Technical Information Management (DG XIII), BR 19142. http://issco-www.unige.ch/projects/isle/van-slype.pdf
8. White, J. S., T. A. O'Connell, & F. E. O'Mara. (1994). Advanced Research Projects Agency Machine Translation Program: 3Q94. Proceedings of the November 1994 Meeting.