How to evaluate machine translation

Benjamins Translation Library

Volumn 35, Issue , 2003, Pages 211-244

  White, John S ^a  

^a PRC Northrop Grumman Information Technology   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84857623162     PISSN: 09297316     EISSN: None     Source Type: Book Series    
DOI: 10.1075/btl.35.16whi     Document Type: Chapter

References (30)

1. ALPAC (1966) Language and Machines: Computers in Translation and Linguistics, Report by the Automatic Language Processing Advisory Committee (ALPAC), Publication 1416, National Academy of Sciences National Research Council, Washington, D. C.
2. ARPA (Advanced Research Projects Agency) (1993) Human Language Technology: Proceedings of a Workshop. San Francisco: Morgan Kaufmann.
3. Arnold, Doug, Louisa Sadler, and R. Lee Humphreys (1993) “Evaluation: an Assessment”, Machine Translation 8, 1-24.
4. Campbell, D. T. and J. C. Stanley (1963) Experimental and Quasi-Experimental Designs for Research. Skokie, Ill.: Rand McNally.
5. Dostert, B. (1973) User's Evaluation of Machine Translation, Georgetown MT System, 1963-1973, Rome Air Development Center Report AD-768 451, Texas A&M University.
6. EAGLES (Expert Advisory Groups in Language Engineering Standards) (1996) EAGLES Evaluation of Natural Language Processing Systems, Center for Sprogteknologi, Copenhagen, Denmark.
7. ELRA (European Language Resources Association) (1998) First International Conference on Language Resources and Evaluation (LREC 1998): Proceedings, Granada, Spain.
8. Farwell, David, Laurie Gerber and Eduard Hovy (eds) Machine Translation and the Information Soup: Third Conference of the Association for Machine Translation in the Americas AMTA'98. Berlin: Springer.
9. Flanagan, Mary A. (1994) “Error Classiªcation for MT Evaluation”, in Technology Partnerships for Crossing the Language Barrier: Proceedings of the First Conference of the Association for Machine Translation in the Americas, Columbia, Maryland, pages 65-71.
10. Heinisz-Dostert, Bozena, R. Ross Macdonald and Michael Zarechnak (1979) Machine Translation. The Hague: Mouton.
11. Helmreich, Stephen and David Farwell (1998) “Translation DiŸerences and Pragmatics-based MT”, Machine Translation 13, 17-39.
12. Hirschman, Lynette (1998) “Language Understanding Evaluations: Lessons Learned from MUC and ATIS”, in ELRA (1998), pp. 117-122.
13. Hirschman, L., F. Reeder, J. Burger and K. Miller (2000) “Name Translation as a Machine Translation Evaluation Task”, in LREC-2000: Second International Conference on Language Resources and Evaluation, Workshop Proceedings: Evaluation of Machine Translation, Athens, pp. 21-28.
14. Hutchins, W. John (1996) “ALPAC: The (In)Famous Report”, MT News International 14 (June 1996), 9-12.
15. Hutchins, W. John (1997) “From First Conception to First Demonstration: The Nascent Years of Machine Translation, 1947-1954. A Chronology”, Machine Translation 12, 195-252.
16. Hutchins, W. John and Harold L. Somers (1992) An Introduction to Machine Translation. London: Academic Press.
17. JEIDA (Japanese Electronic Industry Development Association) (1992) JEIDA Methodology and Criteria on Machine Translation Evaluation. Tokyo: JEIDA.
18. Jordan, Pamela W., Bonnie J. Dorr and John W. Benoit (1993) “A First-Pass Approach for Evaluating Machine Translation Systems”, Machine Translation 8, 49-58.
19. Lange, Elke and Laurie Gerber (1992) “Internal Evaluation: Quality Analysis, an Internal Evaluation Tool at SYSTRAN”, in MT Evaluation: Basis for Future Directions. Proceedings of a workshop sponsored by the National Science Foundation, San Diego, California, pages 55-56.
20. Minnis, Stephen (1993) “Constructive Machine Translation Evaluation”, Machine Translation 8, 67-76.
21. Nagao, Makoto, Jun-ichi Tsujii and Jun-ichi Nakamura (1985) “The Japanese Government Project for Machine Translation”, Computational Linguistics 11, 91-109.
22. Nomura, Hirosato and Hitoshi Isahara (1992) “JEIDA's Criteria on Machine Translation Evaluation”, in International Symposium on Natural Language Understanding and AI (NLU & AI) as a part of International Symposia on Information Sciences (ISKIT'92), Iizuka, Fukuoka, Japan, pages107-117.
23. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu (2001) Bleu: a Method for Automatic Evaluation of Machine Translation, IBM Research Technical Report, Thomas J. Watson Research Center, Yorktown Heights, NY. [available on the Internet]
24. Sinaiko, Harold W. (1979) “Measurement of Usefulness by Performance Test”, in van Slype (1979), page 91.
25. Taylor, Kathryn and John S. White (1998) “Predicting What MT is Good for: User Judgments and Task Performance”, in Farwell et al. (1998), pages 364-374.
26. van Slype, Georges (1979) Critical Methods for Evaluating the Quality of Machine Translation, Report BR 19142 prepared for the European Commission Directorate General Scientiªc and Technical Information and Information Management, Bureau Marcel van Dijk, Bruxelles.
27. Vasconcellos, Muriel (1992) “Panel: Apples, Oranges, or Kiwis? Criteria for the Comparison of MT systems”, in MT Evaluation: Basis for Future Directions. Proceedings of a workshop sponsored by the National Science Foundation, San Diego, California, pages 37-50.
28. White, John S. (1995) “Approaches to Black Box MT Evaluation”, in MT Summit V Proceedings, Luxembourg, [no page numbers].
29. White, John S., Theresa O'Connell and Francis O'Mara (1994) “The ARPA MT Evaluation Methodologies: Evolution, Lessons, and Future Approaches”, in Technology Partnerships for Crossing the Language Barrier: Proceedings of the First Conference of the Association for Machine Translation in the Americas, Columbia, Maryland, pages 193-205.
30. White, John S. and Theresa A. O'Connell (1996) “Adaptation of the DARPA Machine Translation Evaluation Paradigm to End-to-End Systems”, in Expanding MT Horizons: Proceedings of the Second Conference of the Association for Machine Translation in the Americas, Montreal, Quebec, pages 106-114.