Task-based evaluation of machine translation (MT) engines: Measuring how well people extract who, when, where-type elements in MT output

EAMT-2006 - 11th Annual Conference of the European Association for Machine Translation

Volumn , Issue , 2006, Pages 203-212

  Voss, Clare R ^a   Tate, Calandra R ^a,b  

^a ARL   (United States)

^b University of Maryland   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC METRICS; ENGLISH WORD; EXTRACTING INFORMATION; LONG-TERM GOALS; MACHINE TRANSLATIONS; MT EVALUATIONS; RESPONSE RATE; RULE BASED; STATISTICAL MODELS; TASK-BASED;

EXPERIMENTS; INFORMATION THEORY; STATISTICS; SURVEYS;

ENGINES;

EID: 84857588263     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. Banerjee, S. & Lavie, A. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlations with Human Judgments. In Proceedings of Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization at the 43th Annual Meeting of the Association of Computational Linguistics (ACL-2005). Ann Arbor, Michigan.
2. Calison-Burch et al. 2006. Re-evaluating the Role of BLEU in Machine Translation Research. In Proceedings of the European Association for Computational Linguistics (EACL-2006). Trento, Italy.
3. Coughlin, D. (2003). Correlating Automated and Human Assessments of Machine Translation Quality. In Proceedings of MT Summit IX. New Orleans, LA. pp. 63-70.
4. Church, K., & Hovy, E. (1993). Good Applications for Crummy Machine Translation. Machine Translation, 8.
5. Clifford, R., Granoien, N., Jones, D., Shen, W., & Wein-stein, C. (2004). The Effect of Text Difficulty on Machine Translation Performance. In Proceedings of Language Resources and Evaluation Conference (LREC-2004). Lisbon, Portugal.
6. Fisher, F., Schlesiger, C., Decrozant, L., Zuba, R., Holland, M., & Voss, C.R. (1999). Searching and Translating Arabic Documents on a Mobile Platform. In Proceedings of the Advanced Information Processing and Analysis Conference (AIPA-99). McLean, VA.
7. Holland, R. (2005). Embedded Machine Translation Prototypes at MITRE. Presentation at UMIACS Computational Linguistics Colloquium. University of Maryland, College Park. July 27, 2005.
8. Laoudi, J., Tate, C., & Voss, C. (2006). Task-based MT Evaluation: From Who/When/Where Extraction to Event Understanding. In Proceedings of the Language Resources and Evaluation Conference (LREC-2006). Genoa, Italy.
9. Lavie, A., Sagae, K., & Jayaraman, J. (2004). The Significance of Recall in Automatic Metrics for MT Evaluation. In Proceedings of the 6th Conference of the Association for Machine Translation in the Americas (AMTA-2004). Washington, DC.
10. Liu, D. & Guildea, D. (2005). Syntactic Features for Evaluation of Machine Translation. In Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Ann Arbor, MI.
11. Levin, L., Bartlog, B., Llitjos, A., Gates, D., Lavie, A., Wallace, D., Watanabe, T., & Woszczyna, M. (2000). Lessons learned from a task-based evaluation of speech-to-speech machine translation. Language Resources and Evaluation Conference(LREC). Athens, Greece.
12. Melamed, I.D., Green, R., & Turian, J.P. (2003). Precision and Recall of Machine Translation. In Proceedings of HLT/NAACL. Edmonton, Canada.
13. Papineni, K., Roukos, S., Ward, T., & Zhu, W. (2002). BLEU: a method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the ACL. Philadelphia, PA.
14. Resnik, P. (1997). Evaluating Multilingual Gisting of Web Pages. Proceedings of the AAAI Symposium on Natural Language Processing for the World Wide Web. Stanford, CA.
15. Snover, M., Dorr, B.J., Schwartz, R., Makhoul, J., Micciulla, L., & Weischedel, R. (2005). A Study of Translation Error Rate with Targeted Human Annotation. LAMP-TR-126. U. of Maryland, College Park.
16. Spaerck-Jones, K., & Gallier, J.R. (1996). Evaluating Natural Language Processing Systems: An Analysis and Review. Springer, Berlin.
17. Tate, C.R. (2005). Evaluating Machine Translation Output & Predicting Its Utility. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Ann Arbor, MI.
18. Tate, C.R. (2003). Task-based MT Evaluation: Tackling Software, Experimental Design, & Statistical Models. Proceedings of Workshop on Machine Translation Evaluation Towards Systematizing MT Evaluation. MT Summit IX. New Orleans, LA.
19. Taylor, K., & White, J. (1998). Predicting What MT is Good for: User Judgements and Task Performance. Proceedings of AMTA, pages 364-373. Springer, Berlin.
20. Vanni, M., Voss, C.R., & Tate, C.R. (2004). Ground Truth, Reference Truth & "Omniscient Truth" - Parallel Phrases in Parallel Texts for MT Evaluation. Proceedings of LREC. Lisbon, Portugal.
21. Voss, C.R. (2002). MT Evaluation: Measures of Effectiveness in Document Exploitation. DARPA TIDES PI Meeting. Santa Monica, CA.
22. White, J., Doyon, J., & Talbott, S. (2000). Task Tolerance of MT Output in Integrated Text Processes. In Proceedings of Workshop: Embedded Machine Translation Systems. ANLP/NAACL. Seattle, WA.