Scaling the ISLE framework: Use of existing corpus resources for validation of MT evaluation metrics across languages

Proceedings of the 3rd International Conference on Language Resources and Evaluation, LREC 2002

Volumn , Issue , 2002, Pages 1254-1262

  Vanni, Michelle ^a   Miller, Keith ^b  

^a ARMED FORCES INSTITUTE OF PATHOLOGY   (United States)

^b MITRE CORPORATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ENGINEERING RESEARCH; NATURAL LANGUAGE PROCESSING SYSTEMS; PROGRAM TRANSLATORS; SOFTWARE TESTING; STANDARDS; SYNTACTICS;

EVALUATION RESEARCH; INTERNATIONAL STANDARDS; LANGUAGE ENGINEERING; MACHINE TRANSLATIONS; NAMED ENTITY TRANSLATION; RESEARCH PROGRAMS; STRUCTURAL CONTRASTS; STRUCTURAL SIMILARITY;

TRANSLATION (LANGUAGES);

EID: 35048860403     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. Balkan, L. 1994. Test Suites: Some issues on their use and design. Machine Translation Ten Years On, Conference at the University of Cranfield. 26-1.
2. Carlson, L. and D. Marcu. 2001. Discourse Tagging Reference Manual. ISI Technical Report, forthcoming.
3. Chaumier, J., Mallen, M.C., and Van Slype, G. Evaluation du systeme de traduction automatique SYSTRAN; evaluation de la qualite de la traduction. 1977. CEC. Report number 4. Luxembourg.
4. Church, K. and E. Hovy. 1993. Good applications for Crummy Machine Translation. Machine Translation 8:239-258.
5. Doyon, J., Taylor, K., and J. White. 1999. Task-Based Evaluation for Machine Translation. Proceedings of MT Summit 7. Singapore.
6. Flanagan, M. 1994. Error Classification for MT Evaluation. In Technology Partnerships for Crossing the Language Barrier: Proceedings of the First Conference of the Association for Machine Translation in the Americas, Columbia, MD.
7. Gerber, L. and E. Hovy. 1998. Improving Translation Quality by Manipulating Sentence Length. Lecture Notes in Artificial Intelligence 1529: Machine Translation and the Information Soup : Proceedings of the Third Conference of the Association for Machine Translation in the Americas. 448-460.
8. Hovy, E. 1999. Toward Finely Differentiated Evaluation Metrics for Machine Translation. Proceedings of the EAGLES Workshop on Standards and Evaluation. Pisa, Italy.
9. International Standards for Language Engineering. 2000. (http://www.isi.edu/natural-language/mteval) The ISLE Classification of Machine Translation Evaluations, Draft 1, October, 2000. Proceedings of the Hands-on Workshop on Machine Translation Evaluation. Association for Machine Translation in the Americas, Cuernavaca, Mexico.
10. Jones, D. and G. Rusk. 2000. Toward a Scoring Function for Quality-Driven Machine Translation. In Proceedings of COLING-2000.
11. Mann, W., and S. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text 8:3.243-281.
12. Miller, K. 2000. The Machine Translation of Prepositional Phrases. Unpublished PhD Dissertation. Georgetown University. Washington, DC.
13. Miller, K. and M. Vanni. 2001. Scaling the ISLE Taxonomy : Development of Metrics for the Multi-Dimensional Measurement of Machine Translation Quality. In Proceedings of MT Summit VIII. Santiago de Campostela, Spain.
14. Papineni, K., S. Roukos, T. Ward, and W. Zhu. 2001. BLEU : A Method for Automatic Evaluation of Machine Translation, IBM Research Report, RC22176, September 2001.
15. Papineni, K., S. Roukos, T. Ward, J. Henderson, and F. Reeder. 2002. Corpus-Based Comprehensive and Diagnostic MT Evaluation : Initial Arabic, Chinese, French, and Spanish Results. Proceedings of the Human Language Technology Conference.
16. Polvsen, C., N. Underwood, B. Music, and A. Neville. 1998. Evaluating Text-Type Suitability for Machine Translation a Case Study on an English-Danish MT System. Proceedings of ELRA Conference, Granada, Spain.
17. Taylor, K. and J. White. 1998. Predicting What MT is Good for : User Judgments and Task Performance. Proceedings of the 1998 conference of the As sociation of Machine Translation in the Americas. 364-373.
18. Van Slype, G. 1978. Second Evaluation of the English-French SYSTRAN Machine Translation System of the Commission of the European Communities. 1978. CEC. Final Report. Luxembourg.
19. Van Slype, G. 1979. Critical Study of Methods for Evaluating the Quality of Machine Translation. Prepared for the Commission of European Communities Directorate General Scientific and Technical Information and Information Management. Report BR 19142.
20. Vanni, M. 2000. Lessons for Text-Differentiated MT. Proceedings of the Hands-on Workshop on Machine Translation Evaluation. Association for Machine Translation in the Americas, Cuernavaca, Mexico.
21. Vanni, M. and K. Miller. 2001. Scaling the ISLE Taxonomy : Validating Tests of Machine Translation Quality for Multi- Dimensional Measurement. In Proceedings of the MT Evaluation Workshop of the MT Summit VIII, Santigao di Campostela, Spain.
22. Voss, C. and F. Reeder, eds. 1998. Proceedings of the Workshop on Embedded Machine Translation: Design, Construction, and Evaluation of Systems with an MT Component. Association of Machine Translation in the Americas Annual Meeting, Langhorne, PA.
23. Voss, C. and Van Ess-Dykema. 2000. When is an Embedded MT System "Good Enough" for Filtering? Proceedings of Embedded Machine Translation Systems. ANLP/NAACL 2000 Workshop. Seattle, Washington.
24. White, J.S. and T.A. O'Connell. 1994. The ARPA MT Evaluation Methodologies: Evolution, Lessons, and Further Approaches. Proceedings of the 1994 Conference of the Association for Machine Translation in the Americas. Columbia, MD.
25. White, J.S. and K. Taylor. 1998. A Task-Oriented Metric for Machine Translation. Proceedings of the First Language Resources and Evaluation Conference. Granada, Spain.
26. White, J., Doyon, J., and Talbott, S. 2000. Task Tolerance of MT Output in Integrated Text Processes. Proceedings of Embedded Machine Translation Systems. ANLP/NAACL 2000 Workshop. Seattle, Washington.