Computer-Aided specification of quality models for machine translation evaluation

Proceedings of the 3rd International Conference on Language Resources and Evaluation, LREC 2002

Volumn , Issue , 2002, Pages 1239-1246

  Hovy, Eduard ^a   King, Margaret ^b   Popescu Belis, Andrei ^b  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b UNIVERSITY OF GENEVA   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; QUALITY ASSURANCE; STANDARDIZATION; TAXONOMIES; TRANSLATION (LANGUAGES);

COMPUTER AIDED; CONTEXT OF USE; FLEXIBLE FRAMEWORK; MACHINE TRANSLATION EVALUATIONS; MACHINE TRANSLATIONS; MT EVALUATIONS; QUALITY CHARACTERISTIC; QUALITY MODELS;

QUALITY CONTROL;

EID: 0141858971     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. AMTA. 1992. MT evaluation: Basis for future directions (Proceedings of a workshop held in San Diego, CA). Technical report, Association for Machine Translation in the Americas (AMTA).
2. K.W. Church and E. H. Hovy. 1993. Good applications for crummy MT. Machine Translation, 8:239-258.
3. O. Daly-Jones, N. Bevan, and C. Thomas, editors. 1999. Handbook of User-Centred Design: INUSE 6.2. http://www.ejeisa.com/nectar/inuse.
4. J. Doyon, K. Taylor, and J.S. White. 1998. The DARPA MT evaluation methodology: Past and present. In Proceedings of the AMTA Conference, Philadelphia, PA.
5. EAGLES-Evaluation-Workgroup. 1996. EAGLES evaluation of natural language processing systems. Final report, Center for Sprogteknologi, Denmark, October 1996.
6. M. Flanagan. 1994. Error classification for MT evaluation. In Proceedings of the AMTA Conference, Columbia, Maryland.
7. M. Forner and J.S. White. 2001. Predicting MT fidelity from noun-compound handling. In Workshop on MT Evaluation "Who did what to whom?" at Mt Summit VIII, Santiago de Compostela, Spain.
8. E.H. Hovy. 1999. Toward finely differentiated evaluation metrics for MT. In EAGLESWorkshop on Standards and Evaluation, Pisa, Italy.
9. Infoshop. 1999. Language translations: World market overview, current developments and competitive assessment. Technical report, Infoshop Japan, Global Information Inc., Kawasaki, Japan, http://www.infoshopjapan.com/study/ab3365languagetranslationtoc.html.
10. ISO/IEC-14598. 1998-2001. ISO/IEC 14598 - Information technology - Software product evaluation - Part 1: General overview (1999), Part 2: Planning and management (2000), Part 3: Process for developers (2000), Part 4: Process for acquirers (1999), Part 5: Process for evaluators (1998), Part 6: Documentation of evaluation modules (2001). ISO/IEC, Geneva.
11. ISO/IEC-9126-1. 2001. ISO/IEC 9126-1:2001 (E) - Software engineering - Product quality - Part 1: Quality model. ISO/IEC, Geneva, June.
12. ISO/IEC-9126. 1991. ISO/IEC 9126:1991 (E) - Information Technology - Software Product Evaluation - Quality Characteristics and Guidelines for Their Use. ISO/IEC, Geneva.
13. M. Kay. 1980. The proper place of men and machines in language translation. Research Report CSL-80-11, XEROX PARC.
14. M. King and K. Falkedal. 1990. Using test suites in evaluation of MT systems. In 18th Coling Conference, volume 2, Helsinki, Finland.
15. J. Lehrberger and L. Bourbeau. 1988. Machine Translation: Linguistic Characteristics of MT Systems and General Methodology of Evaluation. LingvisticæInvestigationes Suppl. 15. John Benjamins, Amsterdam.
16. J. Mason and A. Rinsche. 1995. Translation technology products. Report, OVUM Ltd.
17. M. Nagao. 1989. A Japanese view on MT in light of the considerations and recommendations reported by ALPAC, USA. Technical report, Japan Electronic Industry Development Association (JEIDA).
18. H. Nomura and J. Isahara. 1992. The JEIDA report on MT. In Workshop on MT Evaluation: Basis for Future Directions, San Diego, CA. Association for Machine Translation in the Americas (AMTA).
19. H. Nomura. 1992. JEIDA methodology and criteria on MT evaluation. Technical report, Japan Electronic Industry Development Association (JEIDA).
20. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2001. BLEU: A method for automatic evaluation of MT. Research Report, Computer Science RC22176 (W0109- 022), IBM Research Division, T.J.Watson Research Center, 17 September 2001.
21. J.R. Pierce, J.B. Carroll, E.P. Hamp, D.G. Hays, C.F. Hockett, A.G. Oettinger, and A. Perlis. 1966. Computers in translation and linguistics (ALPAC report). report 1416, National Academy of Sciences/National Research Council, 1966.
22. A. Popescu-Belis, S. Manzi, and M. King. 2001. Towards a two-stage taxonomy for MT evaluation. In Workshop on MT Evaluation "Who did what to whom?" at Mt Summit VIII, pages 1-8, Santiago de Compostela, Spain.
23. A. Popescu-Belis. 1999. Evaluation of natural language processing systems: A model for coherence verification of quality measures. In Marc Blasband and Patrick Paroubek, editors, A Blueprint for a General Infrastructure for Natural Language Processing Systems Evaluation Using Semi-Automatic Quantitative Black Box Approach in a Multilingual Environment ELSE Project LE4-8340 (Evaluation in Language and Speech Engineering).
24. K. Sparck-Jones and J.R. Galliers. 1996. Evaluating Natural Language Processing Systems: An Analysis and Review. Lecture Notes in Artificial Intelligence 1083. Springer-Verlag, Berlin/New York.
25. TEMAA. 1996. TEMAA final report. Technical Report LRE-62-070 (March 1996), Center fo Sprogteknologi, Copenhagen, Danemark, http://www.cst.ku.dk/projects/temaa/D16/d16exp.html.
26. H. S. Thompson, editor. 1992. The Strategic Role of Evaluation in Natural Language Processing and Speech Technology (Record of a workshop sponsored by DANDI, ELSNET and HCRC). University of Edinburgh (Technical Report, May 1992).
27. G. Van Slype. 1979. Critical study of methods for evaluating the quality of MT. Technical Report BR 19142, European Commission/Directorate for General Scientific and Technical Information Management (DG XIII).
28. J.S. White et al. 1992-1994. ARPA workshops on MT (series of four workshops on comparative evaluation). Technical report, PRC Inc., McLean, Virginia.
29. J.S. White. 2001. Predicting intelligibility from fidelity in MT evaluation. In Workshop on MT Evaluation "Who did what to whom?" at Mt Summit VIII, Santiago de Compostela, Spain.