Evaluating Natural Language Processing Systems

Communications of the ACM

Volumn 39, Issue 1, 1996, Pages 73-79

  King, Margaret ^a,b,c  

^a ISSCO ^*

^b UNIVERSITY OF GENEVA   (Switzerland)

^c ISSCO   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; COMPUTER AIDED LANGUAGE TRANSLATION; DATABASE SYSTEMS; INFORMATION RETRIEVAL; QUALITY ASSURANCE; QUALITY CONTROL; QUERY LANGUAGES; STANDARDS; SYSTEMS ANALYSIS; TESTING;

HUMAN TRANSLATION; MACHINE TRANSLATION SYSTEMS;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 0029780569     PISSN: 00010782     EISSN: None     Source Type: Journal    
DOI: 10.1145/234173.234208     Document Type: Article

References (22)

1. ALPAC. Languages and Machines: Computers in Translation and Linguistics. Report of the Automatic Language Processing Advisory Committee, Division of Behavioral Sciences, National Academy of Sciences, National Research Council Publication 1416, Washington, D.C. 1966.
2. Boisen, S. and Bates, M. A practical methodology for the evaluation of spoken language systems. In Proceedings of the Third Conference on Applied Natural Language Processing. (Trento, Italy, 1992) 162-169.
3. Damerau, F.J. The Transformational Question Answering System: Description, Operating Experience and Implications. Report RC8287, IBM T.J. Watson Research Center, Yorktown Heights, N.Y. 1980.
4. EAGLES Evaluation Working Group. Interim Report, 1994. Obtainable from the Center for Language Technology, University of Copenhagen.
5. Falkedal, K. Evaluation Methods for Machine Translation Systems: An Historical Overview and Critical Account. Draft Report. ISSCO. University of Geneva.
6. Falkedal, K. Ed. Proceedings of the Evaluators' Forum, Les Rasses, ISSCO Tech. Rep., 1994.
7. Flanagan, M. Error Classification for MT Evaluation. In Technology Partnerships for Crossing the Language Barrier: Proceedings of the First Conference of the Association for Machine Translation in the Americas, (Columbia, Md. 1994).
8. Flickinger, D., Nerbonne, J., Sag, I., and Wascow, T. Toward Evaluation of NLP Systems. Hewlett Packard Laboratories, Palo Alto, Calif. 1987.
9. Galliers, J.R. and Sparck Jones, K. Evaluating Natural Language Processing Systems. Tech. Rep. 291. University of Cambridge Computer Laboratory, Cambridge, England. 1993
10. Harman, D. The First Text Retrieval Conference (TREC1). National Institute of Standards and Technology Special Publication 500-207. Gaitherberg, Md. 1995.
11. Hutchins, W.J. Machine Translation. Ellis Horwood, Chichester, England. 1986.
12. ISO/IEC 9126. Information Technology-Software Product Evaluation-Quality Characteristics and Guidelines for their Use. Dec. 1991.
13. Jarke, M., Turner, J.A., Stohr, E.A., Vassiliou, Y., White N.H., and Michielsen, K. A field evaluation of natural language for data retrieval. IEEE Trans. Soft. Eng. 11, 1 (1985), 97-113.
14. Japan Electronic Industry Development Assor. JEIDA Methodology and Criteria on Machine Translation Evaluation. Tokyo, 1992.
15. King, M. and Falkedal, K. Using Test Suites in the Evaluation of Machine Translation Systems. In Proceedings of Coling '90. (Helsinki, 1990) 211-219.
16. Lehnert, W. and Sundheim, B. A performance analysis of text-analysis technologies. AI Mag. 12, 4 (1991), 81-94.
17. Liberman, M. Text on Tap: The ACL/DCI. In Proceedings of the DARPA Speech and Natural Language Workshop. (Oct. 1989) Morgan Kaufmann, San Mateo, Calif. 173-188.
18. MUC-3. Proceedings of the Third Message Understanding Conference. Morgan Kaufmann, San Mateo, Calif. (1991).
19. Neal, J., Feit, E.L., Funke, D.J., and Montgomery, C.A. An Evaluation Methodology for Natural Language Processing Systems. Rome Laboratory Tech. Rep. RL-TR-92-308 (1992)
20. White, J.S., O'Connell, T., and O'Mara, F.E. The ARPA MT Evaluation Methodologies: Evolution, Lessons and Further Approaches. In Technology Partnerships for Crossing the Language Barrier: Proceedings of the First Conference of the Association for Machine Translation in the Americas. (Columbia, Md. 1994.)
21. Woods, W.A. Progress in NLU - An application to lunar geology. In AFIPS Conference Proceedings (1973) 441-450.
22. Whittaker, S. and Walker, M. Comparing Two User-Oriented Database Query Languages: A Field Study. Tech. Rep. HPL-ISC-89-060. Hewlett Packard Laboratories, Bristol, London. (1989.)