Automatic language and information processing: Rethinking evaluationf

Natural Language Engineering

Volumn 7, Issue 1, 2001, Pages 29-291

  Jones, Karen Sparck ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84937340307     PISSN: 13513249     EISSN: 14698110     Source Type: Journal    
DOI: 10.1017/s1351324901002583     Document Type: Article

References (24)

1. ARPA (1993) Report of the Advanced Research Projects Agency Machine Translation Program, System Evaluation, http://ursula.georgetown.edu/.
2. Chinchor, N. (1995) Overview of results of the MUC-6 evaluation. Sixth Message Understanding Conference (MUC-6), pp. 13–31. San Francisco: Morgan Kaufmann.
3. White, J. S., O’Connell, T. and O’Mara, F. E. (1994) Advanced Research Projects Agency Machine Translation Program: 3Q94 Evaluation, http://ursula.georgetown.edu/.
4. Young, S. J. and Chase, L. L. (1998) Speech recognition evaluation: a review of the U.S. CSR and LVCSR programmes. Computer Speech & Lang., 12, 263–279. Chinchor, N. (1998) Overview of MUC-7/MET-2. =Message Understanding Conference Pro-ceedings: MUC-7. http://www.muc.saic.com/proceedings/muc-7-toc.html.
5. EAGLES (1996) EAGLES: Evaluation of Natural Language Processing Systems. Final Report, EAGLES Document EAG-EWG-PR2. (From Centre for Sprogteknologi, Njalsgade 80, 2300 Copenhagen, Denmark.)
6. Falkedal, K. (1991) Evaluation Methods for Machine Translation Systems: An Historical Overview and Critical Account. Report, ISSCO, Universite de Geneve.
7. Firmin, T. and Chrzanowski, M. J. (1999) An evaluation of automatic text summarisation systems. In: I. Mani and M. T. Maybury (eds.), Advances in Automatic Text Summarisation, pp. 325–339. Cambridge, MA: MIT Press.
8. Gaizauskas, R. and Wilks, Y. (1998) Information extraction: beyond document retrieval. J. Documentation, 54(1), 70–105.
9. Gordon, M. and Pathak, P. (1999) Finding information on the World Wide Web: the retrieval effectiveness of search engines. Inf or. Process, and Manage., 35(2), 141–180.
10. Grishman, R. and Sundheim, B. (1995) Design of the MUC-6 evaluation. Sixth Message Understanding Conference (MUC-6), 1–11. San Francisco: Morgan Kaufmann.
11. Johnson, S. E., Jourlin, R, Moore, G. L., Sparck Jones, K. and Woodland, P. C. (1999) Spoken document retrieval for TREC-7 at Cambridge University. In: E. M. Voorhees and D. K. Harman (eds.), Seventh Text REtrieval Conference (TREC-7), 191–200. (Special Publication 500–242, Gaithersburg, MD: National Institute of Standards and Technology.)
12. Mani, I., House, D., Klein, G., Hirschman, L., Obrst, L., Firmin, T., Chrzanowski, M. and Sundheim, B. (1998) The TIPSTER SUMMAC Text Summarisation Evaluation. Final Re-port, The Mitre Corporation, Washington C3 Center, McClean, YA.
13. MUC-5 (1993) Fifth Message Understanding Conference (MUC-5). San Francisco: Morgan Kaufmann.
14. MUC-6 (1995) Sixth Message Understanding Conference (MUC-6). San Francisco: Morgan Kaufmann.
15. MUC-7 (1998) Message Understanding Conference Proceedings: MUC-7. http://www.muc.saic.com/proceedings/muc-7-toc.html.
16. Pallett, D. S., Fiscus, J. G., Fisher, W. M. and Garofolo, J. S. (1993) Benchmark tests for the DARPA spoken language programme. Human Language Technology: Proceedings of a Workshop, 7–18. San Francisco: Morgan Kaufmann.
17. Pallett, D. S., Fiscus, J. G., Fisher, W. M., Garofolo, J. S., Lund, B. A. and Przybocki, M. A. (1994) 1993 benchmark tests for the ARPA spoken language programme. Proceedings Human Language Technology Workshop, 49–74. San Francisco: Morgan Kaufmann.
18. Sparck Jones, K. (1999) Further reflections on TREC. Inf or. Process, and Manage., 36(1), 37–85.
19. Sparck Jones, K. (2000) Summary performance comparisons: TREC-2 through TREC-8. In: Voorhees, E. M. and Harman, D. K. (eds.), Eighth Text REtrieval Conference (TREC-8).
20. Sparck Jones, K. and Galliers, J. (1996) Evaluating Natural Language Processing Systems: Lecture Notes in Artificial Intelligence 1083. Springer-Verlag.
21. Sundheim, B. (1993) Tipster/MUC-5 information extraction evaluation. Fifth Message Understanding Conference (MUC-5), 27–44. San Francisco: Morgan Kaufmann.
22. Vasoncellos. M. (ed.) (1994) MT Evaluation: Basis for Future Directions: Proceedings Workshop Sponsored by the National Science Foundation. Washington, DC: Association for Machine Translation in the Americas.
23. Voorhees, E. M. and Harman, D. K. (1999) Overview of the Seventh Text REtrieval Conference (TREC-7). In: Voorhees, E. M. and Harman, D. K. (eds.) Seventh Text REtrieval Conference (TREC-7). pp. 1–23. Special Publication 500–242, Gaithersburg, MD: National Institute of Standards and Technology.
24. Voorhees, E. M. and Harman, D. K. (2000) Overview of the Eighth Text REtrieval Conference (TREC-8). In: Voorhees, E. M. and Harman, D. K. (eds.) 8th Text REtrieval Conference (TREC-8).