The evolution of evaluation: Lessons from the Message Understanding Conferences

Computer Speech and Language

Volumn 12, Issue 4, 1998, Pages 281-305

  Hirschman, L ^a  

^a MITRE CORPORATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION TECHNOLOGY; INTERACTIVE COMPUTER SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS; REAL TIME SYSTEMS; SPEECH ANALYSIS;

MESSAGE UNDERSTANDING CONFERENCES (MUC);

SPEECH RECOGNITION;

EID: 0032179059     PISSN: 08852308     EISSN: None     Source Type: Journal    
DOI: 10.1006/csla.1998.0102     Document Type: Article

References (36)

1. Aberdeen, J., Burger, J., Day, D., Hirschman, L., Robinson, P. & Vilain, M. (1995). MITRE: Description of the Alembic System used for MUC-6. Proceedings of the Sixth Message Understanding Conference, Columbia, MA, pp. 141-155, November 1995.
2. Allan, J., Carbonell, J., Doddington, G., Yamron, Y. & Yang, Y. (1998). Topic detection and tracking pilot study final report. Proceedings of the Broadcast News Transcription and Understanding Workshop, pp. 194-218, Lansdowne, VA.
3. Ayuso, D., Boisen, S., Fox, H., Gish, H., Ingria, R. & Weischedel, R. (1992). BBN: Description of the PLUM system as used for MUC-4. Proceedings of the Fourth Message Understanding Conference, McLean, VA., pp. 169-176, June 1992.
4. Black, E., Abney, S., Flickenger, D., Gdaniec, C., Grishman, R., Harrison, P., Hindle, D., Ingria, R., Jelinek, F., Klavans, J., Liberman, M., Marcus, M., Roukos, S., Santorini, B. & Strzalkowski, T. (1991). A procedure for quantitatively comparing the syntactic coverage of English grammars. Proceedings of the February 1991 DARPA Speech and Natural Language Workshop, pp. 306-311, Pacific Grove CA, February 1991.
5. Brill, E. (1993). A corpus-based approach to language learning. Dissertation, University of Pennsylvania.
6. Burger, J., Palmer, D. & Hirschman, L. (1998). Named entity scoring for speech input. Proceedings COLING-ACL '98, 36th Annual Meeting of the Assoc. for Computational Linguistics, pp. 201-205 Montreal, August 1998.
7. Carroll, J., Briscoe, T. & Sanfilippo A. (1998). Parser evaluation: a survey and a new proposal. Proceedings of the First International Conference on Language Resources and Evaluation. European Language Resources Association, Granada, Spain, pp. 337-454.
8. Chinchor, N. (1992). MUC-4 evaluation metrics. Proceedings of the Fourth Message Understanding Conference, pp. 22-29 McLean, VA., June 1992.
9. Chinchor, N., Hirschman, L. & Lewis, D. (1993). Evaluating message understanding systems: An analysis of the third message understanding conference (MUC-3). Computational Linguistics, 19, No. 3, pp. 409-449.
10. Collins, M. (1997). Three generative, lexicalised models for statistical parsing. Proceedings of the 35th Annual Meeting of the Association For Computational Linguistics, Madrid, Spain, pp. 16-23, July 1997.
11. Dolan, C. P., Goldsmith, S. R., Cuda, T. V. & Nakamura, A. M. (1991). Hughes trainable text skimmer: Description of the TTS system as used for MUC-3. Proceedings of the Third Message Understanding Conference, Morgan Kaufmann, pp., 155-162, May 1991.
12. Gaizauskas, R., Wakao, T., Humphreys, K., Cunningham, H. & Wilks, Y. (1995). University of Sheffield: Description of the LaSIE system as used for MUC-6. Proceedings of the Sixth Message Understanding Conference (MUC-6). Morgan Kaufmann, pp. 207-220, November 1995.
13. Grishman, R. & Sundheim, B. (1996). Message understanding conference-6: A brief history. Proceedings of COLING-96, Copenhagen, Denmark, pp. 466-471, August 1996.
14. Harman, D. (1998). The Text Retrieval Conferences (TRECs) and the Cross-Language Track. In Proceedings of the First International Conference on Language Resources and Evaluation. (A. Rubio, N. Gallardo, R. Castro & A. Tejadaeck) pp. 517-522. European Language Resources Association, Granada, Spain.
15. Hirschman, L., (1998). Evaluating spoken language interaction: Experiences from the DARPA spoken language program 1980-1985. In Spoken Language Discourse S. Luperfoy ed. MIT Press, Cambridge, MA.
16. Hirschman, L., Robinson, P., Burger, J. & Vilain, M. (1998). Automating coreference: The role of automated training data. AAAI Spring Symposium on Applying Machine Learning to Discourse Processing. Stanford, CA, March 1998.
17. Hobbs, J. R., Appelt, D. E., Tyson, J., Bear, M. & Israel, D. (1992). SRI International: description of the FASTUS system used for MUC-4. Proceedings of the Fourth Message Understanding Conference. pp. 268-275, McLean, VA, June 1992.
18. Jacobs, P. & Rau, L. (1990). SCISOR: Extracting information from on-line news. Communications of the Assoc. for Computing Machinery 33, 88-97.
19. Lehnert, W., Cardie, C., Fisher, D., Riloff, E. & Williams, R. (1991). University of Massachusetts: Description of the CIRCUS systems as used for MUC-3. Proceedings of the Third Message Understanding Conference, Morgan Kaufmann, pp. 223-233, May 1991.
20. Lehnert, W., Cardie, C., Fisher, D., McCarthy, J., Riloff, E. & Soderland, S. (1992). University of Massachusetts: MUC-4 Test Results and Analysis. Proceedings of the Fourth Message Understanding Conference, Morgan Kaufmann, pp. 151-158, June 1992.
21. Lin, D. (1993). University of Manitoba: Description of the NUBA System as Used for MUC-5. Proceedings of the Fifth Message Understanding Conference, Morgan Kaufmann, pp. 263-275, August 1993.
22. Marcus, M., Santorini, B. & Marcinkiewicz, M. (1993). Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics 19, 313-330.
23. Miller, S., Crystal, M., Fox, H., Ramshaw, L., Schwartz, R., Stone, R., Weischedel, R. & the Annotation Group. (1998). Algorithms that learn to extract information; BBN: description of the SIFT system as used for MUC-7. In Proceedings of the Seventh Message Understanding Conference, Fairfax, VA, in press.
24. MUC-3 (1991). Proceedings of the Third Message Understanding Conference (MUC-3), Morgan Kaufmann, San Francisco, CA.
25. MUC-4 (1992). Proceedings of the Fourth Message Understanding Conference (MUC-4), Morgan Kaufmann, San Francisco, CA.
26. MUC-5 (1993). Proceedings of the Fifth Message Undersanding Conference (MUC-5), Morgan Kaufmann, San Francisco, CA.
27. MUC-6 (1995). Proceedings of the Sixth Message Understanding Conference (MUC-6), Morgan Kaufmann, San Francisco, CA.
28. Pallett, D. (1998). The NIST Role in automatic speech recognition benchmark tests. In Proceedings of the First International Conference on Language Resources and Evaluation, (A. Rubio, N. Gallardo, R. Castro and A. Tejadaeck). pp. 327-330. European Language Resources Association, Granada, Spain.
29. Sparck-Jones, K. & Galliers, J. R. (1998). Evaluating Natural Language: An Analysis and Review, Springer, Berlin.
30. Sundheim, B. (1992). Overview of the fourth message understanding evaluation and conference. Proceedings of the Fourth Message Understanding Conference pp. 3-21, Fairfax VA, June 1992.
31. Sundheim, B. (1993). Tipster/MUC-5 information extraction system evaluation, Proceedings of the Fifth Message Understanding Conference, 1993, pp. 25-44, Baltimore, MD, August 1993.
32. Sundheim, B. (1995). Overview of the results of the MUC-6 evaluation. Proceedings of the Sixth Message Understanding Conference, pp. 13-31, Columbia, MD, November 1995.
33. Sundheim, B. & Chinchor, N. (1993). Survey of the message understanding conferences. Proceedings of the Human Language Technology Conference, Princeton, NJ, pp. 56-65, March 1993.
34. Vilain, M., Burger, J., Aberdeen, J., Connoly, D. & Hirschman, L. (1995). A Model-Theoretic Coreference Scoring Scheme. Proceedings of the Sixth Message Understanding Conference (MUC-6), Morgan Kaufmann, pp. 45-52, November 1995.
35. Walter, S. M. (1992). The Neil-Montgomery system evaluation methodology. Proceedings of the Fifth DARPA Speech and Natural Language Workshop. Morgan Kaufmann, pp. 323-326, February 1992.
36. Will, C. A. (1993). Comparing human and machine performance for natural language information extraction: results for English microelectronics from the MUC-5 evaluation. Proceedings of the Fifth Message Understanding Conference, Baltimore, MD, pp. 53-67, August 1993.