Methods for automatically evaluating answers to complex questions

Information Retrieval

Volumn 9, Issue 5, 2006, Pages 565-587

  Lin, Jimmy ^a,b   Demner Fushman, Dina ^a,b  

^a UNIVERSITY OF MARYLAND   (United States)

^b Institute of Advanced Computer Science   (United States)

Author keywords

Evaluation; Question answering

Indexed keywords

EID: 33748772836     PISSN: 13864564     EISSN: 15737659     Source Type: Journal    
DOI: 10.1007/s10791-006-9003-7     Document Type: Article

References (23)

1. Amigó, E., Gonzalo, J., Peinado, V., Peñas, A., & Verdejo, F. (2004). An empirical study of information synthesis task. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL 2004).
2. Babych, B., & Hartley, A. (2004). Extending the BLEU MT evaluation method with frequency weightings. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL 2004).
3. Banerjee, S., & Lavie, A. (2005). METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization.
4. Chib, S., & Greenberg, E. (1995). Understanding the Metropolis-Hastings algorithm. American Statistician, 49(4), 329-345.
5. Dang, H. (2005). Overview of DUC 2005. In Proceedings of the 2005 Document Understanding Conference (DUC 2005) at NLT/EMNLP 2005.
6. Doddington, G. (2002). Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceeding of 2002 Human Language Technology Conference (HLT 2002).
7. Hildebrandt, W., Katz, B., & Lin, J. (2004). Answering definition questions with multiple knowledge sources. In Proceedings of the 2004 Human Language Technology Conference and the. North American Chapter of the Association for Computational Linguistics Annual Meeting (HLT/NAACL 2004).
8. Hovy, E., Lin, C.-Y., & Zhou, L. (2005). Evaluating DUC 2005 using Basic Elements. In Proceedings of the 2005 Document Understanding Conference (DUC 2005) at NLT/EMNLP 2005.
9. Kraaij, W., & Pohlmann, R. (1996). Viewing stemming as recall enhancement. In Proceedings of the 19th Annual International ACM SIGIR Conference, on Research and Development in Information Retrieval (SIGIR 1996).
10. Kulesza, A., & Shieber, S. M. (2004). A learning approach to improving sentence level MT evaluation. In Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation.
11. Lin, J., & Demner-Fushman, D. (2005). Evaluating summaries and answers: Two sides of the same coin? In Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization.
12. Lin, J., & Demner-Fushman, D. (2006). Will pyramids built of nuggets topple over? In Proceedings of the 2006 Human Language Technology Conference and the North American Chapter of the Association for Computational Linguistics Annual Meeting (HLT/NAACL 2006).
13. Lin, C.-Y., & Hovy, E. (2003). Automatic evaluation of summaries using M-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference and the North American Chapter of the Association for Computational Linguistics Annual Meeting (HLT/NAACL 2003).
14. Lin, C.-Y., & Och., F. J. (2004). Orange: A method for evaluating automatic evaluation metrics for machine translation. In Proceedings of the 20th International Conference on Computational Linguistics (COLING 2004).
15. Lin, J., Abels, E., Demner-Fushman, D., Oard, D. W., Wu, P., & Wu, Y. (2005). A menagerie of tracks at Maryland: HARD, Enterprise, QA, and Genomics, oh my! In Proceedings of the Fourteenth Text REtrieval Conference (TREC 2005).
16. Nenkova, A., & Passonneau, R. (2004). Evaluating content selection in summarization: The pyramid method. In Proceedings of the. 2004 Human Language. Technology Conference and the North American Chapter of the Association for Computational Linguistics Annual Meeting (HLT/NAACL 2004).
17. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002).
18. Soricut, R., & Brill, E. (2004). A unified framework for automatic evaluation using n-gram co-occurrence statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL 2004).
19. Voorhees, E. M. (2000). Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management, 36(5), 697-716.
20. Voorhees, E. M. (2003). Overview of the TREC 2003 question answering track. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003).
21. Voorhees, E. M. (2005). Using question series to evaluate question answering system effectiveness. In Proceedings of the 2005 Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005).
22. Voorhees, E. M., Tice, D. M. (2000). Building a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2000).
23. Xu, J., Weischedel, R., & Licuanan, A. (2004). Evaluation of an extraction-based approach to answering definition questions. In Proceedings of the 27th Annual International ACM SIGIR Conference, on Research and Development in Information Retrieval (SIGIR 2004).