A simple measure to assess non-response

ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies

Volumn 1, Issue , 2011, Pages 1415-1424

  Peñas, Anselmo ^a   Rodrigo, Alvaro ^a  

^a UNIVERSIDAD NACIONAL DE EDUCACIÓN A DISTANCIA   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY MEASURES; MULTIPLE CHOICE; READING COMPREHENSION; REWARD SYSTEMS; SENSITIVITY PROPERTIES;

COMPUTATIONAL LINGUISTICS;

EID: 84859070743     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. Chris Buckley and Ellen M. Voorhees. 2000. Evaluating evaluation measure stability. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 33-40. ACM.
2. David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty. 2010. Building Watson: An Overview of the DeepQA Project. AI Magazine, 31(3).
3. Junichi Fukumoto, Tsuneaki Kato, and Fumito Masui. 2002. Question and Answering Challenge (QAC-1): Question Answering Evaluation at NTCIR Workshop 3. In Working Notes of the Third NTCIR Workshop Meeting Part IV: Question Answering Challenge (QAC-1), pages 1-10.
4. Jesús Herrera, Anselmo Peñas, and Felisa Verdejo. 2005. Question Answering Pilot Task at CLEF 2004. In Multilingual Information Access for Text, Speech and Images, CLEF 2004, Revised Selected Papers., volume 3491 of Lecture Notes in Computer Science, Springer, pages 581-590. (Pubitemid 41314068)
5. Bernardo Magnini, Alessandro Vallin, Christelle Ayache, Gregor Erbach, Anselmo Peñas, Maarten de Rijke, Paulo Rocha, Kiril Ivanov Simov, and Richard F. E. Sutcliffe. 2005. Overview of the CLEF 2004 Multilingual Question Answering Track. In Multilingual Information Access for Text, Speech and Images, CLEF 2004, Revised Selected Papers., volume 3491 of Lecture Notes in Computer Science, Springer, pages 371-391. (Pubitemid 41314049)
6. Anselmo Peñas, Álvaro Rodrigo, Valentín Sama, and Felisa Verdejo. 2007. Overview of the Answer Validation Exercise 2006. In Evaluation of Multilingual and Multi-modal Information Retrieval, CLEF 2006, Revised Selected Papers, volume 4730 of Lecture Notes in Computer Science, Springer, pages 257-264.
7. Anselmo Peñas, Álvaro Rodrigo, Valentín Sama, and Felisa Verdejo. 2008a. Testing the Reasoning for Question Answering Validation. In Journal of Logic and Computation. 18(3), pages 459-474. (Pubitemid 351774914)
8. Anselmo Peñas, Álvaro Rodrigo, and Felisa Verdejo. 2008b. Overview of the Answer Validation Exercise 2007. In Advances in Multilingual and Multimodal Information Retrieval, CLEF 2007, Revised Selected Papers, volume 5152 of Lecture Notes in Computer Science, Springer, pages 237-248.
9. Anselmo Peñas, Pamela Forner, Richard Sutcliffe, Álvaro Rodrigo, Corina Forascu, Iñaki Alegria, Danilo Giampiccolo, Nicolas Moreau, and Petya Osenova. 2010. Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation. In Multilingual Information Access Evaluation I. Text Retrieval Experiments, CLEF 2009, Revised Selected Papers, volume 6241 of Lecture Notes in Computer Science, Springer.
10. Alvaro Rodrigo, Anselmo Peñas, and Felisa Verdejo. 2008. Evaluating Answer Validation in Multi-stream Question Answering. In Proceedings of the Second International Workshop on Evaluating Information Access (EVIA 2008).
11. Alvaro Rodrigo, Anselmo Peñas, and Felisa Verdejo. 2011. Evaluating Question Answering Validation as a classification problem. Language Resources and Evaluation, Springer Netherlands (In Press).
12. Tetsuya Sakai. 2006. Evaluating Evaluation Metrics based on the Bootstrap. In SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, USA, August 6-11, 2006, pages 525-532.
13. Tetsuya Sakai. 2007a. On the Reliability of Factoid Question Answering Evaluation. ACM Trans. Asian Lang. Inf. Process., 6(1).
14. Tetsuya Sakai. 2007b. On the reliability of information retrieval metrics based on graded relevance. Inf. Process. Manage., 43(2):531-548. (Pubitemid 44646706)
15. Ellen M. Voorhees and Chris Buckley. 2002. The effect of Topic Set Size on Retrieval Experiment Error. In SIGIR '02: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pages 316-323.
16. Ellen M. Voorhees and Dawn M. Tice. 1999. The TREC-8 Question Answering Track Evaluation. In Text Retrieval Conference TREC-8, pages 83-105.
17. Ellen M. Voorhees. 2001. Overview of the TREC 2001 Question Answering Track. In E. M. voorhees, D. K. Harman, editors: Proceedings of the Tenth Text REtrieval Conference (TREC 2001). NIST Special Publication 500-250.
18. Ellen M. Voorhees. 2002. Overview of TREC 2002 Question Answering Track. In E.M. Voorhees, L. P. Buckland, editors: Proceedings of the Eleventh Text REtrieval Conference (TREC 2002). NIST Publication 500-251.
19. Ellen M. Voorhees. 2003. Overview of the TREC 2003 Question Answering Track. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003).