Building a reusable test collection for question answering

Journal of the American Society for Information Science and Technology

Volumn 57, Issue 7, 2006, Pages 851-861

  Lin, Jimmy ^a   Katz, Boris ^b  

^a UNIVERSITY OF MARYLAND   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DOCUMENT RETRIEVAL TECHNIQUES; LANGUAGE-PROCESSING CAPABILITIES; SYSTEM DEVELOPMENT;

INFORMATION USE; NATURAL LANGUAGE PROCESSING SYSTEMS; REUSABILITY; WEB BROWSERS;

INFORMATION RETRIEVAL;

EID: 33646682407     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.20348     Document Type: Article

References (45)

1. Barry. C., & Schamber. L. (1998). Users' criteria for relevance evaluation: A cross-situational comparison. Information Processing and Management, 34(2/3). 219-236.
2. Bilotti. M. (2004). Query expansion techniques for question answering. Unpublished master's thesis, Massachusetts Institute of Technology, Cambridge, MA.
3. Bilotti, M., Katz., B., & Lin, J. (2004). What works better for question answering: Stemming or morphological query expansion? In R. Gaizauskas. M. Hepple, & M. Greenwood (Eds.), Proceedings of the Information Retrieval for Question Answering (IR4QA) Workshop at SIGIR 2004. New York: ACM.
4. Brill. E., Lin, J., Banko, M., Dumais. S., & Ng, A. (2001). Data-intensive question answering. In E. Voorhees, & D. Harman (Eds.), Proceedings of the Tenth Text REtrieval Conference (TREC 2001). Gaithersburg. MD: National Institute of Standards and Technology.
5. Buckley, C., & Voorhees, E.M. (2000). Evaluating evaluation measure stability. In E. Yannakoudakis, N. Belkin, M.-K. Leong, & P. Ingwersen (Eds.), Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2000) (pp. 33-40). New York: ACM.
6. Buckley, C., & Voorhees, E.M. (2004), Retrieval evaluation with incomplete information. In K. Järvelin. J. Allan, P. Bruza, & M. Sanderson (Eds.). Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2004) (pp. 25-32). New York: ACM.
7. Cieri. C., Strassel, S., Graff. D., Martey. N., Rennert. K., & Libermun. M. (2002). Corpora for topic detection and tracking. In J. Allan (Ed.). Topic detection and tracking: Event-based information organisation (pp. 33-66). Dordrecht: Kluwer.
8. Clarke. C., Cormack G., Kisman. D., & Lynam, T. (2000). Question answering by passage selection (MultiText experiments for TREC-9). In E. Voorhees, & D. Harman (Eds.). Proceedings of the Ninth Text REtrieval Conference (TREC-9). Gaithersburg. MD: National Institute of Standards and Technology.
9. Cleverdon. C., Mills. J., & Keen, E.M. (1968). Factors determining the performance of indexing systems. Cranfield. UK: ASLIB Cranfield Research Project.
10. Cooper, W. (1971). A definition of relevance for information retrieval. Information Storage and Retrieval. 7. 19-37.
11. Cormack. G., Palmer. C., & Clarke. C. (1998). Efficient construction of large test collections. In W. Croft. A. Moffat. C. van Rijsbergen. R. Wilkinson, & J. Zobel (Eds.). Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1998) (pp. 282-289). New York: ACM.
12. Echihabi. A., & Marcu, D. (2003). A noisy-channel approach to question answering, In J. Tsujiji. E. Hinrichs. & D. Roth (Eds.), Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL 2003) (pp. 16-23). East Stroudsburg, PA: Association for Computational Linguistics.
13. Harabagiu, S., Pasca, M., & Maiorano, S. (2000). Experiments with open-domain textual question answering. In Proceedings of the 18th International Conference on Computational Linguistics (COLING 2000) (pp. 292-298). San Francisco: Morgan Kaufmann.
14. Harman, D. (1991). How effective is suffixing? Journal of the American Society for Information Science, 42(1), 7-15.
15. Harter, S. (1992). Psychological relevance and information science. Journal of the American Society for Information Science. 43(9), 602-615.
16. Harter, S. (1996). Variations in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47(1), 37-49.
17. Hirschman, L., & Gaizauskas. R. (2001). Natural language question answering: The view from here. Natural Language Engineering, 7(4), 275-300.
18. Hovy, E., Gerber, L., Hermjakob, U., Lin, C.-Y., & Ravichandran, D. (2001). Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research (HLT 2001) (pp. 339-345).
19. Hull, D. (1996). Stemming algorithms: A case study for detailed evaluation. Journal of the American Society for Information Science, 47(1), 70-84.
20. Katz, B., & Lin, J. (2003). Selectively using relations to improve precision in question answering. In B. Webber, & M. de Rijke (Eds.), Proceedings of the EACL 2003 Workshop on Natural Language Processing for Question Answering.
21. Krovetz, R. (1993). Viewing morphology as an inference process. In R. Korfhage, E. Rasmussen, & P. Willett (Eds.), Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1993) (pp. 191-202). New York: ACM.
22. Li, X., & Roth, D. (2002). Learning question classifiers. In S.-C. Tseng, T.-E. Chen, & Y.-F. Liu (Eds.), Proceedings of the 19th International Conference on Computational Linguistics (COLING 2002) (pp. 556-562). San Francisco: Morgan Kaufmann.
23. Light, M., Mann, G., Riloff, E., & Breck, E. (2001). Analyses for elucidating current question answering technology. Natural Language Engineering, 7(4), 325-342.
24. Lin, J. (2005). Evaluation of resources for question answering evaluation. In G. Marchionini, A. Moffat, J. Tait, R. Baeza-Yates, & N. Ziviani (Eds.), Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005) (pp. 392-399). New York: ACM.
25. Mizzaro, S. (1999). How many relevances in information retrieval? Interacting With Computers, 10(3), 305-322.
26. Moldovan, D., Pasca, M., Harabagiu, S., & Surdeanu, M. (2002). Performance issues and error analysis in an open-domain question answering system. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002) (pp. 33-40). East Stroudsburg, PA: Association for Computational Linguistics.
27. Monz, C. (2003). From document retrieval to question answering. Unpublished doctoral dissertation, Institute for Logic, Language, and Computation, University of Amsterdam.
28. Prager, J., Brown, E., & Coden, A. (2000). Question-answering by predictive annotation. In E. Yannakoudakis, N. Belkin, M.-K. Leong, & P. Ingwersen (Eds.), Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2000) (pp. 184-191). New York: ACM.
29. Robertson, S. (2004). Understanding inverse document frequency: On theoretical arguments for IDF Journal of Documentation, 60(5), 503-520.
30. Saracevic, T. (1975). Relevance: A review of and a framework for thinking on the notion in information science. Journal of the American Society for Information Science, 26(6), 321-343.
31. Sormunen, E. (2002). Liberal relevance criteria of TREC - Counting on negligible documents'? In M. Beaulieu, R. Baeza-Yates, S. Myaeng, & K. Järvelin (Eds.), Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002) (pp. 324-330). New York: ACM.
32. Sparck Jones, K. (2003). Is question answering a rational task? In Proceedings of the 2nd CoLogNET-ElsNET Symposium on Questions and Answers: Theoretical and Applied Perspectives.
33. Spink, A., & Greisdorf, H. (2001). Regions and levels: Mapping and measuring users' relevance judgments. Journal of the American Society for Information Science and Technology. 52(2), 161-173.
34. Srihari, R., & Li, W. (1999), Information extraction supported question answering. In E. Voorhees, & D. Harman (Eds.). Proceedings of the Eighth Text REtrieval Conference (TREC-8). Gaithersburg, MD: National Institute of Standards and Technology.
35. Tellex, S., Katz, B., Lin, J., Marton, G., & Fernandes, A. (2003). Quantitative evaluation of passage retrieval algorithms for question answering. In J. Callan, D. Hawking, A. Smeaton, C. Clarke, & G. Cormack (Eds.), Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2003) (pp. 41-47). New York: ACM.
36. Voorhees, E.M., (2000). Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management, 36(5), 697-716.
37. Voorhees, E.M. (2001a). Overview of the TREC 2001 question answering track. In E. Voorhees, & D. Harman (Eds.), Proceedings of the Tenth Text REtrieval Conference (TREC 2001). Gaithersburg, MD: National Institute of Standards and Technology.
38. Voorhees, E.M. (2001 b). Evaluation by highly relevant documents. In D. Kraft, W. Croft, D. Harper, & J. Zobel (Eds.), Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2001) (pp. 74-82). New York: ACM.
39. Voorhees, E.M. (2002). Overview of the TREC 2002 question answering track. In E. Voorhees, & L. Buckland (Eds.), Proceedings of the Eleventh Text REtrieval Conference (TREC 2002). Gaithersburg, MD: National Institute of Standards and Technology.
40. Voorhees, E.M. (2003). Overview of the TREC 2003 question answering track. In E. Voorhees, & L. Buckland (Eds.), Proceedings of the Twelfth Text REtrieval Conference (TREC 2003). Gaithersburg, MD: National Institute of Standards and Technology.
41. Voorhees, E.M., & Buckley, C. (2002). The effect of topic set size on retrieval experiment error. In M. Beaulieu, R. Baeza-Yates, S. Myaeng, & K. Järvelin (Eds.), Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002) (pp. 316-323). New York: ACM.
42. Voorhees, E.M., & Tice, D.M. (1999). The TREC-8 question answering track evaluation. In E. Voorhees. & D. Harman (Eds.), Proceedings of the Eighth Text REtrieval Conference (TREC-8). Gaithersburg, MD: National Institute of Standards and Technology.
43. Voorhees, E.M., & Tice, D.M. (2000a). Building a question answering test collection. In E. Yannakoudakis, N. Belkin, M.-K. Leong, & P. Ingwersen (Eds.), Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2000) (pp. 200-207). New York: ACM.
44. Voorhees, E.M., & Tice, D.M. (2000b) Overview of the TREC-9 question answering track. In E. Voorhees, & D. Harman (Eds.), Proceedings of the Ninth Text REtrieval Conference (TREC-9). Gaithersburg, MD: National Institute of Standards and Technology.
45. Zobel, J. (1998). How reliable are the results of large-scale information retrieval experiments? In W. Croft, A. Moffat, C. van Rijsbergen, R. Wilkinson, & J. Zobel (Eds.), Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1998) (pp. 307-314). New York: ACM.