Evaluating epistemic uncertainty under incomplete assessments

Information Processing and Management

Volumn 44, Issue 2, 2008, Pages 811-837

  Baillie, Mark ^a   Azzopardi, Leif ^b   Ruthven, Ian ^a  

^a UNIVERSITY OF STRATHCLYDE   (United Kingdom)

^b UNIVERSITY OF GLASGOW   (United Kingdom)

Author keywords

Incompleteness; Information Retrieval evaluation; System comparison; Test collections

Indexed keywords

KNOWLEDGE ACQUISITION; UNCERTAINTY ANALYSIS;

INCOMPLETENESS; INFORMATION RETRIEVAL EVALUATION; SYSTEM COMPARISON; TEST COLLECTIONS;

INFORMATION RETRIEVAL;

EID: 39649091430     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2007.04.002     Document Type: Article

References (42)

1. Aslam, J. A., & Savell, R. (2003). On the effectiveness of evaluating retrieval systems in the absence of relevance judgments. In SIGIR '03: Proceedings of the 26th ACM SIGIR (pp. 361-362). Toronto, Canada.
2. Barry C.L. User-defined relevance criteria: An exploratory study. Journal of the American Society for Information Science and Technology (JASIS) 45 3 (1994) 149-159
3. Blair D.C. Stairs redux: Thoughts on the stairs evaluation, ten years after. Journal of the American Society for Information Science and Technology 47 1 (1996) 4-22
4. Blair D.C. The challenge of commercial document retrieval, part i: Major issues, and a framework based on search exhaustivity, determinacy of representation and document collection size. Information Processing and Management 38 2 (2002) 273-291
5. Blair D.C. Some thoughts on the reported results of trec. Information Processing and Management 38 3 (2002) 445-451
6. Blair D.C., and Maron M.E. An evaluation of retrieval effectiveness for a full-text document-retrieval system. Communications of the ACM 28 3 (1985) 289-299
7. Buckley, C. (2005). Looking at limits and tradeoffs: Sabir research at trec 2005. In Proceedings of the fourteenth Text REtrieval Conference (TREC 2005) notebook. NIST.
8. Buckley, C., Dimmick, D., Soboroff, I., & Voorhees, E. (2006). Bias and the limits of pooling. In SIGIR '06: Proceedings of the 29th ACM SIGIR (pp. 619-620). Seattle, WA.
9. Buckley, C., & Voorhees, E. M. (2004). Retrieval evaluation with incomplete information. In Proceedings of the 27th ACM SIGIR conference (pp. 25-32).
10. Carterette, B., Allan, J., & Sitaraman, R. (2006). Minimal test collections for retrieval evaluation. In SIGIR '06: Proceedings of the 29th ACM SIGIR (pp. 268-275). Seattle, WA.
11. Chen, H., & Karger, D. R. (2006). Less is more: Probabilistic models for retrieving fewer relevant documents. In SIGIR '06: Proceedings of the 29th ACM SIGIR (pp. 429-436). Seattle, WA.
12. Cormack, G. V., Palmer, C. R., & Clarke, C. L. A. (1998). Efficient construction of large test collections. In Proceedings of the 21st ACM SIGIR (pp. 282-289).
13. Harter S.P. Psychological relevance and information science. Journal of the American Society for Information Science and Technology (JASIS) 43 9 (1992) 602-615
14. Hull D. Using statistical testing in the evaluation of retrieval experiments. SIGIR '93: Proceedings of the 16th annual international ACM SIGIR conference on research and development in information retrieval (1993), ACM Press, New York, NY, USA 329-338
15. Ingwersen P., and Järvelin K. The turn: Integration of information seeking and retrieval in context (The information retrieval series) (2005), Springer-Verlag New York, Inc., Secaucus, NJ, USA
16. Järvelin K., and Kekäläinen J. Cumulated gain-based evaluation of ir techniques. ACM Transactions on the Information System 20 4 (2002) 422-446
17. Kwok, K., Grunfeld, L., Dinstl, N., & Deng, P. (2005). Trec 2005 robust track experiments using pircs. In Proceedings of the fourteenth Text REtrieval conference (TREC 2005) notebook. NIST.
18. Liu, S., & Yu, C. (2005). Uic at trec 2005: Robust track. In Proceedings of the fourteenth Text REtrieval conference (TREC 2005) notebook. NIST.
19. Metzler, D., Diaz, F., Strohman, T., & Croft, W. B. (2005). Umass at robust 2005: Using mixtures of relevance models for query expansion. In Proceedings of the fourteenth Text REtrieval Conference (TREC 2005) notebook. NIST.
20. Popper K. Conjectures and refutations. Readings in the Philosophy of Science (2000) 9-13
21. Robertson, S., & Soboroff, I. (2003). The TREC 2002 filtering track report. In Proceedings of the eleventh Text REtrieval conference (TREC 2002).
22. Ruthven I. Integrating approaches to relevance. In New Directions in Cognitive Information Retrieval 19 (2005) 61-80
23. Salton G. The state of retrieval system evaluation. Information Processing and Management 28 4 (1992) 441-449
24. Sanderson, M., & Joho, H. (2004). Forming test collections with no system pooling. In Proceedings of the 27th ACM SIGIR conference (pp. 33-40).
25. Sanderson, M., & Zobel, J. 2005. Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of the 28th ACM SIGIR conference (pp. 162-169).
26. Saracevic T. Evaluation of evaluation in information retrieval. SIGIR '95: Proceedings of the 18th annual international ACM SIGIR conference on research and development in information retrieval (1995), ACM Press, New York, NY, USA 138-146
27. Savoy J. Statistical inference in retrieval effectiveness evaluation. Information Processing and Management 33 4 (1997) 495-512
28. Schamber L., Eisenberg M., and Nilan M.S. A re-examination of relevance: Toward a dynamic, situational definition. Information Processing and Management 26 6 (1990) 755-776
29. Soboroff I. Dynamic test collections: Measuring search effectiveness on the live web. SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval (2006), ACM Press, New York, NY, USA 276-283
30. Soboroff, I., Nicholas, C., & Cahan, P. (2001). Ranking retrieval systems without relevance judgments. In Proceedings of the 24th ACM SIGIR conference (pp. 66-73).
31. Spärck-Jones, K., & Van Rijsbergen, C. J. (1975). Report on the need for and provision of an "ideal" information retrieval test collection. Technical report, British Library Research and Development Report 5266, University of Cambridge.
32. In: Sparck-Jones K., and Willett P. (Eds). Readings in information retrieval (1997), Morgan Kaufmann Publishers Inc., San Francisco, CA, USA
33. Swanson D.R. Searching natural language text by computer. Science 132 3434 (1960) 1099-1104 21 October
34. Swanson D.R. Historical note: Information retrieval and the future of an illusion. Journal of the American Society for Information Science 39 2 (1988) 92-98
35. Van Rijsbergen C.J. Information retrieval. 2nd ed. (1979), Butterworths, London
36. Voorhees E.M. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management 36 5 (2000) 697-716
37. Voorhees E.M. Evaluation by highly relevant documents. Proceedings of the 24th ACM SIGIR conference (2001), ACM 74-82
38. Voorhees, E. M. (2002) The philosophy of information retrieval evaluation. In CLEF '01: Revised papers from the second workshop of the cross-language evaluation forum on evaluation of cross-language information retrieval systems (pp. 355-370). London, UK: Springer-Verlag.
39. Voorhees E., and Harman D. Letters to the editor. Information Processing and Management 39 1 (2003) 153-156
40. In: Voorhees E.M., and Harman D.K. (Eds). TREC: Experiment and evaluation in information retrieval (2005), MIT Press, Cambridge, MA
41. Wallis P., and Thom J.A. Relevance judgements for assessing recall. Information Processing and Management 32 11 (1996) 273-286
42. Zobel, J. (1998). How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 21st ACM SIGIR conference (pp. 307-314).