Measuring Retrieval Effectiveness: A New Proposal and a First Experimental Validation

Journal of the American Society for Information Science and Technology

Volumn 55, Issue 6, 2004, Pages 530-543

  Mea, Vincenzo Della ^a   Mizzaro, Stefano ^a  

^a UNIVERSITY OF UDINE   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

DATA ACQUISITION; DATABASE SYSTEMS; MATHEMATICAL MODELS; PROBABILITY DISTRIBUTIONS; SEARCH ENGINES;

AVERAGE DISTANCE MEASURE (ADM); TEXT RETRIEVAL CONFERENCE (TREC) DATA;

INFORMATION RETRIEVAL SYSTEMS;

EID: 1842680293     PISSN: 15322882     EISSN: None     Source Type: Journal    
DOI: 10.1002/asi.10408     Document Type: Article

References (40)

1. Bookstein, A. (1979). Relevance. Journal of the American Society for Information Science, 30, 269-273.
2. Borlund, P., & Ingwersen, P. (1998). Measures of relative relevance and ranked half-life: Performance indicators for interactive IR. Proceedings of the 21st Annual International ACM SIGIR Conference (pp. 324-331). Melbourne, Australia.
3. Bruce, H.W. (1994). A cognitive view of the situational dynamism of user-centered relevance estimation. Journal of the American Society for Information Science, 45, 142-148.
4. Buckley, C., & Voorhees, E.M. (2000). Evaluating evaluation measure stability. Proceedings of the 23rd Annual International ACM SIGIR Conference (pp. 33-40). Athens, Greece.
5. Buckley, C., & Voorhees, E.M. (2002). The effect of topic set size on retrieval experiment error. Proceedings of the 25th Annual International ACM SIGIR Conference (pp. 316-323). Tampere, Finland.
6. Burgin, R. (1999). The Monte Carlo method and the evaluation of retrieval system performance. Journal of the American Society for Information Science, 50, 181-191.
7. Cooper, W.S. (1994). The formalism of probability theory in IR: A foundation or an encumbrance? Proceedings of the 17th Annual International ACM SIGIR Conference (pp. 243-247). Dublin, Ireland.
8. Eisenberg, M. (1986). Magnitude estimation and the measurement of relevance. Unpublished doctoral thesis, Syracuse University, Syracuse, NY.
9. Eisenberg, M., & Hu, X. (1987). Dichotomous relevance judgments and the evaluation of information systems. Proceedings of the American Society for Information Science (pp. 66-69). Medford, NJ: Learned Information.
10. Eisenberg, M.B. (1988). Measuring relevance judgments. Information Processing & Management, 24, 373-389.
11. Frei, H., & Schauble, P. (1991). Determining the effectiveness of retrieval algorithms. Information Processing & Management, 27, 153-164.
12. Harter, S.P. (1996). Variations in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47, 37-49.
13. Ingwersen, P. (1992). Information retrieval interaction. London: Taylor Graham.
14. Janes, J.W. (1991a). The binary nature of continuous relevance judgments: A case study of users' perceptions. Journal of the American Society for Information Science, 42, 754-756.
15. Janes, J.W. (1991b). Relevance judgments and the incremental presentation of document representations. Information Processing & Management, 27, 629-646.
16. Janes, J.W. (1994). Other people's judgments: A comparison of user's and other's judgments of document relevance, topicality, and utility. Journal of the American Society for Information Science, 45, 160-171.
17. Janes, J.W., & McKinney, R. (1992). Relevance judgments of actual users and secondary judges. Library Quarterly, 62, 150-168.
18. Järvelin, K., & Kekäläinen, J. (2000). IR evaluation methods for retrieving highly relevant documents. Proceedings of the 23rd Annual International ACM SIGIR Conference (pp. 41-48). Athens, Greece.
19. Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), 20, 422-446.
20. Korfhage, R.R. (1997). Information storage and retrieval. New York: John Wiley & Sons.
21. Losee, R.M. (2000). When information retrieval measures agree about the relative quality of document rankings. Journal of the American Society for Information Science, 51, 834-840.
22. Manmatha, R., Rath, T., & Feng, F. (2001). Modeling score distribution for combining the outputs of search engines. Proceedings of the 24th Annual International ACM SIGIR Conference, 267-275. New Orleans, LA.
23. Mizzaro, S. (1997). Relevance: The whole history. Journal of the American Society for Information Science, 48, 810-832.
24. Mizzaro, S. (1998). How many relevances in information retrieval? Interacting with Computers, 10, 305-322.
25. Mizzaro, S. (2001). A new measure of retrieval effectiveness (Or: What's wrong with precision and recall). In T. Ojala (Ed.), International Workshop on Information Retrieval (IR'2001) (pp. 43-52). Infotech Oulu, Oulu, Finland.
26. Robertson, S.E. (1977). The probability ranking principle in IR. Journal of Documentation, 33, 294-304. Republished in K. Sparck Jones, & P. Willett. (1997), Readings in information retrieval (pp. 281-286). San Francisco, CA: Morgan Kaufmann.
27. Robertson, S.E. (1977). The probability ranking principle in IR. Journal of Documentation, 33, 294-304. Republished in K. Sparck Jones, & P. Willett. (1997), Readings in information retrieval (pp. 281-286). San Francisco, CA: Morgan Kaufmann.
28. Rorvig, M.E. (1988). Psychometric measurement and information retrieval. Annual Review of Information Science and Technology, 23, 157-189.
29. Salton, G., & McGill, M.J. (1984). Introduction to modern information retrieval. New York: McGraw-Hill.
30. Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29, 3-48.
31. Schamber, L., Eisenberg, M.B., & Nilan, M.S. (1990). A re-examination of relevance: Toward a dynamic, situational definition. Information Processing & Management, 26, 755-776.
32. Sormunen, E. (2002). Liberal relevance criteria of TREC - Counting on negligible documents? In M. Beaulieu et al. (Eds.), Proceedings of the 25th Annual ACM SIGIR Conference (pp. 324-330). Tampere, Finland.
33. Sparck Jones, K. (1974). Automatic indexing. Journal of Documentation, 30, 393-432.
34. Spink, A., Greisdorf, H., & Bateman, J. (1998). From highly relevant to not relevant: Examining different regions of relevance. Information Processing & Management, 34, 599-621.
35. Swets, J.A. (1967). Effectiveness of information retrieval methods. Cambridge, MA: Bolt, Beranek and Newman.
36. van Rijsbergen, C.J. (1979). Information retrieval (2nd ed.). London: Butterworths.
37. Voorhees, E.M. (2001). Evaluation by highly relevant documents. Proceedings of the 24th Annual International ACM SIGIR Conference (pp. 74-82). New Orleans, LA.
38. Voorhees, E.M., & Harman, D. (2000). Overview of the Eighth Text Retrieval Conference (TREC-8), The 8th Text Retrieval Conference (TREC-8) (pp. 1-24) (NIST SP-500-246). Available: http://trec.nist.gov/
39. Yao, Y.Y. (1995). Measuring retrieval effectiveness based on user preference of documents. Journal of the American Society for Information Science, 46, 133-145.
40. Zobel, J. (1998). How reliable are the results of large-scale information retrieval experiments? Proceedings of the 21st Annual International ACM SIGIR Conference (pp. 307-314). Melbourne, Australia.