The Monte Carlo method and the evaluation of retrieval system performance

Journal of the American Society for Information Science

Volumn 50, Issue 2, 1999, Pages 181-191

  Burgin, Robert ^a  

^a NORTH CAROLINA CENTRAL UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BENCHMARKING; MATHEMATICAL MODELS; MONTE CARLO METHODS; PROBABILITY; QUERY LANGUAGES;

HYPERGEOMETRIC MODELS;

INFORMATION RETRIEVAL SYSTEMS;

EID: 0032761693     PISSN: 00028231     EISSN: None     Source Type: Journal    
DOI: 10.1002/(SICI)1097-4571(1999)50:2<181::AID-ASI8>3.0.CO;2-9     Document Type: Article

References (38)

1. Buckley, C. (1985). Implementation of the SMART information retrieval system. Technical Report No. 85-686. Ithaca, NY: Cornell University.
2. Burgin, R. (1995). The retrieval effectiveness of five clustering algorithms as a function of indexing exhaustivity. Journal of the American Society for Information Science, 46, 562-572.
3. Cleverdon, C.W. (1972). On the inverse relationship of recall and precision. Journal of Documentation, 28, 195- 201.
4. Cooper, M.D. (1973). A simulation model of an information retrieval system. Information Storage and Retrieval, 9, 13- 32.
5. Cooper, M.D. (1971). Evaluation of information retrieval systems: A simulation and cost approach. Unpublished doctoral dissertation, University of California, Berkeley, CA.
6. Cooper, W.S. (1968). Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation, 19, 30-41.
7. Edelbrock, C. (1979). Mixture model tests of hierarchical clustering algorithms: The problem of classifying everything. Multivariate Behavioral Research, 14, 367- 384.
8. Edelbrock, C. & McLaughlin, B., (1980). Hierarchical cluster analysis using intraclass correlations: A mixture model study. Multivariate Behavioral Research, 15, 299- 318.
9. Edgington, E.S. (1995). Randomization tests (3rd ed.) New York: Marcel Dekker.
10. Gordon, M.D. (1991). User-based document clustering by redescribing subject descriptions with a genetic algorithm. Journal of the American Society for Information Science, 42, 311- 322.
11. Gordon, M.D. (1990). Evaluating the effectiveness of information retrieval systems using simulated queries. Journal of the American Society for Information Science, 41, 313- 323.
12. Gordon, M. (1988). Probabilistic and genetic algorithms for document retrieval. Communications of the ACM, 31, 1208- 1218.
13. Hammersley, J.M. & Handscomb, D.C., (1964). Monte Carlo methods. New York: John Wiley & Sons, Inc.
14. Harman, D.K. (Ed.) (1996). The Fourth Text Retrieval Conference (TREC-4). Gaithersburg, Maryland: National Institute of Standards and Technology.
15. Harter, S.P. (1996). Variations in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47, 37-49.
16. Heine, M.D. (1981). Simulation, and simulation experiments. In K. Sparck Jones (Ed.), Information Retrieval Experiment (pp. 179- 182). London: Butterworths.
17. Keen, E.M. (1992). Presenting results of experimental retrieval comparisons. Information Processing and Management, 28, 491- 502.
18. Losee, R., Bookstein, A., & Yu, C. (1986). Probabilistic models for document retrieval: A comparison of performance on experimental and synthetic databases. In F. Rabitti (Ed.), 1986 ACM Conference on Research and Development in Information Retrieval (pp. 258- 264). Pisa, Italy: Organization of the Conference.
19. Losec, R.M. (1995). Determining information retrieval and filtering performance without experimentation. Information Processing & Management, 31, 555-572.
20. Manly, B.F.J. (1997). Randomization and Monte Carlo methods in biology (2nd ed.). London: Chapman and Hall.
21. Marriott, F.H.C. (1979). Barnard's Monte Carlo tests: How many simulations? Applied Statistics, 28, 75- 77.
22. Matsumoto, M., & Kurita, Y. (1994). Twisted GFSR generators. II. ACM Transactions on Modelling and Computer Simulation, 4, 254-266.
23. Noreen, E.W. (1989). Computer Intensive Methods for Testing Hypotheses: An Introduction. New York: John Wiley & Sons, Inc.
24. Park, S.K., & Miller, K.W. (1988). Random number generators: Good ones are hard to find. Communications of the ACM, 31, 1192-1201.
25. Press, W.H., Flannery, B.P., Teukolsky, S.A., & Vetterling, W.T. (1988). Numerical Recipes in C: The Art of Scientific Computing. Cambridge: Cambridge University Press.
26. Salton, G. (1992). The state of retrieval system evaluation. Information Processing and Management, 28, 441- 449.
27. Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing and Management, 24, 513- 523.
28. Salton, G., & McGill, M.J. (1983). Introduction to Modem Information Retrieval. New York: McGraw-Hill.
29. Scheibler, D., & Schneider, W. (1985). Monte Carlo tests of the accuracy of cluster analysis algorithms: A comparison of hierarchical and nonhierarchical methods. Multivariate Behavioral Research, 20, 283- 304.
30. Shaw, W.M., Jr. (1993). Controlled and uncontrolled subject descriptions in the CF database: A comparison of optimal cluster-based retrieval results. Information Processing & Management, 29, 751-763.
31. Shaw, W.M., Jr. (1991). Subject and citation indexing. Part II: The optimal, cluster-based retrieval performance of composite representations. Journal of the American Society for Information Science, 42, 676- 684.
32. Shaw, W.M., Jr. (1990a). Subject indexing and citation searching-Part I: Clustering structure in the cystic fibrosis document collection. Information Processing and Management, 26, 693- 703.
33. Shaw, W.M., Jr. (1990b). Subject indexing and citation searching- Part II: An evaluation and comparison. Information Processing and Management, 26, 705-718.
34. Shaw, W.M., Jr., Burgin, R., & Howell, P. (1997a). Performance standards and evaluations in IR test collections: Cluster-based retrieval models. Information Processing & Management, 33, 1-14.
35. Shaw, W.M., Jr., Burgin, R., & Howell, P. (1997b). Performance standards and evaluations in IR test collections: Vector-space and other retrieval models. Information Processing & Management, 33, 15-36.
36. Sparck Jones, K. (1974). Automatic indexing. Journal of Documentation, 30, 393-432.
37. Tague-Sutcliffe, J., & Blustein, J. (1995). A statistical analysis of the TREC-3 data. In D.K. Harman (Ed.), Overview of the Third Text Retrieval Conference (TREC-3) (pp. 385-398). Gaithersburg, Maryland: National Institute of Standards and Technology.
38. Van Rijsbergen, C. J. (1979). Information Retrieval. London: Butterworths.