Approximate recall confidence intervals

ACM Transactions on Information Systems

Volumn 31, Issue 1, 2013, Pages

  Webber, William ^a  

^a UNIVERSITY OF MARYLAND   (United States)

Author keywords

Posterior distributions; Probabilistic models

Indexed keywords

BAYESIAN METHODS; CONFIDENCE INTERVAL; DOCUMENT SETS; HYPERPARAMETERS; MEASURE OF EFFECTIVENESS; MEDICAL DOMAINS; MONTE CARLO ESTIMATES; NORMAL APPROXIMATION; POSTERIOR DISTRIBUTIONS; PROBABILISTIC MODELS; RANDOM SAMPLE; RELEVANCE ASSESSMENTS; RELEVANT DOCUMENTS; SAMPLING DESIGN; SMALL POPULATION;

BAYESIAN NETWORKS; ESTIMATION; PROBABILITY DISTRIBUTIONS;

MONTE CARLO METHODS;

EID: 84873884780     PISSN: 10468188     EISSN: 15582868     Source Type: Journal    
DOI: 10.1145/2414782.2414784     Document Type: Article

References (46)

1. Agresti, A. and Caffo, B. 2000. Simple and effective confidence intervals for proportions and differences of proportions result from adding two successes and two failures. Am. Statistician 54, 4, 280-288.
2. Agresti, A. and Coull, B. A. 1998. Approximate is better than "exact" for interval estimation of binomial proportions. Am. Statistician 52, 2, 119-126.
3. Al-Jararha, J. 2008. Unbiased ratio estimation for finite populations. Ph.D. dissertion, Colorado State University, Fort Collins, CO.
4. Aslam, J. and Pavlu, V. 2008. A practical sampling strategy for efficient retrieval evaluation. Tech. rep., Northeastern University, Boston, MA.
5. Aslam, J., Pavlu, V., and Yilmaz, E. 2006. A statistical method for system evaluation using incomplete judgments. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, S. Dumais, E. Efthimiadis, D. Hawking, and K. Jarvelin Eds. 541-548.
6. Berger, J. O., Bernardo, J. M., and Sun, D. 2008. Objective priors for discrete parameter spaces. Tech. rep., Duke University, Durham, NC.
7. Bolstad, W. M. 2007. Introduction to Bayesian Statistics. John Wiley & Sons, Hoboken, NJ.
8. Brown, L. D., Cai, T. T., and DasGupta, A. 2001. Interval estimation for a binomial proportion. Stat. Sci. 18, 2, 101-133.
9. Buckland, S. T. 1984. Monte Carlo confidence intervals. Biometrics 40, 3, 811-817.
10. Cai, T. T. 2005. One-sided confidence intervals in discrete distributions. J. Stat. Plan. Inference 131, 1, 63-88.
11. Carterette, B. 2007. Robust test collections for retrieval evaluation. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, C. L. A.
12. Clarke, N. Fuhr, N. Kando, W. Kraaij, and A. de Vries Eds. 55-62.
13. Chen, M.-H. and Shao, Q.-M. 1999. Monte Carlo estimation of Bayesian credible and HPD intervals. J. Comput. Graph. Stat. 8, 1, 69-92.
14. Cheng, R. C. H. 1978. Generating beta variates with nonintegral shape parameters. Commun. ACM 21, 4, 317-322.
15. Clopper, C. J. and Pearson, E. S. 1934. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26, 4, 404-413.
16. Cochran, W. G. 1977. Sampling Techniques 3rd Ed. John Wiley & Sons, Hoboken, NJ.
17. Dutka, J. 1984. The early history of the hypergeometric function. Arch. Hist. Exact Sci. 31, 1, 15-34.
18. Dyer, D. and Chiou, P. 1984. An information-theoretic approach to incorporating prior information in binomial sampling. Commun. Stat. Theory Methods 13, 17, 2051-2083.
19. Dyer, D. and Pierce, R. L. 1993. On the choice of the prior distribution in hypergeometric sampling. Commun. Stat. Theory Methods 22, 8, 2125-2146.
20. Feller, W. 1945. On the normal approximation to the binomial distribution. Ann. Math. Stat. 16, 4, 319-329.
21. Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. B. 2004. Bayesian Data Analysis 2nd Ed. Chapman and Hall/CRC, London.
22. Greenland, S. Agresti, A., and Caffo, B. 2001. Simple and effective confidence intervals for proportions and differences of proportions result from adding two successes and two failures. Amer. Statistician, 54, 280-288: Comment by Greenland and reply. Amer. Statistician 55, 2, 172.
23. Guenther, W. C. 1971. Unbiased confidence intervals. Amer. Statistician 25, 1, 51-53.
24. Hartley, H. O. and Ross, A. 1954. Unbiased ratio estimators. Nature 174, 270-271.
25. Hedin, B., Tomlinson, S., Baron, J. R., and Oard, D. W. 2009. Overview of the TREC 2009 legal track. In Proceedings of the 18th Text REtrieval Conference, E. Voorhees and L. P. Buckland Eds. 1:4:1-40. NIST Special Publication 500-278.
26. Jeffreys, H. 1946. An invariant form for the prior probability in estimation problems. In Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences 186, 1007, 453-461.
27. Kantor, P., Kim, M.-H., Ibraev, U., and Atasoy, K. 1999. Estimating the number of relevant documents in enormous collections. In Proceedings of the ASIS Annual Meeting. 507-514.
28. Koopman, P. A. R. 1984. Confidence intervals for the ratio of two binomial proportions. Biometrics 40, 2, 513-517.
29. Lehmann, E. L. and Romano, J. P. 2005. Testing Statistical Hypotheses 3rd Ed. Springer, Berlin.
30. Liu, Y. K. and Kott, P. S. 2009. Evaluating alternative one-sided coverage intervals for a proportion. J. Off. Stat. 25, 4, 569-588.
31. Newcombe, R. G. 2001. Logit confidence intervals and the inverse sinh transformation. Amer. Statistician 55, 3, 200-202.
32. Neyman, J. 1935. On the problem of confidence intervals. Ann. Math. Stat. 6, 3, 111-116.
33. Nicholson, W. L. 1956. On the normal approximation to the hypergeometric distribution. Ann. Math. Stat. 27, 2, 471-483.
34. Oard, D. W., Hedin, B., Tomlinson, S., and Baron, J. R. 2008. Overview of the TREC 2008 legal track. In
35. Pavlu, V. 2008. Large scale IR evaluation. Ph.D. dissertation, Northeastern University, Boston MA.
36. Roitblat, H. L., Kershaw, A., and Oot, P. 2010. Document categorization in legal electronic discovery: Computer Classification vs. manual review. J. Amer. Soc. Inf. Sci. Technol. 61, 1, 70-80.
37. Sarndal, C.-E., Swensson, B., and Wretman, J. 1992. Model Assisted Survey Sampling. Springer-Verlag, Berlin.
38. Simel, D. L., Samsa, G. P., and Matchar, D. B. 1991. Likelihood ratios with confidence: Sample size estimation for diagnostic test studies. J. Clin. Epidemiol. 44, 8, 763-770.
39. Smithson, M. 2002. Confidence Intervals. Sage Publications, Thousand Oaks, CA.
40. Sunter, A. B. 1977. List sequential sampling with equal or unequal probabilities without replacement. J. R. Stat. Soc. Series C (Applied Statistics) 26, 3, 261-268.
41. Taylor, J. R. 1997. Introduction to Error Analysis 2nd Ed. University Science Books.
42. Thompson, S. K. 2002. Sampling 2nd Ed. John Wiley & Sons, New York, NY.
43. Tomlinson, S., Oard, D. W., Baron, J. R., and Thompson, P. 2007. Overview of the TREC 2007 legal track. In Proceedings of the 16th Text REtrieval Conference, E. Voorhees and L. P. Buckland Eds. 5:1-34. NIST Special Publication 500-274.
44. Webber, W., Oard, D. W., Scholer, F., and Hedin, B. 2010. Assessor error in stratified evaluation. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management. 539-548.
45. Wetherill, G. B. and Glazebrook, K. D. 1986. Sequential Methods in Statistics 3rd Ed. Chapman and Hall, London.
46. Zobel, J. 1998. How reliable are the results of large-scale information retrieval experiments In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel Eds. Melbourne, Australia, 307-314.