Bias correction in a small sample from big data

IEEE Transactions on Knowledge and Data Engineering

Volumn 25, Issue 11, 2013, Pages 2658-2663

  Lu, Jianguo ^a   Li, Dingding ^a  

^a UNIVERSITY OF WINDSOR   (Canada)

Author keywords

bias; Big data; online social networks; size estimation; small sample

Indexed keywords

BIAS; BIG DATUM; ON-LINE SOCIAL NETWORKS; SIZE ESTIMATION; SMALL SAMPLES;

COMPUTER SIMULATION; FISHER INFORMATION MATRIX; POPULATION STATISTICS; SOCIAL NETWORKING (ONLINE);

ESTIMATION;

EID: 84884778591     PISSN: 10414347     EISSN: None     Source Type: Journal    
DOI: 10.1109/TKDE.2012.220     Document Type: Article

References (27)

1. S. Amstrup, T. McDonald, and B. Manly, Handbook of Capture-Recapture Analysis. Princeton Univ Press, 2005.
2. A. Broder et al., "Estimating Corpus Size via Queries," Proc. 15th ACM Int'l Conf. Information and Knowledge Management (CIKM), pp. 594-603, 2006. (Pubitemid 47203605)
3. A. Chao, S. Lee, and S. Jeng, "Estimating Population Size for Capture-Recapture Data When Capture Probabilities Vary by Time and Individual Animal," Biometrics, vol. 48, pp. 201-216, 1992. (Pubitemid 23401166)
4. D. Chapman, "Some Properties of the Hypergeometric Distribution with Applications to Zoological Sample Censuses," Univ. California Publications Statistics, vol. 1, pp. 131-59, 1951.
5. C.L. Chiang, "On the Expectation of the Reciprocal of a Random Variable," The Am. Statistician, vol. 20, no. 4, p. 28, 1966.
6. J. Darroch, "The Multiple-Recapture Census: I. Estimation of a Closed Population," Biometrika, vol. 45, nos. 3/4, pp. 343-359, 1958.
7. A. Dasgupta, G. Das, and H. Mannila, "A Random Walk Approach to Sampling Hidden Databases," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD), pp. 629-640, 2007. (Pubitemid 47630840)
8. A. Dasgupta, X. Jin, B. Jewell, N. Zhang, and G. Das, "Unbiased Estimation of Size and Other Aggregates over Hidden Web Databases," Proc. ACM Int'l Conf. Management of Data (SIGMOD), pp. 855-866, 2010.
9. P. Erdos and A. Rényi, "On the Evolution of Random Graphs," Publication of Math. Inst. of Hungarian Academy of Sciences, vol. 5, pp. 17-61, 1960.
10. W. Gale and G. Sampson, "Good-Turing Frequency Estimation without Tears," J. Quantitative Linguistics, vol. 2, no. 3, pp. 217-237, 1995.
11. M. Gjoka, M. Kurant, C. Butts, and A. Markopoulou, "A Walk in Facebook: Uniform Sampling of Users in Online Social Networks," Arxiv preprint arXiv:0906.0060, 2009.
12. P.J. Haas, J.F. Naughton, S. Seshadri, and L. Stokes, "Sampling-Based Estimation of the Number of Distinct Values of an Attribute," Proc. 21st Int'l Conf. Very Large Data Bases (VLDB), pp. 311-322, 1995.
13. L. Katzir, E. Liberty, and O. Somekh, "Estimating Sizes of Social Networks via Biased Sampling," Proc. 20th Int'l Conf. World Wide Web (WWW), pp. 597-606, 2011.
14. H. Kwak, C. Lee, H. Park, and S. Moon, "What is Twitter, a Social Network or a News Media," Proc. 19th Int'l Conf. World Wide Web (WWW), pp. 591-600, 2010.
15. S. Lawrence and C. Giles, "Searching the World Wide Web," Science, vol. 280, no. 5360, pp. 98-100, 1998. (Pubitemid 28169089)
16. J. Leskovec and C. Faloutsos, "Sampling from Large Graphs," Proc. 12th ACM Int'l Conf. Knowledge Discovery and Data Mining (SIGKDD), pp. 631-636, 2006. (Pubitemid 44535566)
17. L. Lovász, "Random Walks on Graphs: A Survey," Combinatorics, Paul Erdos is Eighty, vol. 2, no. 1, pp. 1-46, 1993.
18. J. Lu, "Ranking Bias in Deep Web Size Estimation Using Capture Recapture Method," Data and Knowledge Eng., vol. 69, no. 8, pp. 866-879, 2010.
19. J. Lu and D. Li, "Estimating Deep Web Data Source Size by Capture-Recapture Method," Information Retrieval, vol. 13, no. 1, pp. 70-95, 2010.
20. J. Lu and D. Li., "Sampling Online Social Networks by Random Walk," Proc. First ACM Int'l Workshop Hot Topics on Interdisciplinary Social Networks Research (SIGKDD), pp. 33-40, 2012.
21. N. Metropolis, A. Rosenbluth, M. Rosenbluth, A. Teller, and E. Teller, "Equation of State Calculations by Fast Computing Machines," The J. Chemical Physics, vol. 21, no. 6, pp. 1087-1092, 1953.
22. M. Newman, Networks: An Introduction. Oxford Univ. Press, Inc., 2010.
23. M. Salganik and D. Heckathorn, "Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling," Sociological Methodology, vol. 34, no. 1, pp. 193-240, 2004.
24. J. Wittes, "On the Bias and Estimated Variance of Chapman's Two-Sample Capture-Recapture Population Estimate," Biometrics, vol. 28, pp. 592-597, 1972.
25. S. Ye and S. Wu, "Estimating the Size of Online Social Networks," Int'l J. Social Computing and Cyber-Physical Systems, vol. 1, no. 2, pp. 160-179, 2011.
26. C. Zhai and J. Lafferty, "A Study of Smoothing Methods for Language Models Applied to Information Retrieval," ACM Trans. Information Systems, vol. 22, no. 2, pp. 179-214, 2004.
27. J. Zhou, Y. Li, V. Adhikari, and Z. Zhang, "Counting Youtube Videos via Random Prefix Sampling," Proc. ACM SIGCOMM, pp. 371-380, 2011.