Ranking bias in deep web size estimation using capture recapture method

Data and Knowledge Engineering

Volumn 69, Issue 8, 2010, Pages 866-879

  Lu, Jianguo ^a  

^a UNIVERSITY OF WINDSOR   (Canada)

Author keywords

Capture recapture; Deep web; Estimators; Ranked data source

Indexed keywords

CAPTURE-RECAPTURE; CAPTURE-RECAPTURE METHOD; DATA SOURCE; DEEP WEB; DIRECT IMPACT; ESTIMATION METHODS; INITIAL ESTIMATION; OVERFLOW RATE; SAMPLING PROCESS; SIZE ESTIMATION;

ESTIMATION;

EID: 77955684574     PISSN: 0169023X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.datak.2010.03.007     Document Type: Article

References (39)

1. S.C. Amstrup, T.L. McDonald, and B.F.J. Manly Handbook of Capture-Recapture Analysis 2005 Princeton University Press
2. L. Barbosa, and J. Freire Siphoning Hidden-Web Data through Keyword-based Interfaces, SBBD 2004
3. I. Anagnostopoulos, and P. Stavropoulos Adopting wildlife experiments for web evolution estimations: the role of an AI web page classifier Proceedings of the 2006 IEEE/WIC/ACM international Conference on Web intelligence (December 18-22, 2006) 2006 Web Intelligence. IEEE Computer Society 897 901
4. Z. Bar-Yossef, and M. Gurevich Random Sampling from a Search Engine's Index. WWW 2006 367 376
5. Z. Bar-Yossef, and M. Gurevich Efficient Search Engine Measurements. WWW 2007 401 410
6. Ziv Bar-Yossef Maxim Gurevich: random sampling from a search engine's index Journal of the ACM 55 5 2008 1 74
7. Michael K. Bergman The deep web: surfacing hidden value The Journal of Electronic Publishing 7 1 2001
8. K. Bharat, and A. Broder A Technique for Measuring the Relative Size and Overlap of Public Web Search Engines. WWW 1998 379 388
9. Igor A. Bolshakov, and Sofia N. Galicia-Haro Can We Correctly Estimate the Total Number of Pages in Google for a Specific Language? CICLing 2003 415 419
10. A. Broder, M. Fontura, V. Josifovski, R. Kumar, R. Motwani, S. Nabar, R. Panigrahy, A. Tomkins, and Y. Xu Estimating Corpus Size via Queries. CIKM '06 2006 594 603
11. J. Callan, and M. Connell Query-based sampling of text databases ACM Transactions on Information Systems 2001 97 130
12. James Caverlee, Ling Liu, and David Buttler Probe, Cluster, and Discover: Focused Extraction of QA-Pagelets from the Deep Web, ICDE 2004 103 114
13. A. Chao, and S.-M. Lee Estimating the number of classes via sample coverage Journal of American Statistical Association 87 1992 210 217
14. J.N. Darroch The multiple-recapture census: I. estimation of a closed population Biometrika 45 3/4 Dec., 1958 343 359
15. A. Dobra, and S. Fienberg How large is the World Wide Web? Web Dynamics 2004 23 44
16. Peter J. Haas, Jeffrey F. Naughton, and S. Seshadri Lynne Stokes: Sampling-based Estimation of the Number of Distinct Values of an Attribute. VLDB 1995 311 322
17. E. Hatcher, and O. Gospodnetic Lucene in Action 2004 Manning Publications
18. P.G. Ipeirotis, L. Gravano, and M. Sahami Probe, Count, and Classify: Categorizing Hidden Web Databases. SIGMOD 2001
19. Stephen W. Liddle, David W. Embley, Del T. Scott, and Sai Ho. Yau Extracting data behind web forms Advanced Conceptual Modeling Techniques 2002 402 413
20. K. Liu, C. Yu, and W. Meng Discovering the representative of a search engine Proceedings of the ACM Conference on Information and Knowledge Management 2002 652 654
21. Jianguo Lu Efficient Estimation of the Size of Text Deep Web Data Source, Proceedings of the ACM Conference on Information and Knowledge Management 2008 1485 1486
22. Jianguo Lu, and Dingding Li Estimating deep web data source size by capture-recapture method Information Retrieval, Springer 13 1 2010 70 95
23. Jianguo Lu, Yan Wang, Jie Liang, Jessica Chen, and Jiming Liu An Approach to Deep Web Crawling by Sampling, IEEE / WIC / ACM International Conference on Web Intelligence 2008 718 724
24. Jayant Madhavan, David Ko, Lucja Kot, Vignesh Ganapathy, Alex Rasmussen, and Alon Halevy Google's Deep-Web Crawl. VLDB 2008 1241 1252
25. Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho Downloading Textual Hidden Web Content through Keyword Queries. Proceedings of the 5 th ACM/IEEE-CS Joint Conference on Digital Libraries 2005 100 109
26. K.H. Pollock, J.D. Nichols, C. Brownie, and J.E. Hines Statistical inference for capture-recapture experiments Wildlife Monographs 1990 107
27. S. Raghavan, and H. Garcia-Molina Crawling the Hidden Web, VLDB 2001
28. H. Scott, and C. Wohlin Capture-recapture in software unit testing: a case study Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (Kaiserslautern, Germany, October 09-10, 2008). ESEM '08 2008 ACM New York, NY 32 40
29. Denis Shestakov, Sourav S. Bhowmick, and Ee-Peng Lim DEQUE: querying the deep web Journal of Data and Knowledge Engineering 52 3 2005 273 311
30. M. Shokouhi, J. Zobel, F. Scholer, and S.M.M. Tahaghoghi Capturing Collection Size for Distributed Non-cooperative Retrieval, SIGIR 2006 316 323
31. F.X. Schumacher, and R.W. Eschmeyer The estimation of fish populations in lakes or ponds Journal of the Tennesse Academy of Science 18 1943 228 249
32. L. Si, and J. Callan Relevant Document Distribution Estimation Method for Resource Selection. SIGIR 2003
33. Paul Thomas David Hawking 2007 Evaluating Sampling Methods for Uncooperative Collections SIGIR
34. Paul Thomas Generalising Multiple Capture-Recapture to Non-uniform Sample Sizes. SIGIR 2008 839 840
35. R. Weaver, and M.P. Collins Fishing for Phishes: Applying Capture-Recapture Methods to Estimate Phishing Populations Proceedings of the Anti-Phishing Working Groups 2nd Annual Ecrime Researchers Summit (Pittsburgh, Pennsylvania, October 04-05, 2007). eCrime vol. 269 2007 ACM New York, NY 14 25
36. S. Wu, F. Gibb, and F. Crestani Experiments with document archive size detection 25th European Conference on IR Research 2003 294 304
37. Ping Wu, Ji-Rong Wen, Huan Liu, and Wei-Ying Ma Query Selection Techniques for Efficient Crawling of Structured Web Sources, ICDE 2006 47 56
38. J. Xu, S. Wu, and X. Li Estimating Collection Size with Logistic Regression. SIGIR 2007 789 790
39. F. Yan, W.C. Hou, Z. Jiang, C. Luo, and Q. Zhu Selectivity estimation of range queries based on data density approximation via cosine series Data and Knowledge Engineering 63 3 December 2007 855 878