Estimating deep web data source size by capture-recapture method

Information Retrieval

Volumn 13, Issue 1, 2010, Pages 70-95

  Lu, Jianguo ^a,b   Li, Dingding ^a  

^a UNIVERSITY OF WINDSOR   (Canada)

^b NANJING UNIVERSITY   (China)

Author keywords

Capture recapture; Deep web; Estimators

Indexed keywords

EID: 76349085418     PISSN: 13864564     EISSN: 15737659     Source Type: Journal    
DOI: 10.1007/s10791-009-9107-y     Document Type: Article

References (37)

1. Amstrup, S. C., McDonald, T. L., & Manly, B. F. J. (2005). Handbook of capture-recapture analysis. Princeton University Press.
2. Barbosa, L., & Freire, J. (2004). Siphoning hidden-web data through keyword-based interfaces. In Proceedings of SBBD, 2004.
3. Bar-Yossef, Z., & Gurevich, M. (2006). Random sampling from a search engine's index. In Proceedings of WWW, 2006, pp, 367-376.
4. Bar-Yossef, Z., & Gurevich, M. (2007). Efficient search engine measurements. In Proceedings of WWW, 2007, pp. 401-410.
5. Bergman, M. K. (2001). The deep web: Surfacing hidden value. The Journal of Electronic Publishing, 7(1).
6. Bharat, K., & Broder, A. (1998). A technique for measuring the relative size and overlap of public Web search engines. In Proceedings of WWW, 1998, pp. 379-388.
7. Bolshakov, I. A., & Galicia-Haro, S. N. (2003). Can we correctly estimate the total number of pages in Google for a specific language? CICLing 2003, pp. 415-419.
8. Broder, A., Fontura, M., Josifovski, V., Kumar, R., Motwani, R., Nabar, S., etal. (2006). Estimating corpus size via queries. In CIKM'06, pp. 594-603.
9. Callan, J., & Connell, M. (2001). Query-based sampling of text databases. ACM Transactions on Information Systems, 19(2), 97-130.
10. Caverlee, J., Liu, L., & Buttler, D. (2004). Probe, cluster, and discover: Focused extraction of QA-pagelets from the deep web. In Proceedings of ICDE 2004, pp. 103-114.
11. Chao, A., & Lee, S.-M. (1992). Estimating the number of classes via sample coverage. Journal of American Statistical Association, 87, 210-217.
12. Crescenzi, V., Mecca, G., & Merialdo, P. (2001). RoadRunner: Towards automatic data extraction from large web sites. In Proceedings of VLDB 2001, pp. 109-118.
13. Darroch, J. N. (1958). The multiple-recapture census: I. Estimation of a closed population. Biometrika, 45(3/4), 343-359.
14. Dobra, A., & Fienberg, S. (2004). How large is the World Wide Web? Web Dynamics, Springer, pp. 23-44.
15. Gulli, A., & Signorini A. (2005). The indexable web is more than 11.5 billion pages. In Proceedings of WWW 2005, pp. 902-903.
16. Haas, P. J., Naughton, J. F., Seshadri, S., & Stokes, L. (1995). Sampling-based estimation of the number of distinct values of an attribute. In Proceedings of VLDB 1995, pp. 311-322.
17. Hatcher, E., & Gospodnetic, O. (2004). Lucene in action. Manning Publications.
18. Holst, L. (1979). A unified approach to limit theorems for urn models. Journal of Applied Probability, 16(1), 154-162.
19. Ipeirotis, P. G., Gravano, L., & Sahami, M. (2001). Probe, count, and classify: Categorizing hidden web databases. In Proceedings of SIGMOD'01.
20. Knoblock, C. A., Lerman, K., Minton, S., & Muslea, I. (2000). Accurately and reliably extracting data from the web: A machine learning approach. IEEE Data Engineering Bulletin, 23(4), 33-41.
21. Lang, K. (1995). Newsweeder: Learning to filter netnews. In Twelfth international conference on machine learning, pp. 331-339.
22. Liddle, S. W., Embley, D. W., Scott, D. T., & Yau, S. H. (2002). Extracting data behind web forms, advanced conceptual modeling techniques, pp. 402-413.
23. Liu, K., Yu, C., & Meng, W. (2002). Discovering the representative of a search engine. In Proceedings of CIKM'02, pp. 652-654.
24. Lu, J. (2008). Efficient estimation of the size of text deep web data source. In Proceedings of CIKM 2008, pp. 1485-1486.
25. Lu, J., Wang, Y., Liang, J., Chen, J., & Liu, J. (2008). An approach to deep web crawling by sampling. In Proceedings of Web Intelligence, pp. 718-724.
26. Nelson, M. L., Smith, J. A., & del Campo, I. G. (2006). Efficient, automatic web resource harvesting. In Proceedings of WIDM'06, pp. 43-50.
27. Ntoulas, A., Zerfos, P., & Cho, J. (2005). Downloading textual hidden web content through keyword queries. In Proceedings of JCDL, 2005, pp. 100-109.
28. Pollock, K. H., Nichols, J. D., Brownie, C., & Hines, J. E. (1990). Statistical inference for capture crecapture experiments. The Wildlife Society. Wildlife Monographs, 107, 3-97.
29. Raghavan, S., & Garcia-Molina, H. (2001). Crawling the hidden web. Proceedings of VLDB 2001.
30. Schumacher, F. X., & Eschmeyer, R. W. (1943). The estimation of fish populations in lakes or ponds. Journal. Tennessee Academy of Science, 18, 228-249.
31. Shestakov, D., Bhowmick, S. S. & Lim, E.-P. (2005). DEQUE: Querying the deep web. Journal of Data & Knowledge Engineering, 52(3), 273-311.
32. Shokouhi, M., Zobel, J., & Scholer, F. (2006). SMM Tahaghoghi, capturing collection size for distributed non-cooperative retrieval. In Proceedings of SIGIR'06, pp. 316-323.
33. Si, L., & Callan, J. (2003). Relevant document distribution estimation method for resource selection. In Proceedings of SIGIR'03.
34. Thomas, P., & Hawking, D. (2007). Evaluating sampling methods for uncooperative collections. In Proceedings of SIGIR, 2007.
35. Wu, S., Gibb, F., & Crestani, F. (2003). Experiments with document archive size detection. 25th European conference on IR research, pp. 294-304.
36. Wu, P., Wen, J.-R., Liu, H., & Ma, W.-Y. (2006). Query selection techniques for efficient crawling of structured web sources. In Proceedings of ICDE, 2006, pp. 47-56.
37. Xu, J., Wu, S., & Li, X. (2007). Estimating collection size with logistic regression. In Proceedings of SIGIR'07, pp. 789-790.