Classification-aware hidden-web text database selection

ACM Transactions on Information Systems

Volumn 26, Issue 2, 2008, Pages

  Ipeirotis, Panagiotis G ^a   Gravano, Luis ^b,c  

^a NEW YORK UNIVERSITY   (United States)

^b Och Spine at New York Presbyterian Hospitals

^c New York University ^*  (United States)

Author keywords

Database selection; Distributed information retrieval; Web search

Indexed keywords

DATABASE SELECTION; DISTRIBUTED INFORMATION RETRIEVAL; WEB SEARCH;

CLASSIFICATION (OF INFORMATION); QUERY PROCESSING; STATISTICAL METHODS; TEXT PROCESSING; VOCABULARY CONTROL; WEB SERVICES;

DATABASE SYSTEMS;

EID: 42049120674     PISSN: 10468188     EISSN: 15582868     Source Type: Journal    
DOI: 10.1145/1344411.1344412     Document Type: Article

References (77)

1. ADAMIC, L. A. 2002. Zipf, power-laws, and Pareto - A ranking tutorial, http://ginger.hpl.hp.com/shl/papers/ranking/ranking.html.
2. AGICHTEIN, E. AND GRAVANO, L. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of the 5th ACM Conference on Digital Libraries (DL).
3. AGICHTEIN, E. AND GRAVANO, L. 2003. Querying text databases for efficient information extraction. In Proceedings of the 19th IEEE International Conference on Data Engineering (ICDE).
4. BAAYEN, R. H. 2006. Word Frequency Distributions. Springer.
5. BAUMGARTEN, C. 1999. A probabilistic solution to the selection and fusion problem in distributed information retrieval. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 246-253.
6. BAUMGARTEN, C. 1997. A probabilistic model for distributed information retrieval. In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 258-266.
7. BERGHOLZ, A. AND CHIDLOVSKII, B. 2004. Using query probing to identify query language features on the Web. In Proceedings of the Distributed Multimedia Information Retrieval, SIGIR 2003 Workshop on Distributed Information Retrieval, Revised Selected and Invited Papers. Lecture Notes in Computer Science, vol. 2924, Springer, 21-30.
8. BERGMAN, M. K. 2001. The deep Web: Surfacing hidden value. J. Electron. Publ. 7, 1 (Aug.).
9. BERNERS-LEE, T., HENDLER, J., AND LASSILA, O. 2001. The semantic Web. Sci. Amer. 284, 5 (May), 34-43.
10. CALLAN, J. P. AND CONNELL, M. 2001. Query-Based sampling of text databases. ACM Trans. Inf. Syst. 19, 2, 97-130.
11. CALLAN, J. P., CONNELL, M., AND DU, A. 1999. Automatic discovery of language models for text databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD), 479-490.
12. CALLAN, J. P., LU, Z., AND CROFT, W. B. 1995. Searching distributed collections with inference networks. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 21-28.
13. CHAKRAVARTHY, A. S. AND HAASE, JR., K. W. 1995. Netserf: Using semantic knowledge to find Internet information archives. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 4-11.
14. CHOI, Y. S. AND YOO, S. I. 2001. Text database discovery on the Web: Neural net based approach. J. Intell. Inf. Syst. 16, 1 (Jan.), 5-20.
15. COHEN, W. W. 1996. Learning trees and rules with set-valued features. In Proceedings of the 13th National Conference on Artificial Intelligence (AAAI), 8th Conference on Innovative Applications of Artificial Intelligence (IAAI), 709-716.
16. COHEN, W. W. AND SINGER, Y. 1996. Learning to query the Web. In Proceedings of the AAAI Workshop on Internet-Based Information Systems, 16-25.
17. CRASWELL, N., BAILEY, P., AND HAWKING, D. 2000. Server selection on the World Wide Web. In Proceedings of the 5th ACM Conference on Digital Libraries (DL). 37-46.
18. CROFT, W. B. AND LAFFERTY, J. 2003. Language Modeling for Information Retrieval. Kluwer Academic.
19. DEMPSTER, A. P., LAIRD, N. M., AND RUBIN, D. B. 1977. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Statis. Soc. B, 39, 1-38.
20. DOLIN, R. A. 1998. Pharos: A scalable distributed architecture for locating heterogeneous information sources. Ph.D. thesis, University of California, Santa Barbara.
21. DREILINGER, D. AND HOWE, A. E. 1997. Experiences with selecting search engines using metasearch. ACM Trans. Inf. Syst. 15, 3, 195-222.
22. DUDA, R. O., HART, P. E., AND STORK, D. G. 2000. Pattern Classification, 2nd ed. Wiley.
23. FELLBAUM, C. 1998. WordNet: An Electronic Lexical Database. MIT Press.
24. FLAKE, G., GLOVER, E., LAWRENCE, S., AND GILES, C. L. 2002. Extracting query modifications from nonlinear SVMs. In Proceedings of the 11th International World Wide Web Conference (WWW).
25. FRENCH, J. C., POWELL, A. L., CALLAN, J. P., VILES, C. L., EMMITT, T., PREY, K. J., AND MOU, Y. 1999. Comparing the performance of database selection algorithms. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 238-245.
26. FRENCH, J. C., POWELL, A. L., VILES, C. L., EMMITT, T., AND PREY, K. J. 1998. Evaluating database selection techniques: A testbed and experiment. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 121129.
27. FUHR, N. 1999. A decision-theoretic approach to database selection in networked IR. ACM Trans. Inf. Syst. 17, 3 (May), 229-249.
28. GAUCH, S., WANG, G., AND GOMEZ, M. 1996. ProFusion*: Intelligent fusion from multiple, distributed search engines. J. Universal Comput. Sci. 2, 9 (Sept.), 637-649.
29. GHANI, R., JONES, R., AND MLADENIC, D. 2001. Using the Web to create minority language corpora. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM), 279-286.
30. GRAVANO, L., IPEIROTIS, P. G., AND SAHAMI, M. 2003. QProber: A system for automatic classification of hidden-Web databases. ACM Trans. Inf. Syst. 21, 1 (Jan.), 1-41.
31. GRAVANO, L., GARCÍA-MOLINA, H., AND TOMASIC, A. 1999. GlOSS: Text-Source discovery over the Internet. ACM Trans. Database Syst. 24, 2 (Jun.), 229-264.
32. GRAVANO, L., CHANG, K. C-C., GARCÍA- MOLINA, H., AND PAEPCKE, A. 1997. STARTS: Stanford proposal for Internet meta-searching. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD), 207-218.
33. GREFENSTETTE, G. AND NIOCHE, J. 2000. Estimation of English and non-English language use on the WWW. In Recherche d'Information Assistée par Ordinateur (RIAO).
34. HARMAN, D. 1996. Overview of the fourth text retrieval conference (TREC-4). In NIST Special Publication 500-236: The 4th Text REtrieval Conference (TREC-4), 1-24.
35. HASTIE, T., TIBSHIRANI, R., AND FRIEDMAN, J. H. 2001. The Elements of Statistical Learning. Springer.
36. HAWKING, D. AND THOMAS, P. 2005. Server selection methods in hybrid portal search. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 75-82.
37. HAWKING, D. AND THISTLEWAITE, P. B. 1999. Methods for information server selection. ACM Trans. Inf. Syst. 17, 1 (Jan.), 40-76.
38. IPEIROTIS, P. G. AND GRAVANO, L. 2004. When one sample is not enough: Improving text database selection using shrinkage. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD), 767-778.
39. IPEIROTIS, P. G. AND GRAVANO, L. 2002. Distributed search over the hidden Web: Hierarchical database sampling and selection. In Proceedings of the 28th International Conference on Very Large Databases (VLDB), 394-405.
40. JELINEK, F. 1999. Statistical Methods for Speech Recognition. MIT Press.
41. JOACHIMS, T. 1998. Text categorization with support vector machines: Learning with many relevant features. In Proceedings of the 10th European Conference on Machine Learning (ECML), 137-142.
42. KAHLE, B., MORRIS, H., GOLDMAN, J., ERICKSON, T., AND CURRAN, J. 1993. Interfaces for distributed systems of information servers. J. Amer. Soc. Inf. Sci. 44, 8 (Sept.), 453-467.
43. LARREY, L. S., CONNELL, M. E., AND CALLAN, J. P. 2000. Collection selection and results merging with topically organized U.S. patents and TREC data. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM), 282-289.
44. LIU, Z., LUO, C., CHO, J., AND CHU, W. 2004. A probabilistic approach to metasearching with adaptive probing. In Proceedings of the 20th IEEE International Conference on Data Engineering (ICDE), 547-559.
45. MANBER, U. AND BIGOT, P. A. 1997. The search broker. In 1st USENIX Symposium on Internet Technologies and Systems (USITS).
46. MANDELBROT, B. B. 1988. Fractal Geometry of Nature. W. H. Freeman.
47. MARQUES DE SÁ, J. P. 2003. Applied Statistics. Springer.
48. MCCALLUM, A., ROSENFELD, R., MITCHELL, T. M., AND NG, A. Y. 1998. Improving text classification by shrinkage in a hierarchy of classes. In Proceedings of the 15th International Conference on Machine Learning (ICML), 359-367.
49. MENG, W., YU, C. T., AND LIU, K.-L. 1999. Detection of heterogeneities in a multiple text database environment. In Proceedings of the 4th IFCIS International Conference on Cooperative Information Systems (CoopIS), 22-33.
50. MENG, W., LIU, K.-L., YU, C. T., WANG, X., CHANG, Y., AND RISHE, N. 1998. Determining text databases to search in the Internet. In Proceedings of the 24th International Conference on Very Large Databases (VLDB), 14-25.
51. MONROE, G. A., FRENCH, J. C., AND POWELL, A. L. 2002. Obtaining language models of Web collections using query-based sampling techniques. In Proceedings of the 35th Annual Hawaii International Conference on System Sciences (HICSS), 67-73.
52. PERKOWITZ, M., DOORENBOS, R. B., ETZIONI, O., AND WELD, D. S. 1997. Learning to understand information on the Internet: An example-based approach. J. Intell. Inf. Syst. 8, 2 (Mar.), 133-153.
53. POWELL, A. L. AND FRENCH, J. C. 2003. Comparing the performance of collection selection algorithms. ACM Trans. Inf. Syst. 21, 4 (Oct.), 412-456.
54. POWELL, A. L., FRENCH, J. C., CALLAN, J. P., CONNELL, M., AND VILES, C. L. 2000. The impact of database selection on distributed searching. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 232-239.
55. QUINLAN, J. R. 1992. C4.5: Programs for Machine Learning. Morgan Kaufmann.
56. RAGHAVAN, S. AND GARCÍA- MOLINA, H. 2001. Crawling the hidden web. In Proceedings of the 27th International Conference on Very Large Databases (VLDB), 129-138.
57. SALTON, G. A. AND MCGILL, M. J. 1983. Introduction to Modern Information Retrieval. McGraw-Hill.
58. SHELDON, M. A. 1995. Content routing: A scalable architecture for network-based information discovery. Ph.D. thesis, Massachusetts Institute a Technology.
59. SHOKOUHI, M. 2007. Central-Rank-Based collection selection in uncooperative distributed information retrieval. In Proceedings of the 29th European Conference on IR Research (ECIR).
60. SI, L. AND CALLAN, J. 2005. Modeling search engine effectiveness for federated search. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 83-90.
61. SI, L. AND CALLAN, J. 2004a. Unified utility maximization framework for resource selection. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM), 32-41.
62. SI, L. AND CALLAN, J. P. 2004b. The effect of database size distribution on resource selection algorithms. In Proceedings of the Distributed Multimedia Information Retrieval, SIGIR 2003 Workshop on Distributed Information Retrieval, Revised Selected and Invited Papers. Lecture Notes in Computer Science, vol. 2924, Springer, 31-42.
63. SI, L. AND CALLAN, J. P. 2003. Relevant document distribution estimation method for resource selection. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 298-305.
64. SI, L., JIN, R., CALLAN, J. P., AND OGILVIE, P. 2002. A language modeling framework for resource selection and results merging. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM), 391-397.
65. SUGIURA, A. AND ETZIONI, O. 2000. Query routing for web search engines: Architecture and experiments. In Proceedings of the 9th International World Wide Web Conference (WWW).
66. VOORHEES, E. M., GUPTA, N. K., AND JOHNSON-LAIRD, B. 1995. Learning collection fusion strategies. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 172-179.
67. VOORHEES, E. M. AND HARMAN, D. 1998. Overview of the sixth text retrieval conference (TREC-6). InNIST Special Publication 500-240: The 6th Text REtrieval Conference (TREC-6), 1-24.
68. XU, J. AND CALLAN, J. P. 1998. Effective retrieval with distributed collections. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 112-120.
69. XU, J. AND CROFT, W. B. 1999. Cluster-Based language models for distributed retrieval. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 254-261.
70. YANGARBER, R. AND GRISHMAN, R. 1998. NYU: Description of the Proteus/PET system as used for MUC-7. In Proceedings of the 7th Message Understanding Conference (MUC).
71. YU, C. T., MENG, W., WU, W., AND LIU, K.-L. 2001. Efficient and effective metasearch for text databases incorporating linkages among documents. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD).
72. YU, C. T., MENG, W., LIU, K.-L., WU, W., AND RISHE, N. 1999. Efficient and effective metasearch for a large number of text databases. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM), 217-224.
73. YUWONO, B. AND LEE, D. L. 1997. Server ranking for distributed text retrieval systems on the internet. In Proceedings of the 5th International Conference on Database Systems for Advanced Applications (DASFAA), 41-50.
74. ZHAI, C. AND LAFFERTY, J. D. 2004. A study of smoothing methods for language models applied to information retrieval. ACM Trans. Inf. Syst. 22, 2 (Apr.), 179-214.
75. ZHAI, C. AND LAFFERTY, J. D. 2002. Two-Stage language models for information retrieval. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 49-56.
76. ZHAI, C. AND LAFFERTY, J. D. 2001. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 334-342.
77. ZIPF, G. K. 1949. Human Behavior and the Principle of Least Effort. Addison-Wesley.