QProber: A system for automatic classification of hidden-Web databases

ACM Transactions on Information Systems

Volumn 21, Issue 1, 2003, Pages 1-41

  Gravano, Luis ^a,c   Ipeirotis, Panagiotis G ^a,c   Sahami, Mehran ^b,d  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b STANFORD UNIVERSITY   (United States)

^c Columbia University ^*  (United States)

^d Stanford University   (United States)

Author keywords

Database classification; Hidden Web; Web databases

Indexed keywords

DATABASE CLASSIFICATION; HIDDEN WEB; QPROBER; WEB DATABASES;

ALGORITHMS; AUTOMATION; CODES (SYMBOLS); DATABASE SYSTEMS; HIERARCHICAL SYSTEMS; INFORMATION RETRIEVAL; MATHEMATICAL MODELS; QUERY LANGUAGES;

WEBSITES;

EID: 0344127987     PISSN: 10468188     EISSN: None     Source Type: Journal    
DOI: 10.1145/635484.635485     Document Type: Article

References (56)

1. AGICHTEIN, E. AND GRAVANO, L. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of the Fifth ACM Conference on Digital Libraries (DL 2000).
2. AGICHTEIN, E. AND GRAVANO, L. 2003. Querying text databases for efficient information extraction. In Proceedings of the Nineteenth IEEE International Conference on Data Engineering (ICDE 2003).
3. AGRAWAL, R. AND SRIKANT, R. 1994. Fast algorithms for mining association rules. In Proceedings of the Twentieth International Conference on Very Large Databases (VLDB'94), 487-499.
4. APTE, C., DAMERAU, F., AND WEISS, S. M. 1994. Automated learning of decision rules for text categorization. ACM Trans. Inf. Syst. 12, 3, 233-251.
5. BURGES, C. J. 1998. A tutorial on support vector machines for pattern recognition. Data Mining Knowl. Discov. 2, 2 (June), 121-167.
6. CALLAN, J. AND CONNELL, M. 2001. Query-based sampling of text databases. ACM Trans. Inf. Syst. 19, 2, 97-130.
7. CALLAN, J. P., CONNELL, M., AND DU, A. 1999. Automatic discovery of language models for text databases. In Proceedings of the 1999 ACM SIGMOD International Conference on Management of Data (SIGMOD'99), 479-490.
8. CLEVERDON, C. W. AND MILLS, J. 1963. The testing of index language devices. Aslib Proc. 15, 4, 106-130.
9. COHEN, W. AND SINGER, Y. 1996. Learning to query the Web. In AAAI Workshop on Internet-Based Information Systems, 16-25.
10. COHEN, W. W. 1996. Learning trees and rules with set-valued features. In Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96) Eighth Conference on Innovative Applications of Artificial Intelligence (IAAI-96), 709-716.
11. CRASWELL, N., BAILEY, P., AND HAWKING, D. 2000. Server selection on the World Wide Web. In Proceedings of the Fifth ACM Conference on Digital Libraries (DL 2000), 37-46.
12. CRAVEN, M. 1996. Extracting comprehensible models from trained neural networks. PhD Thesis, University of Wisconsin-Madison, Department of Computer Sciences. Also appears as UW Tech. Rep. CS-TR-96-1326.
13. DEERWESTER, S. C., DUMAIS, S. T., LANDAUER, T. K., FURNAS, G. W., AND HARSHMAN, R. A. 1990. Indexing by latent semantic analysis. J. Amer. Soc. Inf. Sci. 41, 6, 391-407.
14. DOLIN, R., AGRAWAL, D., AND EL ABBADI, A. 1999. Scalable collection summarization and selection. In Proceedings of the Fourth ACM International Conference on Digital Libraries (DL'99), 49-58.
15. DUDA, R. O. AND HART, P. E. 1973. Pattern Classification and Scene Analysis. Wiley, New York.
16. DUMAIS, S. T., PLATT, J., HECKERMAN, D., AND SAHAMI, M. 1998. Inductive learning algorithms and representations for text categorization. In Proceedings of the 1998 ACM CIKM International Conference on Information and Knowledge Management, 148-155.
17. FLAKE, G., GLOVER, E., LAWRENCE, S., AND GILES, C. 2002. Extracting query modifications from non-linear SVMs. In Proceedings of the Eleventh International World Wide Web Conference (WWW11).
18. GAUCH, S., WANG, G., AND GOMEZ, M. 1996. ProFusion*: Intelligent fusion from multiple, distributed search engines. J. Univ. Comput. Sci. 2, 9 (Sept.), 637-649.
19. GHANI, R., JONES, R., AND MLADENIC, D. 2001. Using the Web to create minority language corpora. In Proceedings of the 2001 ACM CIKM International Conference on Information and Knowledge Management, 279-286.
20. GRAVANO, L., GARCÍA-MOLINA, H., AND TOMASIC, A. 1999. GlOSS: Text-source discovery over the Internet. ACM Trans. Database Syst. 24, 2 (June), 229-264.
21. GRAVANO, L., IPEIROTIS, P. G., AND SAHAMI, M. 2002. Query- vs. crawling-based classification of searchable web databases. IEEE Data Eng. Bull. 25, 1 (Mar.), 43-50.
22. GREFENSTETTE, G. AND NIOCHE, J. 2000. Estimation of English and non-English language use on the WWW. In Recherche d'Information Assistée par Ordinateur (RIAO 2000).
23. HAWKING, D. AND THISTLEWAITE, P. B. 1999. Methods for information server selection. ACM Trans. Inf. Syst. 17, 1 (Jan.), 40-76.
24. IPEIROTIS, P. G. AND GRAVANO, L. 2002. Distributed search over the hidden Web: Hierarchical database sampling and selection. In Proceedings of the 28th International Conference on Very Large Databases (VLDB 2002).
25. IPEIROTIS, P. G., GRAVANO, L., AND SAHAMI, M. 2001a. PERSIVAL demo: Categorizing hidden-Web resources. In Proceedings of the First ACM+IEEE Joint Conference on Digital Libraries (JCDL 2001), 454.
26. IPEIROTIS, P. G., GRAVANO, L., AND SAHAMI, M. 2001b. Probe, count, and classify: Categorizing hidden-Web databases. In Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data (SIGMOD 2001), 67-78.
27. JOACHIMS, T. 1998. Text categorization with support vector machines: Learning with many relevant features. In ECML-98, Tenth European Conference on Machine Learning, 137-142.
28. JOHNSTON, R. 1971. Gershgorin theorems for partitioned matrices. Lin. Algeb. Appl. 4, 3 (July), 205-220.
29. KOHAVI, R. AND JOHN, G. H. 1997. Wrappers for feature subset selection. Artif. Intell. 97, 1-2, (special issue on Relevance), 273-323.
30. KOHAVI, R. AND PROVOST, F. 1998. Glossary of terms. J. Mach. Learn. 30, 2/3, 271-274. Editorial for the special issue on Applications of Machine Learning and the Knowledge Discovery Process.
31. KOLLER, D. AND SAHAMI, M. 1996. Toward optimal feature selection. In Proceedings of the Thirteenth International Conference on Machine Learning (ICML'96), 284-292.
32. KOLLER, D. AND SAHAMI, M. 1997. Hierarchically classifying documents using very few words. In Proceedings of the Fourteenth International Conference on Machine Learning (ICML'97), 170-178.
33. KOSTER, M. 2002. Robots exclusion standard. Available at http://www.robotstxt.org/.
34. LEWIS, D. D., SCHAPIRE, R. E., CALLAN, J. P., AND PAPKA, R. 1996. Training algorithms for linear text classifiers. In Proceedings of the Nineteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'96, 298-306.
35. McCALLUM, A. AND NIGAM, K. 1998. A comparison of event models for naive Bayes text classification. In Learning for Text Categorization: Papers from the 1998 AAAI Workshop, 41-48.
36. MENG, W., LIU, K.-L., YU, C. T., WANG, X., CHANG, Y., AND RISHE, N. 1998. Determining text databases to search in the Internet. In Proceedings of the 24th International Conference on Very Large Databases (VLDB'98), 14-25.
37. MENG, W., Yu, C. T., AND LIU, K.-L. 1999. Detection of heterogeneities in a multiple text database environment. In Proceedings of the Fourth IFCIS International Conference on Cooperative Information Systems (CoopIS 1999), 22-33.
38. MITCHELL, T. M. 1997. Machine Learning. McGraw-Hill, New York.
39. NILSSON, N. J. 1990. The Mathematical Foundations of Learning Machines. Morgan-Kaufmann, San Francisco. Previously published as: Learning Machines, 1965.
40. NILSSON, N. J. 1990. The Mathematical Foundations of Learning Machines. Morgan-Kaufmann, San Francisco. Previously published as: Learning Machines, 1965.
41. PERKOWITZ, M., DOORENBOS, R. B., ETZIONI, O., AND WELD, D. S. 1997. Learning to understand information on the Internet: An example-based approach. J. Intell. Inf. Syst. 8, 2 (Mar.), 133-153.
42. QUINLAN, J. 1992. C4.5: Programs for Machine Learning. Morgan-Kaufmann, San Francisco.
43. RAGHAVAN, S. AND GARCÍA-MOLINA, H. 2001. Crawling the hidden Web. In Proceedings of the 27th International Conference on Very Large Databases (VLDB 2001), 129-138.
44. ROCCHIO, J. 1971. Relevance feedback in information retrieval. In The SMART Information Retrieval System. Prentice-Hall, Englewood Cliffs, NJ, 313-323.
45. SAHAMI, M. 1998. Using machine learning to improve information access. PhD Thesis, Stanford University, Computer Science Department.
46. SALTON, G. AND BUCKLEY, C. 1988. Term-weighting approaches in automatic text retrieval. Inf. Process. Manage. 24, 513-523.
47. SALTON, G. AND MCGILL, M. J. 1983. Introduction to Modern Information Retrieval. McGraw-Hill, New York.
48. SALTON, G. AND MCGILL, M. J. 1997. The SMART and SIRE experimental retrieval systems. In Readings in Information Retrieval. Morgan-Kaufmann, San Francisco, 381-399.
49. SCHUETZE, H., HULL, D., AND PEDERSEN, J. 1995. A comparison of document representations and classifiers for the routing problem. In Proceedings of the Eighteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'95, 229-237.
50. SUGIURA, A. AND ETZIONI, O. 2000. Query routing for Web search engines: Architecture and experiments. In Proceedings of the Ninth International World Wide Web Conference (WWW9).
51. VAN RIJSBERGEN, K. 1979. Information Retrieval (2nd edition). Butterworths, London.
52. WANG, W., MENG, W., AND YU, C. 2000. Concept hierarchy based text database categorization in a metasearch engine environment. In Proceedings of the First International Conference on Web Information Systems Engineering (WISE 2000), 283-290.
53. XU, J. AND CALLAN, J. P. 1998. Effective retrieval with distributed collections. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'98, 112-120.
54. YANG, Y. AND LIU, X. 1999. A re-examination of text categorization methods. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'99, 42-49.
55. YANGARBER, R. AND GRISHMAN, R. 1998. NYU: Description of the Proteus/PET system as used for MUC-7. In Proceedings of the Seventh Message Understanding Conference (MUC-7).
56. ZIPF, G. K. 1949. Human Behavior and the Principle of Least Effort. Addison-Wesley, Reading, MA.