Improving the performance of focused web crawlers

Data and Knowledge Engineering

Volumn 68, Issue 10, 2009, Pages 1001-1013

  Batsakis, Sotiris ^a   Petrakis, Euripides G M ^a   Milios, Evangelos ^b  

^a TECHNICAL UNIVERSITY OF CRETE   (Greece)

^b DALHOUSIE UNIVERSITY   (Canada)

Author keywords

Focused crawler; Hidden Markov Model (HMM) crawler; Learning crawler; World Wide Web

Indexed keywords

COMPARATIVE ANALYSIS; EVALUATION FRAMEWORK; FOCUSED CRAWLER; FOCUSED WEB CRAWLER; HIDDEN MARKOV MODEL (HMM) CRAWLER; LEARNING CRAWLER; PRIORITY ASSIGNMENT; WEB PAGE;

EDUCATION; OBJECT RECOGNITION; WORLD WIDE WEB;

HIDDEN MARKOV MODELS;

EID: 69249230866     PISSN: 0169023X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.datak.2009.04.002     Document Type: Article

References (34)

1. J. Cho, Crawling the Web: Discovery and Maintenance of a Large-Scale Web Data, Ph.D. Thesis, Stanford University, 2001.
2. S. Chakrabarti, M. van den Berg, B. Dom, Focused crawling: a new approach for topic specific resource discovery, in: Proceedings of the 8th International World Wide Web Conference (WWW8), 1999.
3. Menczer F., Pant G., and Srinivasan P. Topical web crawlers: evaluating adaptive algorithms. ACM Transactions on Internet Technology (TOIT) 4 4 (2004) 378-419
4. Salton G., Wong A., and Yang C.S. A vector space model for automatic indexing. Communications of the ACM 18 11 (1975) 613-620
5. M. Ehrig, A. Maedche, Ontology-focused crawling of web documents, in: Proceedings of the Symposium on Applied Computing (SAC 2003), March 9-12, 2003.
6. G. Varelas, E. Voutsakis, P. Raftopoulou, E.G.M. Petrakis, E. Milios, Semantic similarity methods in wordnet and their application to information retrieval on the web, in: Seventh ACM International Workshop on Web Information and Data Management (WIDM 2005), Bremen Germany, 2005.
7. A. Hliaoutakis, G. Varelas, E. Voutsakis, E.G.M. Petrakis, E. Milios, Information retrieval by semantic similarity, International Journal on Semantic Web and Information Systems (IJSWIS) 3 (3) (2006) 55-73 (Special issue of multimedia semantics).
8. Pant G., and Srinivasan P. Learning to crawl: comparing classification schemes. ACM Transactions on Information Systems (TOIS) 23 4 (2005) 430-462
9. Jun, Li, K. Furuse, K. Yamaguchi, Focused crawling by exploiting anchor text using decision tree, in: Proceedings of the 14th International World Wide Web Conference, 2005, pp. 1190-1191.
10. M. Diligenti, F. Coetzee, S. Lawrence, C. Giles, M. Gori, Focused crawling using context graphs, in: Proceedings of the 26th International Conference on Very Large Databases (VLDB 2000), 2000, pp. 527-534.
11. Liu H., Janssen J., and Milios E. Using HMM to learn user browsing patterns for focused web crawling. Data & Knowledge Engineering 59 2 (2006) 270-329
12. Y. Chen, A Novel Hybrid Focused Crawling Algorithm to Build Domain-Specific Collections, Ph.D. Thesis, Virginia Polytechnic Institute and State University, 2007.
13. Hersovici M., Jacovi M., Maarek Y.S., Pelleg D., Shtalhaim M., and Ur S. The Shark-search algorithm - an application:tailored web site mapping. Computer Networks and ISDN Systems 30 1-7 (1998) 317-326
14. S. Brin, L. Page, The anatomy of a large-scale hypertextual web search engine, in: Proceedings of the 7th International World Wide Web Conference, Brisbane, Australia, April 14-18, 1998, pp. 107-117.
15. R. Kraft, R. Stata, Finding buying guides with a web carnivore, in: First Latin American Web Congress (LA-WEB'03), 2003, pp. 84-92.
16. G. Pant, K. Tsjoutsiouliklis, J. Johnson, C.L. Giles, Panorama: extending digital libraries with topical crawlers, in: Proceedings of the 4th ACM/IEEE-CS Joint Conference on Digital Libraries, 2004b, pp. 142-150.
17. P. de Bra, G. Houben, Y. Kornatzky, R. Post, Information retrieval in distributed hypertexts, in: Proceedings of RIAO'94, Intelligent Multimedia, Information Retrieval Systems and Management, New York, 1994, pp. 481-491.
18. C. Aggarwal, F. Al-Garawi, P. Yu, Intelligent crawling on the world wide web with arbitrary predicates, in: Proceedings of the 10th International World Wide Web Conference, 2001, pp. 96-105.
19. C. Corley, R. Mihalcea, Measuring the semantic similarity of texts, in: Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, MI, June 2005, pp. 13-18.
20. A. Badia, T. Muezzinoglu, O. Nasraoui, Focused crawling: experiences in a real world project, in: Proceedings of the 15th International Conference on World Wide Web, Edinburgh, 2006, pp. 1043-1044.
21. Qingyang Xu, Wanli. Zuo, First-order focused crawling, in: Proceedings of the 16th International Conference on World Wide Web, Banff, Alberta, Canada, 2007, pp. 1159-1160.
22. Pant G., and Srinivasan P. Link contexts in classifier-guided topical crawlers. IEEE Transactions on Knowledge and Data Engineering 18 01 (2006) 107-122
23. D. Bergmark, C. Lagoze, A. Sbityakov, Focused crawls,tunneling, and digital libraries, in: Sixth European Conference on Digital Libraries, Rome, 2002.
24. S. Chakrabarti, K. Punera, M. Subramanyam, Accelerated focused crawling through online relevance feedback, in: Proceedings of the 11th International Conference on World Wide Web (WWW2002), 2002, pp. 148-159.
25. D. Bergmark, Collection synthesis, in: Proceedings of the Second ACM/IEEE-CS Joint Conference on Digital Libraries, Portland OR, 2002.
26. Porter M.F. An algorithm for suffix stripping. Program 14 3 (1980) 130-137
27. M. Steinbach, G. Karypis, V. Kumar, A comparison of document clustering techniques, in: Sixth ACM SIGKDD, World Text Mining Conference, Boston, MA, 2000.
28. Rabiner L., and Juang B. An introduction to hidden Markov models. ASSP Magazine, IEEE 3 1 (1986) 4-16 (Part 1)
29. Cover T., and Hart P. Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13 1 (1967) 21-27
30. H. Liu, E. Milios, J. Janssen, Probabilistic models for focused web crawling, in: Proceedings of Sixth ACM International Workshop on Web Information and Data Management (WIDM 2004), 2004, pp. 16-22.
31. Bao S., Li R., Yu Y., and Cao Y. Competitor mining with the web knowledge. IEEE Transactions on Data Engineering 20 10 (2008) 1297-1310
32. F. McCown, M. Nelson, Agreeing to disagree: search engines and their public interfaces, in: ACM IEEE Joint Conference on Digital Libraries (JCDL 2007), Vancouver, British Columbia, Canada, June 17-23, 2007, pp. 309-318.
33. Pivk A., Cimiano P., Sure Y., Gams M., Rajkovic V., and Studer R. Transforming arbitrary tables into logical form with TARTAR. Data & Knowledge Engineering 60 3 (2007) 567-595
34. C. Chang, M. Kayed, M.R. Girgis, K.F. Shaalan, A survey of web information extraction systems, in: IEEE Transactions on Knowledge and Data Engineering, TKDE-0475-1104.R3, 2006.