Adaptive focused crawling

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 4321 LNCS, Issue , 2007, Pages 231-262

  Micarelli, Alessandro ^a   Gasparetti, Fabio ^a  

^a ROMA TRE UNIVERSITY   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE SYSTEMS; INFORMATION RETRIEVAL; RESOURCE ALLOCATION; SEARCH ENGINES;

CRAWLING; SEARCH DOMAIN; SYSTEM BEHAVIOR;

WEB SERVICES;

EID: 38849196327     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-540-72079-9_7     Document Type: Conference Paper

References (83)

1. Aggarwal, C.C., Al-Garawi, F., Yu, P.S.: Intelligent crawling on the world wide web with arbitrary predicates. In: Proceedings of the 10th World Wide Web Conference (WWW10), Hong Kong (2001) 96-105 http://www10.org/cdrom/papers/ 110/.
2. Arasu, A., Cho, J., Garcia-Molina, H., Paepcke, A., Raghavan, S.: Searching the web. ACM Transactions on Internet Technology (TOIT) 1(1) (2001) 2-43
3. Bailey, P., Craswell, N., Hawking, D.: Dark matter on the web. In: Poster Proceedings of the 9th World Wide Web Conference(WWW9), Amsterdam, Netherlands (2000) http://www9.org/final-posters/poster30.html.
4. Bailey, P., Craswell, N., Hawking, D.: Engineering a multi-purpose test collection for web retrieval experiments. Information Processing and Management 39(6) (2003) 853-871
5. Balabanović, M.: Exploring versus exploiting when learning user models for text recommendation, User Modeling and User-Adapted Interaction 8(1-2) (1998) 71-102
6. Balabanović, M., Shoham, Y.: Fab: content-based, collaborative recommendation. Communications of the ACM 40(3) (1997) 66-72
7. Beitzel, S.M., Jensen, E.C, Chowdhury, A., Grossman, D., Frieder, O.: Hourly analysis of a very large topically categorized web query log. In: SIGIR '04: Proceedings of the 27th annual international conference on Research and development in information retrieval, New York, NY, USA, ACM Press (2004) 321-328
8. Bergman, M.K.: The deep web: Surfacing hidden value. The Journal of Electronic Publishing 7(1) (August 2001)
9. Bergmark, D., Lagoze, C., Sbityakov, A.: Focused crawls, tunneling, and digital libraries. In: ECDL '02: Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries, London, UK, Springer-Verlag (2002) 91-106 http://citeseer.ist.psu.edu/bergmark02focused. html.
10. Bharat, K., Kamba, T., Albers, M.: Personalized, interactive news on the web. Multimedia Systems 6(5) (1998) 349-358
11. Boldi, P., Codenotti, B., Santini, M., Vigna, S.: Ubicrawler: a scalable fully distributed web crawler. Software, Practice and Experience 34(8) (2004) 711-726
12. Bonabeau, E., Dorigo, M., Theraulaz, G.: Inspiration for optimization from social insect behavior. Nature 406 (2000) 39-42
13. Bra, P.D., Houben, G.J., Komatzky, Y.: Information retrieval in distributed hypertexts, In: Proceedings of the 4th RIAO, Intelligent Multimedia, Information Retrieval Systems and Management, New York, NY, USA (1994) 481-491 http://citeseer.ist.psu.edu/debra94information.html.
14. Bra, P.D., Post, R.: Searching for arbitrary information in the www: The fish-search for mosaic. In: Proceedings of the 2nd World Wide Web Conference (WWW2), Chicago, USA (1994)http://citeseer.ist.psu.edu/172936.html.
15. Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems 30 (1998) 107-117
16. Broder, A., Kumar, R., Maghoul, F., Raghavan, P., Rajagopalan, S., Stata, R., Tomkins, A., Wiener, J.: Graph structure in the web. In: Proceedings of the 9th World Wide Web Conference (WWW9), Amsterdam, Netherlands (2000) 309-320 http://www9.org/w9cdrom/160/160.html.
17. Chakrabarti, S.: Recent results in automatic web resource discovery. ACM Computing Surveys 31(4es) (1999) 17
18. Chakrabarti, S., Punera, K., Subramanyam, M.: Accelerated focused crawling through online relevance feedback. In: WWW '02: Proceedings of the 11th international conference on World Wide Web, New York, NY, USA, ACM Press (2002) 148-159 http://www2002.org/CDROM/refereed/336/.
19. Chakrabarti, S., van den Berg, M., Dom, B.: Distributed hypertext resource discovery through examples. In: VLDB '99: Proceedings of the 25th International Conference on Very Large Data Bases, San Francisco, CA, USA, Morgan Kaufmann Publishers Inc. (1999) 375-386 http://www.vldb.org/conf/1999/ P37.pdf.
20. Chakrabarti, S., van den Berg, M., Dom, B.: Focused crawling: A new approach to topic-specific web resource discovery. In: Proceedings of the 8th World Wide Web Conference (WWW8), Toronto, Canada (1999) 1623-1640 http://www8.org/w8-papers/5a-search-query/crawling/index.html.
21. Chen, H., Chung, Y.M., Ramsey, M., Yang, C.C.: A smart itsy bitsy spider for the web. Journal of the American Society for Information Science 49(7) (1998) 604-618
22. Chi, E.H., Pirolli, P., Chen, K., Pitkow, J.: Using information scent to model user information needs and actions on the web. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI2001), Seattle, WA, USA (2001) 490-497
23. Cho, J., Garcia-Molina, H.: The evolution of the web and implications for an incremental crawler. In: VLDB '00: Proceedings of the 26th International Conference on Very Large Data Bases, San Francisco, CA, USA, Morgan Kaufmann Publishers Inc. (2000) 200-209
24. Cho, J., Garcia-Molina, H.: Synchronizing a database to improve freshness. In: SIGMOD '00: Proceedings of the 2000 ACM SIGMOD international conference on Management of data, New York, NY, USA, ACM Press (2000) 117-128
25. Cho, J., Garcia-Molina, H.: Parallel crawlers. In: WWW '02: Proceedings of the 11th international conference on World Wide Web, New York, NY, USA, ACM Press (2002) 124-135 http://www2002.org/CDROM/refereed/108/index.html.
26. Cho, J., Garcia-Molina, H.: Estimating frequency of change. ACM Transactions on Internet Technology (TOIT) 3(3) (2003) 256-290
27. Cho, J., Garcia-Molina, H., Page, L.: Efficient crawling through url ordering. Computer Networks and ISDN Systems 30(1-7) (1998) 161-172
28. Chung, C., Clarke, C.L.A.: Topic-oriented collaborative crawling. In: CIKM '02: Proceedings of the eleventh international conference on Information and knowledge management, New York, NY, USA, ACM Press (2002) 34-42
29. Davison, B.D.: Topical locality in the web. In: SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, USA, ACM Press (2000) 272-279
30. de Carvalho Fontes, A., Silva, F.S.: Smartcrawl: a new strategy for the exploration of the hidden web. In: WIDM '04: Proceedings of the 6th annual ACM international workshop on Web information and data management, New York, NY, USA, ACM Press (2004) 9-15
31. Delort, J.Y., Bouchon-Meunier, B., Rifqi, M.: Enhanced web document summarization using hyperlinks. In: HYPERTEXT '03: Proceedings of the fourteenth ACM conference on Hypertext and hypermedia, New York, NY, USA, ACM Press (2003) 208-215
32. Diligenti, M., Coetzee, F., Lawrence, S., Giles, C.L., Gori, M.: Focused crawling using context graphs. In: VLDB '00: Proceedings of the 26th International Conference on Very Large Data Bases, San Francisco, CA, USA, Morgan Kaufmann Publishers Inc. (2000) 527-534 http://www.vldb.org/conf/2000/ P527.pdf.
33. Diligenti, M., Maggini, M., Pucci, F.M., Scarselli, F.: Design of a crawler with bounded bandwidth. In: WWW Alt. '04: Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, New York, NY, USA, ACM Press (2004) 292-293 http://www2004.org/proceedings/ docs/2p292.pdf.
34. Doorenbos, R.B., Etzioni, O., Weld, D.S.: A scalable comparison-shopping agent for the world-wide web. In: AGENTS '97: Proceedings of the first international conference on Autonomous agents, New York, NY, USA, ACM Press (1997) 39-48
35. Ehrig, M., Maedche, A.: Ontology-focused crawling of web documents. In: SAC '03: Proceedings of the 2003 ACM symposium on Applied computing, New York, NY, USA, ACM Press (2003) 1174-1178
36. Fetterly, D., Manasse, M., Najork, M., Wiener, J.: A large-scale study of the evolution of web pages. In: WWW '03: Proceedings of the 12th international conference on World Wide Web, New York, NY, USA, ACM Press (2003) 669-678 http://www2003.org/odrom/papers/refereed/pO97/P97sources/p97-fetterly.html.
37. Furnas, G.W.: Effective view navigation. In: CHI '97: Proceedings of the SIGCHI conference on Human factors in computing systems, New York, NY, USA, ACM Press (1997) 367-374
38. Furnas, G.W., Landauer, T.K., Gomez, L.M., Dumais, S.T.: The vocabulary problem in human-system communication. Communications of the ACM 30(11) (1987) 964-971
39. Gasparetti, F., Micarelli, A.: Adaptive web search based on a colony of cooperative distributed agents. In Klusch, M., Ossowski, S., Omicini, A., Laamanen, H., eds.: Cooperative Information Agents. Volume 2782., Springer-Verlag (2003) 168-183
40. Gasparetti, F., Micarelli, A.: Swarm intelligence: Agents for adaptive web search. In: Proceedings of the 16th European Conference on Artificial Intelligence (ECAI 2004). (2004) 1019-1020 http://citeseer.ist.psu.edu/738711. html.
41. Gasparetti, F., Micarelli, A.: User profile generation based on a memory retrieval theory. In: Proc. 1st International Workshop on Web Personalization, Recommender Systems and Intelligent User Interfaces (WPRSIUI'05). (2005) 59-68 http://citeseer.ist.psu.edu/gasparetti05user.html.
42. Gauch, S., Speretta, M., Chandramouli, A., Micarelli, A.: User profiles for personalized information access. In Brusilovsky, P., Kobsa, A., Nejdl, W., eds.: The Adaptive Web: Methods and Strategies of Web Personalization. Volume 4321 of Lecture Notes in Computer Science. Springer-Verlag, Berlin Heidelberg New York (2007) this volume
43. Gulli, A., Signorini, A.: The indexable web is more than 11.5 billion pages. In: WWW '05: Special interest tracks and posters of the 14th international conference on World Wide Web, New York, NY, USA, ACM Press (2005) 902-903 http://www.cs.uiowa.edu/~asignori/web-size/.
44. Hersovicia, M., Jacovia, M., Maareka, Y.S., Pellegb, D., Shtalhaima, M., Ura, S.: The shark-search algorithm - an application: tailored web site mapping. In: Proceedings of the 7th World Wide Web Conference (WWW7), Brisbane, Australia (1998) 317-326 http://www7.scu.edu.au/1849/com1849.htm.
45. Heydon, A., Najork, M.: Mercator: A scalable, extensible web crawler. World Wide Web 2(4) (1999) 219-229
46. Joachims, T., Freitag, D., Mitchell, TM.: Webwatcher: A tour guide for the world wide web. In: Proceedings of the 15h International Conference on Artificial Intelligence (IJCAI1997). (1997) 770-777 http://citeseer.ist.psu.edu/ 16829. html.
47. Kleinberg, J.: Authoritative sources in a hyperlinked environment. In: Proceedings of the 9th annual ACM-SIAM symposium on Discrete algorithms, San Francisco, CA. USA (1998) 668-677 http://www.es.cornell.edu/home/kleinber/auth. pdf.
48. Kruger, A., Giles, C.L., Coetzee, F.M., Glover, E., Flake, G.W., Lawrence, S., Omlin, C.: Deadliner: building a new niche search engine. In: CIKM '00: Proceedings of the ninth international conference on Information and knowledge management, New York, NY, USA, ACM Press (2000) 272-281 http://citeseer.ist.psu.edu/kruger00deadliner.html.
49. Kumar, R., Novak, J., Raghavan, P., Tomkins, A.: On the bursty evolution of blogspace. In: WWW '03: Proceedings of the 12th international conference on World Wide Web, New York, NY, USA, ACM Press (2003) 568-576 http://www2003.org/ cdrom/papers/refereed/p477/p477-kumar/p477-kumar.htm.
50. Lage, J.P., da Silva, A.S., Golgher, P.B., Laender, A.H.F.: Automatic generation of agents for collecting hidden web pages for data extraction. Data and Knowledge Engineering 49(2) (2004)177-196
51. Lau, T., Horvitz, E.: Patterns of search: analyzing and modeling web query refinement. In: UM '99: Proceedings of the seventh international conference on User modeling, Secaucus, NJ, USA, Springer-Verlag New York, Inc. (1999) 119-128
52. Lawrence, S., Giles, L.C.: Searching the world wide web. Science 280 (1998) 98-100
53. Lawrence, S., Giles, L.C.: Accessibility of information on the web. Nature 400 (1999) 107-109
54. Levene, M., Poulovassilis, A.: Web dynamics. Software Focus 2(2) (2001) 60-67
55. Lu, W.H., Chien, L.F., Lee, H.J.: Translation of web queries using anchor text mining. ACM Transactions on Asian Language Information Processing (TALIP) 1(2) (2002) 159-172
56. Marchiori, M.: The quest for correct information on the web: Hyper search engines. In: Proceedings of the 6th World Wide Web Conference (WWW6), Santa Clara, CA, USA (1997) 1225-1235 http://www.w3.org/People/Massimo/papers/WWW6/.
57. Menczer, F.: Lexical and semantic clustering by web links. Journal of the American Society for Information Science and Technology 55(14) (2004) 1261-1269
58. Menczer, F., Belew, R.K.: Adaptive retrieval agents: Internalizing local context and scaling up to the web. Machine Learning 31(11-16) (2000) 1653-1665
59. Menczer, F., Pant, G., Srinivasan, P.: Topical web crawlers: Evaluating adaptive algorithms. ACM Transactions on Internet Technology 4(4) (2004) 378-419
60. Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A., Teller, E.: Equations of state calculations by fast computing machines. Journal of Chemical Physics 21(6) (1953)
61. Micarelli, A., Gasparetti, F., Sciarrone, F., Gauch, S.: Personalized search on the world wide web. In Brusilovsky, P., Kobsa, A., Nejdl, W., eds.: The Adaptive Web: Methods and Strategies of Web Personalization. Volume 4321 of Lecture Notes in Computer Science. Springer-Verlag, Berlin, Heidelberg, and New York (2007) this volume
62. Micarelli, A., Sciarrone, F., Marinilli, M.: Web document modeling. In Brusilovsky, P., Kobsa, A., Nejdl, W., eds.: The Adaptive Web: Methods and Strategies of Web Personalization. Volume 4321 of Lecture Notes in Computer Science. Springer-Verlag, Berlin Heidelberg New York (2007) this volume
63. Mizuuchi, Y., Tajima, K.: Finding context paths for web pages. In: Proceedings of the 10th ACM Conference on Hypertext and Hypermedia: Returning to Our Diverse Roots (HYPERTEXT99), Darmstadt, Germany (1999) 13-22
64. Najork, M., Heydon, A.: High-performance web crawling. In Abello, J., Pardalos, P.M., Resende, M.G., eds.: Handbook of massive data sets. Kluwer Academic Publishers, Norwell, MA, USA (2002) 25-45
65. Najork, M., Wiener, J.L.: Breadth-first search crawling yields high-quality pages. In: Proceedings of the 10th World Wide Web Conference (WWW10), Hong Kong (2001) 114-118 http://www10.org/cdrom/papers/208/.
66. Ntoulas, A., Cho, J., Olston, C.: What's new on the web?: the evolution of the web from a search engine perspective. In Feldman, S.I., Uretsky, M., Najork, M., Wills, C.E., eds.: Proceedings of the 13th international conference on World Wide Web, WWW 2004, New York, NY, USA, May 17-20, 2004, ACM (2004) 1-12 http://www2004.org/proceedings/docs/1p1.pdf.
67. Ntoulas, A., Zerfos, P., Cho, J.: Downloading textual hidden web content through keyword queries. In Marilno, M., Sumner, T., Ill, F.M.S., eds.: ACM/IEEE Joint Conference on Digital Libraries, JCDL 2005, Denver, CA, USA, June 7-11, 2005, Proceedings, ACM (2005) 100-109
68. Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: Bringing order to the web. Technical report, Stanford Digital Library Technologies Project (1998) http://dbpubs.stanford.edu/pub/1999-66.
69. Pinkerton, B.: Finding what people want: Experiences with the webcrawler. In: Proceedings of the 2nd World Wide Web Conference(WWW2), Chicago, USA (1994) 821-829
70. Pirolli, P., Card, S.K.: Information foraging. Psychological Review 106 (1999) 643-675
71. Raghavan, S., Garcia-Molina, H.: Crawling the hidden web. In: VLDB '01: Proceedings of the 27th International Conference on Very Large Data Bases, San Francisco, CA, USA, Morgan Kaufmann Publishers Inc. (2001) 129-138
72. Rennie, J., McCallum, A.: Using reinforcement learning to spider the web efficiently. In: ICML '99: Proceedings of the Sixteenth International Conference on Machine Learning, San Francisco, CA, USA, Morgan Kaufmann Publishers Inc. (1999) 335-343 http://citeseer.ist.psu.edu/7537.html.
73. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In Rumelhart, D.E., McClelland, J.L., eds.: Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations. MIT Press, Cambridge, MA, USA (1986) 318-362
74. Rungsawang, A., Angkawattanawit, N.: Learnable topic-specific web crawler. Journal of Network and Computer Applications 28(2) (2005) 97-114
75. Salton, G., McGill, M.J.: Introduction to Modem Information Retrieval. McGraw-Hill, New York (1983)
76. Shkapenyuk, V., Suel, T.: Design and implementation of a high-performance distributed web crawler. In: Proceedings of the 18th International Conference on Data Engineering (ICDE'02), Washington, DC, USA, IEEE Computer Society (2002) 357
77. Spink, A., Jansen, B.J.: A study of web search trends. Webology 1(2) (December 2004) http://www.webology.ir/2004/v1n2/a4.html.
78. Spink, A., Wolfram, D., Jansen, M.B.J., Saracevic, T.: Searching the web: the public and their queries. Journal of the American Society for Information Science 52(3) (2001) 226-234
79. Srinivasan, P., Menczer, F., Pant, G.: A general evaluation framework for topical crawlers. Information Retrieval 8(3) (2005) 417-447
80. Steele, R.: Techniques for specialized search engines. In: Proc. Internet Computing 2001, Las Vegas (June 25-28 2001) http://citeseer.ist.psu.edu/ steele01techniques.html.
81. Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA (1998)
82. Yang, C.C., Yen, J., Chen, H.: Intelligent internet searching agent based on hybrid simulated annealing. Decision Support Systems 28(2) (2000) 269-277
83. Yuwono, B., Lam, S.L.Y., Ying, J.H., Lee, D.L.: A World Wide Web resource discovery system. In: Proceedings of the 4th World Wide Web Conference (WWW4), Boston, Massachusetts, USA (1995) 145-158 http://www.w3.org/Conferences/WWW4/ Papers/66/.