Accelerated focused crawling through online relevance feedback

Proceedings of the 11th International Conference on World Wide Web, WWW '02

Volumn , Issue , 2002, Pages 148-159

  Chakrabarti, Soumen ^a   Punera, Kunal ^a   Subramanyam, Mallela ^b  

^a INDIAN INSTITUTE OF TECHNOLOGY BOMBAY   (India)

^b UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Document object model; Focused crawling; Reinforcement learning

Indexed keywords

AUTOMATIC PROGRAMS; DOCUMENT OBJECT MODEL; FALSE POSITIVE; FOCUSED CRAWLER; FOCUSED CRAWLING; HUMAN BEHAVIORS; INFORMATION NEED; OPEN DIRECTORY; RECTANGULAR REGIONS; RELEVANCE FEEDBACK; RESOURCE DISCOVERY; TREE STRUCTURES;

APPRENTICES; BEHAVIORAL RESEARCH; INFORMATION SCIENCE; REINFORCEMENT; REINFORCEMENT LEARNING; TREES (MATHEMATICS); WORLD WIDE WEB;

FEEDBACK;

EID: 77953064623     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/511446.511466     Document Type: Conference Paper

References (34)

1. C. C. Aggarwal, F. Al-Garawi, and P. S. Yu. Intelligent crawling on the World Wide Web with arbitrary predicates. In WWW2001, Hong Kong, May 2001. ACM. Online at http: //www10.org/cdrom/papers/110/.
2. C. Apte, F. Damerau, and S. M. Weiss. Automated learning of decision rules for text categorization. ACM Transactions on Information Systems, 1994. IBM Research Report RC18879.
3. A. Blum and T. M. Mitchell. Combining labeled and unlabeled data with co-training. In Computational Learning Theory, pages 92-100, 1998.
4. S. Chakrabarti. Integrating the document object model with hyperlinks for enhanced topic distillation and information extraction. In WWW10, Hong Kong, May 2001. Online at http://www10.org/cdrom/papers/489.
5. S. Chakrabarti, B. Dom, R. Agrawal, and P. Raghavan. Scalable feature selection, classification and signature generation for organizing large text databases into hierarchical topic taxonomies. VLDB Journal, Aug. 1998. Online at http: //www.cs.berkeley.edu/~soumen/VLDB54-3.PDF.
6. S. Chakrabarti, B. Dom, D. Gibson, J. Kleinberg, P. Raghavan, and S. Rajagopalan. Automatic resource compilation by analyzing hyperlink structure and associated text. In 7th Worldwide web conference (WWW7), 1998. Online at http://www7.scu.edu.au/programme/fullpapers/1898/com1898.html.
7. S. Chakrabarti, B. Dom, and P. Indyk. Enhanced hypertext categorization using hyperlinks. In SIGMOD Conference. ACM, 1998. Online at http://www.cs.berkeley.edu/~soumen/sigmod98.ps.
8. S. Chakrabarti, B. E. Dom, D. A. Gibson, R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. Topic distillation and spectral filtering. Artificial Intelligence Review, 13(5-6):409-435, 1999.
9. S. Chakrabarti, M. van den Berg, and B. Dom. Focused crawling: a new approach to topic-specific web resource discovery. Computer Networks, 31:1623-1640, 1999.
10. First appeared in the 8th International World Wide Web Conference, Toronto, May 1999. Available online at http://www8.org/w8-papers/5a-search- query/crawling/index.html.
11. J. Cho, H. Garcia-Molina, and L. Page. Efficient crawling through URL ordering. In 7th World Wide Web Conference, Brisbane, Australia, Apr. 1998. Online at http://www7.scu.edu.au/programme/fullpapers/1919/com1919.htm.
12. B. D. Davison. Topical locality in the Web. In Proceedings of the 23rd Annual International Conference on Research and Development in Information Retrieval (SIGIR 2000), pages 272-279, Athens, Greece, July 2000. ACM. Online at http:// www.cs.rutgers.edu/~davison/pubs/2000/sigir/.
13. P. M. E. De Bra and R. D. J. Post. Information retrieval in the world-wide web: Making client-based searching feasible. In Proceedings of the First International World Wide Web Conference, Geneva, Switzerland, 1994. Online at http://www1.cern.ch/PapersWWW94/reinpost.ps.
14. P. M. E. De Bra and R. D. J. Post. Searching for arbitrary information in the WWW: The fish search for Mosaic. In Second World Wide Web Conference '94: Mosaic and the Web, Chicago, Oct. 1994. Online at http://archive.ncsa.uiuc.edu/ SDG/IT94/Proceedings/Searching/debra/article.html and http: //citeseer.nj.nec. com/172936.html.
15. M. Diligenti, F. Coetzee, S. Lawrence, C. L. Giles, and M. Gori. Focused crawling using context graphs. In A. E. Abbadi, M. L. Brodie, S. Chakravarthy, U. Dayal, N. Kamel, G. Schlageter, and K.-Y. Whang, editors, VLDB 2000, Proceedings of 26th International Conference on Very Large Data Bases, September 10-14, 2000, Cairo, Egypt, pages 527-534. Morgan Kaufmann, 2000. Online at http://www.neci.nec.com/~lawrence/papers/focus-vldb00/focus-vldb00.pdf.
16. W. A. Gale, K. W. Church, and D. Yarowsky. A method for disambiguating word senses in a large corpus. Computer and the Humanities, 26:415-439, 1993.
17. M. Hersovici, M. Jacovi, Y. S. Maarek, D. Pelleg, M. Shtalhaim, and S. Ur. The shark-search algorithm|an application: Tailored Web site mapping. In WWW7, 1998. Online at http://www7.scu.edu.au/programme/fullpapers/1849/com1849. htm.
18. T. Joachims, D. Freitag, and T. Mitchell. WebWatcher: A tour guide for the web. In IJCAI, Aug. 1997. Online at http://www.cs.cmu.edu/~webwatcher/ ijcai97.ps.
19. H. Leiberman. Letizia: An agent that assists Web browsing. In International Joint Conference on Artificial Intelligence (IJCAI), Montreal, Aug. 1995. See Website at http://lieber.www.media.mit.edu/people/lieber/ Lieberary/Letizia/Letizia.html.
20. H. Leiberman, C. Fry, and L. Weitzman. Exploring the Web with reconnaissance agents. CACM, 44(8):69-75, Aug. 2001. http://www.acm.org/cacm.
21. A. McCallum. Bow: A toolkit for statistical language modeling, text retrieval, classification and clustering. Software available from http://www.cs.cmu.edu/~mccallum/bow/, 1998.
22. A. McCallum and K. Nigam. A comparison of event models for naive Bayes text classification. In AAAI/ICML-98 Workshop on Learning for Text Categorization, pages 41-48. AAAI Press, 1998. Online at http://www.cs.cmu.edu/ ~knigam/.
23. A. McCallum and K. Nigam. A comparison of event models for naive Bayes text classification. In AAAI/ICML-98 Workshop on Learning for Text Categorization, pages 41-48. AAAI Press, 1998. Also technical report WS-98-05, CMU; online at http: //www.cs.cmu.edu/~knigam/papers/multinomial-aaaiws98.pdf.
24. F. Menczer. Links tell us about lexical and semantic Web content. Technical Report Computer Science Abstract CS.IR/0108004, arXiv.org, Aug. 2001. Online at http://arxiv.org/abs/cs.IR/0108004.
25. F. Menczer and R. K. Belew. Adaptive retrieval agents: Internalizing local context and scaling up to the Web. Machine Learning, 39(2/3):203-242, 2000. Longer version available as Technical Report CS98-579, http://dollar.biz.uiowa.edu/~fil/Papers/MLJ.ps, University of California, San Diego.
26. F. Menczer, G. Pant, M. Ruiz, and P. Srinivasan. Evaluating topic-driven Web crawlers. In SIGIR, New Orleans, Sept. 2001. ACM. Online at http://dollar.biz.uiowa.edu/~fil/Papers/sigir-01.pdf.
27. T. Mitchell. Machine Learning. McGraw Hill, 1997.
28. T. Mitchell. Mining theWeb. In SIGIR 2001, Sept. 2001. Invited talk.
29. S. Mukherjea. WTMS: a system for collecting and analyzing topic-specific Web information. WWW9/Computer Networks, 33(1-6):457-471, 2000. Online at http://www9.org/w9cdrom/293/293.html.
30. S. RaviKumar, P. Raghavan, S. Rajagopalan, D. Sivakumar, A. Tomkins, and E. Upfal. Stochastic models for the Web graph. In FOCS, volume 41, pages 57-65. IEEE, nov 2000. Online at http://www.cs.brown.edu/people/eli/papers/focs00.ps.
31. J. Rennie and A. McCallum. Using reinforcement learning to spider the web efficiently. In ICML, 1999. Online at http:// www.cs.cmu.edu/~mccallum/papers/ rlspider-icml99s.ps.gz.
32. G. Salton and M. J. McGill. Introduction to Modern Information Retrieval. McGraw-Hill, 1983.
33. M. Subramanyam, G. V. R. Phanindra, M. Tiwari, and M. Jain. Focused crawling using TFIDF centroid. Hypertext Retrieval and Mining (CS610) class project, Apr. 2001. Details available from manyam@cs.utexas.edu.
34. L. Torgo and J. Gama. Regression by classification. In D. Borgesand C. Kaestner, editors, Brasilian AI Symposium, volume 1159 of Lecture Notes in Artificial Intelligence, Curitiba, Brazil, 1996. Springer-Verlag. Online at http://www.ncc.up.pt/~ltorgo/Papers/list-pub.html.