Probabilistic models for focused web crawling

Proceedings of the Interntational Workshop on Web Information and Data Management

Volumn , Issue , 2004, Pages 16-22

  Liu, Hongyu ^a   Milios, Evangelos ^a   Janssen, Jeannette ^a  

^a DALHOUSIE UNIVERSITY   (Canada)

Author keywords

Conditional Random Fields; Focused crawling; Hidden Markov Models; Web Graph; World Wide Web

Indexed keywords

ALGORITHMS; INDEXING (OF INFORMATION); MARKOV PROCESSES; MATHEMATICAL MODELS; PARAMETER ESTIMATION; PROBABILITY; SEARCH ENGINES;

CONDITIONAL RANDOM FIELDS (CRF); FOCUSED CRAWLING; HIDDEN MARKOV MODELS (HDD); WEB GRAPH;

WEBSITES;

EID: 18744408155     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1031453.1031458     Document Type: Conference Paper

References (23)

1. C. Aggarwal, F. Al-Garawi, and P. Yu. Intelligent Crawling on the World Wide Web with Arbitrary Predicates. In Proceedings of the 10th International WWW Conference, Hong Kong, May 2001.
2. M. W. Berry. LSI: Latent Semantic Indexing Web Site. http://www.cs.utk. edu/~lsi/, Accessed May 15, 2004.
3. S. Chakrabarti, K. Punera, and M. Subramanyam. Accelerated focused crawling through online relevance feedback. In Proceedings of the 11th International WWW Conference, Hawaii, 2002, USA.
4. S. Chakrabarti, M. van den Berg, and B. Dom. Focused Crawling: a new approach to topic-specific web resource discovery. In Proceedings of the Eighth International WWW Conference, Toronto, Canada,1999.
5. J. Dean and M. Henzinger. Finding Related Pages in the World Wide Web. In Proceedings of the 8th International WWW Conference, pages 389-401, 1999.
6. M. Diligenti, F. Coetzee, S. Lawrence, C. Giles, and M. Gori. Focused Crawling Using Context Graphs. In Proceedings of the 26th International Conference on Very Large Databases (VLDB 2000), Cairo, Egypt, September 2000.
7. F.Sha and F. Pereira. Shallow Parsing with Conditional Random Fields. In Proceedings of Human Language Technology, NAACL, 2003.
8. M. Hersovici, M. Jacovi, Y. Maarek, D. Pelleg, M. Shtalhaim, and S. Ur. The Shark-Search Algorithm - An Application: Tailored Web Site Mapping. In Proceedings of the Seventh International WWW Conference, Brisbane, Australia, April 1998.
9. T. Joachims, D. Freitag, and T. Mitchell. WebWatcher: A Tour Guide for the World Wide Web. In Proceedings of IJCAI97, August 1997. Accessed May 2004. http://www-2.cs.cmu.edu/afs/cs.cmu.edu/project/theo-6/web-agent/www/ project-home.html.
10. J. Johnson, K. Tsioutsiouliklis, and C. L. Giles. Evolving Strategies for Focused Web Crawling. In The Twentieth International Conference on Machine Learning (ICML-2003), August 21-24, 2003 Washington, DC USA.
11. J. Lafferty, A. McCallum, and F.Pereira. Conditional random fields: Probabilistic models for segmenting an labeling sequence data. In International Conference on Macvhine Learning(ICML-2001), 2001.
12. H. Liu, E. Milios, and J. Janssen. Focused Crawling by Learning HMM from User's Topic-specific Browsing. In 2004 IEEE/WIC International Conference on Web Intelligence, Sep.20-24, 2004.
13. D. MacKay. Information Theory, Inference, and Learning Algorithms. Book ISBN:0521642981, Copyright Cambridge University Press, 2003.
14. F. Menczer and R. Belew. Adaptive retrieval agents: Internalizing local context and scaling up to the web. Machine Learning, 39(2/3):203-242, 2000.
15. F. Menczer, G. Pant, and P. Srinivasan. Myspiders: Evolve your own intelligent web crawlers. In Autonomous Agents and Multi-Agent Systems, pages 5, 241-249, 2003.
16. F. Menczer, G. Pant, and P. Srinivasan. Topical web crawlers: Evaluating adaptive algorithms. In To appear in ACM TOIT, Accessed May 2004. http://www.informatics.indiana.edu/fil/Papers/TOIT.pdf
17. F. Menczer, G. Pant, P. Srinivasan, and M. Ruiz. Evaluating Topic-Driven Web Crawlers. In Proceedings of the 24th Annual International ACM/SIGIR Conference, New Orleans, USA, 2001.
18. M. Najork and I. N. Wiener. Breadth-first Search crawling yields high-quality pages. In Proceedings of the 10th International WWW Conference, Hong Kong, May 2001.
19. D. Pinto, A. McCallum, X. Wei, and W. B. Croft. Table Extraction Using Conditional Random Fields. In 26th Annual International ACM SIGIR conference, pages Jul 28 - Aug. 1, 2003.
20. L. R. Rabiner. A Tutorial on Hidden Markov Model and Selected Applications in Speech Recognition. Proceedings of the IEEE, 77(2):257-285, 1989.
21. J. Rennie and A. McCallum. Using Reinforcement Learning to Spider the Web Efficiently. In Proceedings of the Sixteenth International Conference on Machine Learning(ICML-99), pages 335-343, 1999.
22. S.C.Deerwester, S.T.Dumais, T.K.Landauer, G.W.Furnas, and R.A.Harshman. Indexing by Latent Semantic Analysis. Journal of the American Society of Information Science, 41(6):391-407, 1990.
23. P. Srinivasan, F. Menczer, and G. Pant. A general evaluation framework for topical web crawlers. In Information Retrieval, Submitted, Accessed May 2004. http://www.informatics.indiana.edu/fil/Papers/crawl-framework.pdf.