Using HMM to learn user browsing patterns for focused Web crawling

Data and Knowledge Engineering

Volumn 59, Issue 2, 2006, Pages 270-291

  Liu, Hongyu ^a   Janssen, Jeannette ^a   Milios, Evangelos ^a  

^a DALHOUSIE UNIVERSITY   (Canada)

Author keywords

Focused crawling; Hidden Markov models; Pattern learning; Relevance modelling; User modelling; Web Graph; Web searching; World Wide Web

Indexed keywords

COMPUTER AIDED SOFTWARE ENGINEERING; DATA ACQUISITION; MATHEMATICAL MODELS; ONLINE SYSTEMS; PORTALS; WEB BROWSERS;

FOCUSED CRAWLING; PATTERN LEARNING; RELEVANCE MODELLING; WEB GRAPH; WEB SEARCHING;

MARKOV PROCESSES;

EID: 33748132951     PISSN: 0169023X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.datak.2006.01.012     Document Type: Article

References (29)

1. P. Lyman, H. Varian, J. Dunn, A. Strygin, K. Swearingen, How much information? 2003. Available from: . Link checked on March 10, 2006.
2. R. Zakon, Hobbes' internet timeline v7.0. Available from: . Link checked on March 10, 2006.
3. Search engine sizes. Available from: . Link checked on March 10, 2006.
4. Site position and coverage. Available from: . Link checked on March 10, 2006.
5. S. Chakrabarti, M. van den Berg, B. Dom, Focused crawling: a new approach to topic-specific Web resource discovery, in: Proceedings of the 8th International WWW Conference, Toronto, Canada, 1999.
6. D. Bergmark, C. Lagoze, A. Sbityakov, Focused crawls, tunneling, and digital libraries, in: Proceedings of the 6th European Conference on Digital Libraries, Rome, Italy, 2002.
7. P.D. Bra, R. Post, Information retrieval in the World Wide Web: making client-base searching feasible, in: Proceedings of the 1st International WWW Conference, Geneva, Switzerland, 1994.
8. M. Hersovici, M. Jacovi, Y. Maarek, D. Pelleg, M. Shtalhaim, S. Ur, The Shark-search algorithm-an application: tailored Web site mapping, in: Proceedings of the 7th International WWW Conference, Brisbane, Australia, 1998.
9. C. Aggarwal, F. Al-Garawi, P. Yu, Intelligent crawling on the World Wide Web with arbitrary predicates, in: Proceedings of the 10th International WWW Conference, Hong Kong, 2001.
10. S. Chakrabarti, K. Punera, M. Subramanyam, Accelerated focused crawling through online relevance feedback, in: Proceedings of the 11th International WWW Conference, Hawaii, USA, 1999.
11. J. Cho, H. Garcia-Molina, L. Page, Efficient crawling through URL ordering, in: Proceedings of the 7th World Wide Web Conference, Brisbane, Australia, 1998.
12. K. Stamatakis, V. Karkaletsis, G. Paliouras, J. Horlock, et al., Domain-specific Web site identification: the CROSSMARC focused Web crawler, in: Proceedings of the 2nd International Workshop on Web Document Analysis (WDA2003), Edinburgh, UK, 2003.
13. J. Rennie, A. McCallum, Using reinforcement learning to spider the Web efficiently, in: Proceedings of the 16th International Conference on Machine Learning (ICML-99), Bled, Slovenia, 1999.
14. J. Johnson, K. Tsioutsiouliklis, C.L. Giles, Evolving strategies for focused Web crawling, in: Proceedings of the 20th International Conference on Machine Learning (ICML-2003), Washington, DC, USA, 2003.
15. M. Diligenti, F. Coetzee, S. Lawrence, C. Giles, M. Gori, Focused crawling using context graphs, in: Proceedings of the 26th International Conference on Very Large Databases (VLDB 2000), Cairo, Egypt, 2000.
16. F. Menczer, G. Pant, P. Srinivasan, M. Ruiz, Evaluating topic-driven Web crawlers, in: Proceedings of the 24th Annual International ACM/SIGIR Conference, New Orleans, USA, 2001.
17. Menczer F., Pant G., and Srinivasan P. Topical Web crawlers: evaluating adaptive algorithms. ACM TOIT 4 4 (2004) 378-419
18. Srinivasan P., Menczer F., and Pant G. A general evaluation framework for topical crawlers. Information Retrieval 8 3 (2005) 417-447
19. G. Pant, K. Tsioutsiouliklis, J. Johnson, C. Giles, Panorama: extending digital libraries with topical crawlers, in: Proceedings of ACM/IEEE Joint Conference on Digital Libraries (JCDL 2004), Tucson, Arizona, June 2004, pp. 142-150.
20. Algorithmic Solutions. Available from: . Link checked on March 10, 2006.
21. M.W. Berry, LSI: Latent Semantic Indexing Web Site. Available from: . Link checked on March 10, 2006.
22. Deerwester S.C., Dumais S.T., Landauer T.K., Furnas G.W., and Harshman R.A. Indexing by latent semantic analysis. Journal of the American Society of Information Science 41 6 (1990) 391-407
23. Berry M.W. Large scale singular value computations. International Journal of Supercomputer Applications 6 (1992) 13-49
24. M.W. Berry et al., SVDPACKC: Version 1.0 User's Guide, Technical Report CS-93-194, University of Tennessee, Knoxville, TN, October 1993.
25. Pelleg D., and Moore A. X-means: extending K-means with efficient estimation of the number of clusters. Proceedings of the 17th International Conference on Machine Learning (2000), Morgan Kaufmann, San Francisco
26. Rabiner L.R. A tutorial on hidden Markov model and selected applications in speech recognition. Proceedings of the IEEE 77 2 (1989) 257-285
27. Google. Available from: . Link checked on March 10, 2006.
28. Lafferty J., McCallum A., and Pereira F. Conditional random fields: probabilistic models for segmenting and labeling sequence data. Proceedings of the International Conference on Machine Learning (ICML-2001) (2001), Williams College, USA
29. D. Pinto, A. McCallum, X. Wei, W.B. Croft, Table extraction using conditional random fields, in: Proceedings of the 26th Annual International ACM SIGIR Conference, Toronto, Canada, 2003.