Web dynamics and their ramifications for the development of Web search engines

Computer Networks

Volumn 50, Issue 10, 2006, Pages 1430-1447

  Ke, Yiping ^a   Deng, Lin ^a   Ng, Wilfred ^a   Lee, Dik Lun ^a  

^a HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

Personalized web searching; Web dynamics; Web link structures; Web pages; Web search engine; Web size; Web user interests

Indexed keywords

COMPUTER NETWORKS; ELECTRIC CONNECTORS; HYPERTEXT SYSTEMS; INFORMATION RETRIEVAL SYSTEMS; WEBSITES;

WEB DYNAMICS; WEB SIZE; WEB USER INTERESTS;

SEARCH ENGINES;

EID: 33646346081     PISSN: 13891286     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.comnet.2005.10.012     Document Type: Article

References (73)

1. AltaVista, http://www.altavista.com.
2. Dogpile, http://www.dogpile.com.
3. EuroSeek, http://www.euroseek.com.
4. Excite, http://www.excite.com.
5. Google, http://www.google.com.
6. Google Directory, http://dir.google.com.
7. Google Personalized Web Search, http://labs.google.com/personalized.
8. HotBot, http://www.hotbot.com.
9. Infoseek, http://www.infoseek.com.
10. Lycos, http://www.lycos.com.
11. MSN Search, http://search.msn.com.
12. Netcraft Web Server Survey, http://news.netcraft.com/archives/web_server_survey.html.
13. Northern Light, http://www.nlsearch.com.
14. Snap, http://www.snap.com.
15. The Open Directory Project: Web directory for over 2.5 million URLs, http://www.dmoz.org.
16. Yahoo Directory, http://dir.yahoo.com.
17. L.A. Adamic, The small world Web, in: Proc. of Third European Conference of Research and Advanced Technology for Digital Libraries, 1999.
18. Albert R., Jeong H., and Barabási A. Diameter of the world wide web. Nature 401 6749 (1999) 130-131
19. Baeza-Yates R., Castillo C., and Saint-Jean F. Web dynamics, structure, and page quality. In: Levene M., and Poulovassilis A. (Eds). Web Dynamics: Adapting to Change in Content, Size, Topology and Use (2004) (Chapter 5)
20. Baeza-Yates R., and Ribeiro-Neto B. Modern Information Retrieval (1999), Addison-wesley-Longman, Harlow, UK
21. Barroso L.A., Dean J., and Holzle U. Web search for a planet: the google cluster architecture. IEEE Micro March-April (2003) 22-28
22. M.K. Bergman, The deep Web: surfacing hidden value, White paper, Bright Planet, July, 2000.
23. K. Bharat, A. Broder, A technique for measuring the relative size and overlap of public Web search engines, in: Proc. of the 7th International WWW Conference, April 1998, pp. 379-388.
24. K. Bharat, M.R. Henzinger, Improved algorithms for topic distillation in a hyperlinked environment, in: Proc. of SIGIR, 1998, pp. 104-111.
25. A. Blum, T. Mitchell, Combining labeled and unlabeled data with co-training, in: Proc. of COLT, 1998, pp. 92-100.
26. B.E. Brewington, G. Cybenko, How dynamic is the Web, in: Proc. of the 9th International WWW Conference, 2000.
27. Brewington B.E., and Cybenko G. Keeping up with the changing Web. IEEE Computer 33 5 (2000) 52-58
28. Brin S., and Page L. The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems 30 1-7 (1998) 107-117
29. A. Broder, R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins, J. Wiener, Graph structure in the Web: Experiments and models, in: Proc. of the 9th International WWW Conference, May 2000.
30. Chakrabarti S., Berg M., and Dom B. Focused crawling: a new approach to topic-specific Web resource discovery. Computer Networks 31 11-16 (1999) 1623-1640
31. S. Chakrabarti, B. Dom, D. Gibson, J. Kleinberg, P. Raghavan, S. Rajagopalan, Automatic resource list compilation by analyzing hyperlink structure and associated text, in: Proc. of the 7th International WWW Conference, 1998.
32. S. Chakrabarti, A. Frieze, J. Vera, The influence of search engines on preferential attachment, in: Proc. of SODA, 2005, pp. 293-300.
33. S. Chien, C. Dwork, R. Kumar, D. Sivakumar, Towards exploiting link evolution, in: Workshop on Algorithms and Models for the Web Graph, November 2002.
34. J. Cho, H. Garcia-Molina, The evolution of the Web and implications for an incremental crawler, in: Proc. of VLDB, 2000.
35. J. Cho, H. Garcia-Molina, Synchronizing a database to improve freshness, in: Proc. of SIGMOD, 2000, pp. 117-128.
36. L. Deng, X. Chai, Q. Tan, W. Ng, D.L. Lee, Spying out real user preferences for metasearch engine adaptation, in: Proc. of WebKDD, 2004, pp. 71-82.
37. M. Diligenti, F. Coetzee, S. Lawrence, C.L. Giles, M. Gori, Focused crawling using context graphs, in: Proc. of VLDB, September 2000, pp. 527-534.
38. J. Edwards, K.S. McCurley, J.A. Tomlin, An adaptive model for optimizing performance of an incremental Web crawler, in: Proc. of the 10th International WWW Conference, 2001, pp. 106-113.
39. M. Ehrig, A. Maedche, Ontology-focused crawling of Web documents, in: Proc. of the 2003 ACM Symposium on Applied Computing, 2003.
40. D. Fetterly, M. Manasse, M. Najork, J. Wiener, A large-scale study of the evolution of Web pages, in: Proc. of the 12th International WWW Conference, 2003.
41. G. Fung, J. Yu, H. Lu, P.S. Yu, Text classification without labeled negative documents, in: Proc. of ICDE, 2005, pp. 594-605.
42. T.H. Haveliwala, Topic-sensitive PageRank, in: Proc. of the 11th International WWW Conference, 2002, pp. 517-526.
43. B.A. Huberman, L.A. Adamic, Evolutionary dynamics of the World Wide Web, Technical report, Xerox Palo Alto Research Center, February 1999.
44. P.G. Ipeirotis, L. Gravano, When one sample is not enough: improving text database selection using shrinkage, in: Proc. of SIGMOD, 2004.
45. G. Jeh, J. Widom, Scaling personalized Web search, in: Proc. of the 12th International WWW Conference, 2003, pp. 271-279.
46. T. Joachims, Optimizing search engines using clickthrough data, in: Proc. of KDD, 2002, pp. 133-142.
47. W. Kießling, Foundations of preferences in database systems, in: Proc. of VLDB, 2002, pp. 311-322.
48. J.M. Kleinberg, Authoritative sources in a hyperlinked environment, in: Proc. of SODA, 1998, pp. 668-677.
49. Koehler W. An analysis of Web page and Web site constancy and permanence. Journal of the American Society for Information Science 50 2 (1999) 162-180
50. Koehler W. Digital libraries and World Wide Web sites and page persistence. Information Research 4 4 (1999)
51. Koehler W. Web page change and persistence-a four-year longitudinal study. Journal of the American Society for Information Science and Technology 53 2 (2002) 162-171
52. Koehler W. A longitudinal study of Web pages continued: a consideration of document persistence. Information Research 9 2 (2004)
53. Kumar R., Ragoavan P., Rajagopalan S., and Sivakumar D. Stochastic models for the Web graph. IEEE Symposium on Foundations of Computer Science (FOCS) (2000)
54. Lawrence S., and Giles C.L. Searching the World Wide Web. Science 280 5360 (1998) 98-100
55. Lawrence S., and Giles C.L. Accessibility of information on the Web. Nature 400 6740 (1999) 107-109
56. L. Lim, M. Wang, S. Padmanabhan, J.S. Vitter, R. Agarwal, Characterizing Web document change, in: Proc. of the 2nd International Conference on Advances in Web-Age Information Management, 2001, pp. 133-144.
57. F. Liu, C. Yu, W. Meng, Personalize Web search by mapping user queries to categories, in: Proc. of CIKM, 2002, pp. 558-565.
58. Liu F., Yu C., and Meng W. Personalized Web search for improving retrieval effectiveness. TKDE 16 (2004) 28-40
59. F. Menczer, G. Pant, P. Srinivasan, M.E. Ruiz, Evaluating topic-driven Web crawlers, in: Proc. of SIGIR, 2001, pp. 241-249.
60. J.F. Mendes, Theory of random networks and their role in communications networks, in: M. Levene, A. Poulovassilis, (Eds.), Web Dynamics: Adapting to Change in Content, Size, Topology and Use, 2004 (Chapter 4).
61. Meng W., Yu C., and Liu K. Building efficient and effective metasearch engines. ACM Computing Surveys 34 1 (2002)
62. A.Y. Ng, A.X. Zheng, M.I. Jordan, Link analysis, eigenvectors and stability, in: Proc. of IJCAI, 2001, pp. 903-910.
63. A.Y. Ng, A.X. Zheng, M.I. Jordan, Stable algorithms for link analysis, in: Proc. of SIGIR, 2001.
64. A. Ntoulas, J. Cho, C. Olston, What's new on the Web? The evolution of the Web from a search engine perspective, in: Proc. of the 13th International WWW Conference, May 2004, pp. 1-12.
65. L. Page, S. Brin, R. Motwani, T. Winograd, The PageRank citation ranking: Bringing order to the Web, Technical report, Computer Science Department, Stanford University, 1999.
66. Pitkow J., Schinze H., Cass T., Cooley R., Turnbull D., Edmonds A., Adar E., and Breuel T. Personalized search. Communications of the ACM 45 9 (2002) 50-55
67. Risvik K.M., and Michelsen R. Search engines and Web dynamics. Computer Networks 39 (2002) 289-302
68. Robertson S.E., and Jones K.S. Relevance weighting of search terms. Journal of the American Society for Information Sciences 27 3 (1976)
69. Salton G., Fox E.A., and Wu H. Extended boolean information retrieval. Communications of the ACM 26 11 (1983) 1022-1036
70. Schikc S. A glimpse into google's hardware solutions. Internet Marketing News (2003)
71. E.W. Selberg, Towards Comprehensive Web Search, Ph.D. thesis, University of Washington, June 1999.
72. K. Sugiyama, K. Hatano, M. Yoshikawa, Adaptive Web search based on user profile constructed without any effort from users, in: Proc. of the 13th International WWW Conference, 2004, pp. 675-684.
73. Q. Tan, X. Chai, W. Ng, D.L. Lee, Applying co-training to clickthrough data for search engine adaptation, in: Proc. of DASFAA, 2004, pp. 519-532.