Web-Crawling Reliability

Journal of the American Society for Information Science and Technology

Volumn 55, Issue 14, 2004, Pages 1228-1238

  Cothey, Viv ^a  

^a UNIVERSITY OF WOLVERHAMPTON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

DATA ACQUISITION; GRAPH THEORY; INFORMATION RETRIEVAL; MATHEMATICAL MODELS; SEARCH ENGINES; SOCIAL SCIENCES;

DATA COLLECTIONS; LINK CRAWLING; WEB CRAWLING; WEB LINKS;

WORLD WIDE WEB;

EID: 9744219710     PISSN: 15322882     EISSN: None     Source Type: Journal    
DOI: 10.1002/asi.20078     Document Type: Article

References (37)

1. Amitay, E., Carmel, D., Darlow, A., Lempel, R., & Soffer, A. (2003). The connectivity sonar: Detecting site functionality by structural patterns. Proceedings of the 14th Conference on Hypertext and Hypermedia (pp. 38-47), Nottingham, UK, August 26-30, 2003. New York: ACM.
2. Bailey, P., Craswell, N., & Hawking, D. (2003). Engineering a multi-purpose test collection for Web retrieval experiments. Information Processing and Management, 39(6), 853-871.
3. Bar-Ilan, J. (2001). Data collection methods on the Web for informetric purposes: A review and analysis. Scientometrics, 50(1), 7-32.
4. Bar-Ilan, J., & Peritz, B.C. (2002). Informetric theories and methods for exploring the Internet: An analytical survey of recent research literature. Library Trends, 50(3), 371-392.
5. Bergmark, D., Lagoze, C., & Sbityakov, A. (2002). Focused crawls, tunneling, and digital libraries. In V. Agosti & C. Thanos (Eds.), Proceedings of the Sixth European Conference on Digital Libraries (pp. 91-106), Rome, Italy, September 16-18, 2002. London: Springer.
6. Bharat, K., & Broder, A. (1998). A technique for measuring the relative size and overlap of public Web search engines. Computer Networks and ISDN Systems, 30(1-7), 379-388.
7. Bharat, K., Broder, A., Dean, J., & Henzinger, M.R. (2000). A comparison of techniques to find mirrored hosts on the WWW. Journal of the American Society for Information Science and Technology, 51(12), 1114-1122.
8. Borgman, C.L., & Furner, J. (2002). Scholarly communication and bibliometrics. In B. Cronin (Ed.), Annual review of information science and technology: 36, (chap. 1, pp. 3-94). Medford, NJ: Information Today for American Society for Information Science & Technology.
9. Brin, S., & Page, L. (1998). The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems, 30(1-7), 107-117.
10. Broder, A., Kumar, R., Maghoul, F., Raghavan, P., Rajagopalan, S., Stata, R., et al. (2000). Graph structure in the Web. Computer Networks, 33(1-6), 309-320.
11. Burner, M. (1997). Crawling towards eternity: Building an archive of the World Wide Web. New Architect: Internet Strategies for Technology Leaders, 2(5). Retrieved May 17, 2003, from http://www.newarchitectmag.com/archives/1997/05/ burner/
12. Chakrabarti, S., Joshi, M.M., Punera, K., & Pennock, D.M. (2002). The structure of broad topics on the Web. Proceedings of the 11th World Wide Web Conference (pp. 508-516), Honolulu, Hawaii, May 7-11, 2002. New York: ACM.
13. Cho, J., & Garcia-Molina, H. (2002). Parallel crawlers. Proceedings of the 11th World Wide Web Conference (pp. 124-134), Honolulu, Hawaii, May 7-11, 2002. New York: ACM.
14. Cronin, B. (2001). Bibliometrics and beyond: Some thoughts on Web-based citation analysis. Journal of Information Science, 27(1), 1-7.
15. Edwards, J., McCurley, K., & Tomlin, J. (2002). An adaptive model for optimizing performance of an incremental Web crawler. Proceedings of the 11th World Wide Web Conference (pp. 106-113), Honolulu, Hawaii, May 7-11, 2002. New York: ACM.
16. Egghe, L., & Rousseau, R. (1990). Introduction to informetrics: Quantitative methods in library, documentation and information science. Oxford: Elsevier.
17. Eichmann, D. (1994). The RBSE spider: Balancing effective search against Web load. In Proceedings of the First World Wide Web Conference, Geneva, Switzerland, May 25-27, 1994. Retrieved May 12, 2003, from http://mingo.info- science.uiowa.edu/eichmann/www94/Spider/Spider.html
18. Gordon, M., & Pathak, P. (1999). Finding information on the World Wide Web: The retrieval effectiveness of search engines. Information Processing and Management, 35(2), 141-180.
19. Harvest-NG, (2003). Harvest-NG homepage. Retrieved June 18, 2003, from http://webharvest.sourceforge.net/ng/
20. Hawking, D., Craswell, N., Thistlewaite, P., & Harman, D. (1999). Results and challenges in Websearch evaluation. Computer Networks, 31(11-16), 1321-1330.
21. Henzinger, M.R. (2001). Hyperlink analysis for the Web. IEEE Internet Computing, 5(1), 45-50.
22. Henzinger, M.R. (2003). Algorithmic challenges in Web search engines. Internet Mathematics, 1(1), 115-126. Retrieved September 10, 2003, from http://www.internetmathematics.org/volumes/1/1/pp 115_123.pdf
23. Kerlinger. F.N., & Lee, H.B. (2000). Foundations of behavioral research (4th ed.), London: Harcourt College.
24. Kleinberg, J.M., Kumar, R., Raghavan, P., Rajagopalan, S., & Tomkins, A.S. (1999). The Web as a graph: measurements, models, and methods. In T. Asano, H. Imai, D.T. Lee, S.-I. Nakano, & T. Tokuyama (Eds.), Proceedings of the 5th International Conference on Computing and Combinatorics (pp. 1-17), Tokyo, Japan, July 26-28, 1999. London: Springer.
25. Koster, M. (1993). Guidelines for robot writers. Retrieved May 12, 2003, from http://www.robotstxt.org/wc/guidelines.html
26. Koster, M. (2003). The Web robots page. Retrieved May 12, 2003, from http://www.robotstxt.org/wc/robots.html
27. Lawrence. S., & Giles, C.L. (1998). Searching the World Wide Web. Science, 280, 98-100.
28. Lawrence, S., & Giles. C.L. (1999). Accessibility of information on the Web. Nature, 400, 107-109.
29. Meghabghab, G. (2001). Google's Web page ranking applied to different topological Web graph structures. Journal of the American Society for Information Science and Technology, 52(9), 736-747.
30. Mettrop, W., & Nieuwenhuysen, P. (2001). Internet search engines: Fluctuations in document accessibility. Journal of Documentation, 57(5), 623-651.
31. Najork, M., & Heydon, A. (2001). High-performance Web crawling. Research Report 173, Compaq: Systems Research Center, Palo Alto, California. Retrieved December 10, 2003, from ftp://gatekeeper.research.compaq.com/pub/DEC/ SRC/research-reports/SRC-173.pdf
32. Najork, M., & Wiener, J.L. (2001). Breadth-first crawling yields highquality pages. Proceedings of the 10th World Wide Web Conference (pp. 114-118), Hong Kong, May 1-5, 2001. New York: ACM.
33. Stewart, D.W., & Kamins, M.A. (1993). Secondary research: Information sources and methods (2nd ed.), London: Sage.
34. Tadić, B. (2002). Growth and structure of the World Wide Web: Towards realistic modelling. Computer Physics Communications, 147(1-2), 586-589.
35. Thelwall, M. (2001). A Web crawler design for data mining. Journal of Information Science, 27(5), 319-325.
36. Thelwall, M. (2002). Methodologies for crawler based surveys. Internet Research: Electronic Net-Working Applications and Policy, 12(2), 124-138.
37. Wall, L., Christiansen, T., & Schwartz, R.L. (1996). Programming Perl (2nd ed.), Cambridge, MA: O'Reilly.