Web crawling ethics revisited: Cost, privacy, and denial of service

Journal of the American Society for Information Science and Technology

Volumn 57, Issue 13, 2006, Pages 1771-1779

  Thelwall, Mike ^a   Stuart, David ^a  

^a UNIVERSITY OF WOLVERHAMPTON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

COMMUNICATION SYSTEMS; COST ACCOUNTING; COST BENEFIT ANALYSIS; WEBSITES;

COMMUNICATION TECHNOLOGY; ETHICAL CRAWLING; TECHNOLOGICAL ISSUES; WEB CRAWLERS;

INFORMATION SCIENCE;

EID: 33750897224     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.20388     Document Type: Article

References (68)

1. ACM. (1992). ACM code of ethics and professional conduct. Retrieved March 1, 2005, from http://www.acm.org/constitution/code.html
2. Arasu, A., Cho, J., Garcia-Molina, H., Paepcke, A., & Raghavan, S. (2001). Searching the Web. ACM Transactions on Internet Technology, 1(1), 2-43.
3. Arrington, R.L. (1998). Western ethics: An historical introduction. Oxford, England: Blackwell.
4. BBC News. (2005, January 24). US duo in first spam. Retrieved February 23, 2005, from http://news.bbc.co.uk/1/hi/technoIogy/3981099.stm
5. Björneborn, L. (2004). Small-world link structures across an academic Web space - A library and information science approach. Copenhagen, Denmark: Royal School of Library and Information Science.
6. Borrull, A.L., & Oppenheim, C. (2004). Legal aspects of the Web. Annual Review of Information Science & Technology, 38, 483-548.
7. Brin, S., & Page, L. (1998). The anatomy of a large scale hypertextual Web search engine. Computer Networks and ISDN Systems, 30(1-7), 107-117.
8. Buell, K.C. (2000). Start spreading the news: Why republishing material from disreputable news reports must be constitutionally protected. New York University Law Review, 75(4), 966-1003.
9. Calluzzo, V.J., & Cante, C.J. (2004). Ethics in information technology and software use. Journal of Business Ethics, 51(3), 301-312.
10. Carey, P. (2004). Data protection: A practical guide to UK and EU law. Oxford: Oxford University Press.
11. Carlin, A.P. (2003). Disciplinary debates and bases of interdisciplinary studies: The place of research ethics in library and information science. Library and Information Science Research, 25(1), 3-18.
12. Casey, T.D. (2000). ISP survival guide: Strategies for managing copyright, spam, cache and privacy regulations. New York: Wiley.
13. Chakrabarti, S. (2003). Mining the Web: Analysis of hypertext and semi structured data. New York: Morgan Kaufmann.
14. Chambers. (1991). Chambers concise dictionary. Edinburgh: W & R Chambers Ltd.
15. Davies, M. (2001). Creating and using multi-million word corpora from Webbased newspapers. In R.C. Simpson & J.M. Swales (Eds.), Corpus Linguistics in North America (pp. 58-75). Ann Arbor: University of Michigan.
16. Davis, M. (1991). Thinking like an engineer: The place of a code of ethics in the practice of a profession. Philosophy and Public Affairs, 20(2), 150-167.
17. du Gay, P., Hall, S.. Janes, L., Mackay, H., & Negus, K. (1997). Doing cultural studies: The story of the Sony Walkman. London: Sage.
18. Eichmann, D. (1994). Ethical Web agents. Paper presented at the Second International World Wide Web Conference, Chicago, IL.
19. Ess, C. (with Association of Internet Researchers [AoIR] Ethics Committee) A.E.W. (2002). Ethical decision-making and Internet research. Recommendations from the AoIR ethics working committee. Retrieved February 23, 2005, from http://aoir.org/reports/ethics.pdf
20. Faba-Perez, C., Guerrero-Bote, V. P., & De Moya-Anegon, F. (2003). Data mining in a closed Web environment. Scientometrics, 58(3), 623-640.
21. Fausett, B.A. (2002, October). Into the Deep: How deep linking can sink you. New Architect. Retrieved June 13, 2006, from http://www.ddj.com/dept/ architect/184411709
22. Foot, K.A., Schneider, S.M., Dougherty, M., Xenos, M., & Larsen, E. (2003). Analyzing linking practices: Candidate sites in the 2002 US electoral Web sphere. Journal of Computer Mediated Communication, 8(4). Retrieved February 23, 2005, from http://www.ascusc.org/jcmc/vol8/issue4/foot.html
23. Frankel, M.S., & Siang, S. (1999). Ethical and legal aspects of human subjects research on the internet. Retrieved from American Association for the Advancement of Science Web site. Retrieved June 13, 2006, from http://www.aaas.org/spp/sfrl/projects/intres/main.htm
24. Gopal, R.D., Sanders, G. L., Bhattacharjee, S., Agrawal, M., & Wagner, S. C. (2004). A behavioral model of digital music piracy. Journal of Organisational Computing and Electronic Commerce, 14(2), 89-105.
25. Heimeriks, G., Hörlesberger, M., & van den Besselaar, P. (2003). Mapping communication and collaboration in heterogeneous research networks. Scientometrics, 58(2), 391-413.
26. Holdsworth, D. (1995). Ethical decision-making in science and technology. In B. Almond (Ed.), Introducing applied ethics (pp. 130-147). Oxford, England: Blackwell.
27. IAC Search & Media. (2005). Technology & features: Teoma search technology. Retrieved February 23, 2005, from http://web.archive.org/web/ 20050217044207/http://sp.ask.com/docs/about/tech_teoma.html
28. Internet Archive. (2001, March). Internet Archive's terms of use, privacy policy, and copyright policy. Retrieved February 23, 2005, from http://www.archive.org/about/terms.php
29. Introna, L.D., & Nissenbaum, H. (2000). The Internet as a democratic medium: Why the politics of search engines matters, The Information Society, 16(3), 169-185.
30. Jankowski, N., & van Selm, M. (2001). Research ethics in a virtual world: Some guidelines and illustrations. Retrieved February 23, 2005, from http://oase.uci.kun.nl/n~jankow/Jankowski/publications/ Research_Ethics_in_a_VirtuaI_World.pdf
31. Johnson, D.g. (2004). Computer Ethics. In H. Nissenbaum & M.E. Price (Eds.), Academy & the Internet (pp. 143-167). New York: Peter Lang.
32. Jones, R.A. (1994). The ethics of research in Cyberspace. Internet Research: Electronic Networking Applications and Policy, 4(3), 30-35.
33. Koehler, W. (2002). Web page change and persistence - A four-year longitudinal study. Journal of American Society for Information Science. 53(2), 162-171.
34. Koster, M. (1993). Guidelines for robot writers. Retrieved February 23, 2005, from http://www.robotstxt.org/wc/guidelines.html
35. Koster, M. (1994). A standard for robot exclusion. Retrieved March 3, 2005. from http://www.robotstxt.org/wc/norobots.html
36. Koster, M. (1996). Evaluation of the standard for robots exclusion. Retrieved February 23, 2005, from http://www.robotstxt.org/wc/eval.html
37. Krogh, C. (1996). The rights of agents. In M. Wooldridge, J. P. Muller, & M. Tambe (Eds.). Intelligent agents II, Agent theories, architectures and languages (pp. 1-16). Berlin: Springer Verlag.
38. Lamirel, J.-C, Al Shehabi, S., Francois, C., & Polanco, X. (2004). Using a compound approach based on elaborated neural network for Webometrics: An example issued from the EICSTES project. Scientometrics. 61(3), 427-441.
39. Leydesdorff, L. (2004). The university-industry knowledge relationship: Analyzing patents and the science base of technologies. Journal of the American Society for Information Science and Technology, 54(11), 991-1001.
40. Lin, D., & Loui, M. C. (1998). Taking the byte out of cookies: Privacy, consent and the Web. ACM SIGCAS Computers and Society, 28(2), 39-51.
41. Manion, M., & Goodrum, A. (2000). Terrorism or civil disobedience: Toward a hacktivist ethic. ACM SIGCAS Computers and Society, 30(2), 14-19.
42. Negroponte, N. (1995). Being digital (p. 188). London: Coronet.
43. Nicholas, D., Huntington, P., & Williams, P. (2002). Evaluating metrics for comparing the use of Web sites: a case study of two consumer health Web sites. Journal of Information Science, 28(1), 63-75.
44. Office of Public Sector Information. (1999). The telecommunications (data protection and privacy) regulations 1999. Retrieved February 23, 2005, from http://www.legislation.hmso.gov.uk/si/si1999/19992093.htm
45. Olsen, S. (2003). Google cache raises copyright concerns. Retrieved June 13, 2006, from http://news.com.com/2100-1038_3-1024234.html
46. Penslar, R.L. (1995). Science and coercion. In R.L. Penslar (Ed.), Research ethics: Cases and materials (pp. 125-136). Bloomington, IN: Indiana University Press.
47. Rall, D.N. (2004a). Exploring the breadth of disciplinary backgrounds in Internet scholars participating in AoIR meetings, 2000-2003. Proceedings of AoIR 5.0, Retrieved February 23, 2005, from http://gsb.haifa.ac.il/n~sheizaf/AOIRS/ 399.html
48. Rall, D.N. (2004b). Locating Internet research methods within five qualitative research traditions. Proceedings of the AoIR-ASIST 2004 Workshop on Web Science Research Methods. Retrieved February 23, 2005, from http://cybermetrics.wlv.ac.uk/AoIRASIST/
49. Reiman, J.H. (1995). Driving to the Panopticon: A philosophical exploration of the risks to privacy posed by the highway technology of the future. Santa Clara Computer and High Technology Law Journal, 11(1), 27-44.
50. Rogers, R. (2004). Information politics on the Web. Cambridge, MA: MIT Press.
51. Rooksby, E. (2002). E-mail and ethics. London: Routledge.
52. Schneier, B. (2004). Secrets and lies: Digital security in a networked world. New York: Hungry Minds, Inc.
53. Search Engine Optimization Ethics (2002). SEO code of ethics. Retrieved March 1, 2005, from http://www.searchengineethics.com/seoethics.htm
54. Shaw, William H. (1999). Contemporary ethics: Taking account of utilitarianism. Oxford, England: Blackwell.
55. Stitt, R. (2004). Curbing the Spam problem. IEEE Computer, 37(12), 8.
56. Sullivan, D. (2004). Search engines and legal issues. SearchEngineWatch. Retrieved February 23, 2005, from http://xearchengincwatch.com/resources/ article.php/2156541
57. Underwood, J. (2001). Competitive intelligence. England: Wiley-Capstone.
58. Van Couvering, E. (2004). New media? The political economy of Internet search engines. Paper presented at the Annual Conference of the International Association of Media & Communications Researchers, Porto Alegre, Brazil.
59. Vardy, P., & Grosch, P. (1999). The puzzle of ethics. London: Fount.
60. Vaughan, L. (2004). Web hyperlinks reflect business performance: A study of US and Chinese IT companies, Canadian Journal of Information and Library Science, 28(1), 17-32.
61. Vaughan, L., & Thelwall, M. (2003). Scholarly use of the Web: What are the key inducers of links to journal Web sites? Journal of American Society for Information Science and Technology, 54(1), 29-38.
62. Vaughan, L., & Thelwall, M. (2004). Search engine coverage bias: Evidence and possible causes. Information Processing & Management, 40(4), 693-707.
63. Wilkinson, D., Harries, G., Thelwall, M., & Price, E. (2003). Motivations for academic Web site interlinking: Evidence for the Web as a novel source of information on informal scholarly communication. Journal of Information Science, 29(1), 49-56.
64. Williamson, E., & Smyth M. (Eds.). (2004). Researchers and their subjects, ethics, power, knowledge and consent. Bristol: Policy Press.
65. Wormell, I. (2001). Informetrics and Webometrics for measuring impact, visibility, and connectivity in science, politics and business. Competitive Intelligence Review, 12(1), 12-23.
66. Wouters, P., & de Vries, R. (2004). Formally citing the Web. Journal of the American Society for Information Science and Technology, 55(14), 1250-1260.
67. Wronkiewicz, K. (1997). Spam like fax. Internet World, 8(2), 10.
68. Yahoo! Inc. (2005). How can I reduce the number of requests you make on my Web site? Retrieved February 23, 2005, from http://help.yahoo.com/help/us/ ysearch/slurp/slurp-03.html