Web robot detection in the scholarly information environment

Journal of Information Science

Volumn 34, Issue 5, 2008, Pages 726-741

  Huntington, Paul ^a   Nicholas, David ^a   Jamali, Hamid R ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Electronic journals; Robot detection; Web crawlers; Web log analysis

Indexed keywords

INTERNET; INTERNET PROTOCOLS; ROBOTICS; WORLD WIDE WEB;

ELECTRONIC JOURNALS; GLYCOBIOLOGY; INFORMATION ENVIRONMENT; OPEN ACCESS; OXFORD UNIVERSITY; ROBOT DETECTION; USAGE PATTERNS; WEB CRAWLERS; WEB LOG ANALYSIS; WEB ROBOT DETECTION;

ROBOTS;

EID: 51849090709     PISSN: 01655515     EISSN: 17416485     Source Type: Journal    
DOI: 10.1177/0165551507087237     Document Type: Article

References (14)

1. D. Nicholas, P. Huntington and H.R. Jamali, The impact of open access publishing (and other access initiatives) on use and users of digital scholarly journals, Learned Publishing 20 (1) (2007) 11-15.
2. A. Gutzman, Analysing Traffic on Your E-commerce Site (1999). Available at: www.ecommerceguide.com/solutions/technology/article.php/9561_186011 (accessed 21 October 2007).
3. D. Nicholas, P. Huntington and H.R. Jamali, Open access in context: a user study, Journal of Documentation 63 (6) (2007) 853-78.
4. P. Tan and V. Kumar, Discovery of web robot sessions based on their navigational patterns, Data Mining and Knowledge Discovery 6 (1) (2002) 9-35.
5. A.G. Loureno and O.O. Belo, Catching web crawlers in the act. In: D. Wolber et al. (eds), Proceedings of the 6th International Conference on Web Engineering, Palo Alto, California, USA (New York, ACM, 2006) 265-72.
6. M. Yoon, Web Robot Detection (2000). Available at: http://photo.net/doc/ robot-detection.html (accessed 20 June 2007).
7. S. Jackson, Building a Better Spider Trap (1998). Available at: www.spiderhunter.com/spidertrap (accessed 21 October 2007).
8. D. Eichman, Ethical web agents, Computer Networks and ISDN Systems 28 (1) (1995) 127-36.
9. M. Koster, Guidelines for Robots Writers (1993). Available at: www.robotstxt.org/wc/guidelines.html (accessed 21 October 2007).
10. The Robots Exclusion Standard (1996). Available at: www.robotstxt.org/wc/ exclusion.html#robotstxt (accessed 21 October 2007).
11. M.D. Dikaiakosa, A. Stassopouloub and L. Papageorgioua, An investigation of web crawler behavior: characterization and metrics, Computer Communications 28 (8) (2005) 880-97.
12. V.S. Pai, L. Wang, K. Park, R. Pang and L. Peterson, The dark side of the web: an open proxy's view, ACM SIGCOMM Computer Communications Review 34 (1) (2004) 57-62.
13. K. Park, V.S. Pai, K.W. Lee and S. Calo, Securing web service by automatic robot detection. In: Proceedings of USENIX Annual Technical Conference, 30 May to 3 June 2006, Boston, MA, USA (USENIX, Boston, MA, 2006) 225-60.
14. L. von Ahn, M. Blum, N. Hopper and J. Langford, CAPTCHA: using hard AI problems for security. In: E. Biham (ed.), Advances in Cryptology: Proceedings of EUROCRYPT, International Conference on the Theory and Applications of Cryptographic Techniques, Warsaw, Poland, May 4-8, 2003 (Springer, Berlin, 2003) 294-311.