Web crawlers on a health related portal: Detection, characterisation and implications

Proceedings - 4th International Conference on Developments in eSystems Engineering, DeSE 2011

Volumn , Issue , 2011, Pages 24-29

  Jawaheer, Gawesh ^a   Kostkova, Patty ^a  

^a CITY UNIVERSITY   (United Kingdom)

Author keywords

Crawlers; Visualisation; Web analytics; Web server logs

Indexed keywords

CRAWLERS; E-MAIL ADDRESS; REMOVAL PROCESS; SANITY CHECK; SPAM DATABASE; TECHNIQUES USED; UNSOLICITED EMAIL; WEB ANALYTICS; WEB CRAWLERS; WEB SERVER LOGS; WEB TRAFFIC;

ELECTRONIC MAIL; SEARCH ENGINES; VISUALIZATION; WEB SERVICES;

PORTALS;

EID: 84858038290     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/DeSE.2011.83     Document Type: Conference Paper

References (6)

1. Y. Sun, Z. Zhuang, and C.L. Giles, "A largescale study of robots.txt, " Proceedings of the 16th international conference on World Wide Web - WWW '07, 2007, p. 1123.
2. C. Kanich, C. Kreibich, K. Levchenko, B. Enright, G. Voelker, V. Paxson, and S. Savage, "Spamalytics: An empirical analysis of spam marketing conversion, " Proceedings of the 15th ACM conference on Computer and communications security, ACM, 2008, p. 3-14.
3. M.D. Dikaiakos, A. Stassopoulou, and L. Papageorgiou, "An investigation of web crawler behavior: characterization and metrics, " Computer Communications, vol. 28, 2005, pp. 880-897. (Pubitemid 40556772)
4. P. Huntington, D. Nicholas, and H.R. Jamali, "Web robot detection in the scholarly information environment, " Journal of Information Science, vol. 34, 2008, pp. 726-741.
5. "Web analytics, " Wikipedia, 2010.
6. R. Varghese, "A New Visualization for Web Server Logs, " O'Reilly Sys Admin, 2010.