Catching web crawlers in the act

ICWE'06: The Sixth International Conference on Web Engineering

Volumn , Issue , 2006, Pages 265-272

  Lourenço, Anália ^a   Belo, Orlando ^a  

^a UNIVERSITY OF MINHO   (Portugal)

Author keywords

Data webhousing; Traverse patterns; Web crawler; Web usage mining

Indexed keywords

COPYRIGHTS; DATA MINING; DATA WAREHOUSES; DECISION MAKING; REAL TIME SYSTEMS; SECURITY OF DATA; WEBSITES;

DATA DIFFERENTIATION; DATA WEBHOUSING; TRAVERSE PATTERNS; WEB CRAWLER; WEB USAGE MINING;

COMPUTER CRIME;

EID: 34250684198     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1145581.1145634     Document Type: Conference Paper

References (17)

1. Acme.Spider: Java class used in custom robot construction. http://www.acme.com/java/software/Acme.Spider.html.
2. Almeida, V., Menasce, D. A., Riedi, R. H., Peligrinelli, F., Fonseca, R. C., and Meira Jr, W. Analyzing Web Robots and their Impact on Caching. In Proceedings of the 6th Web Caching and Content Delivery Workshop (Boston MA, June 2001).
3. BotSpot: the spot for all bots. http://www.botspot.com/.
4. Clifton, C. and Gengo, G. Developing Custom Intrusion Detection Filters Using Data Mining. 2000 Military Communications International Symposium (MILCOM2000), (Los Angeles, California, October 22-25, 2000).
5. Dikaiakos, MD., Stassopoulou, A. and Papageorgiou, L. An investigation of web crawler behavior: characterization and metrics. Computer Communications 28 (8) (May 2005), 880-897.
6. Eichmann, D. Ethical Web agents. Computer Networks and ISDN Systems, Volume 28, Issues 1-2, (December 1995), 127-136.
7. Grishman, R., Huttunen, S. and Yangarber, R. Information extraction for enhanced access to disease outbreak reports. Journal of Biomedical Informatics, Volume 35, Issue 4, (August 2002), pp. 236-246.
8. Koster, M. Guidelines for robots writers. 1993. http://www.robotstxt.org/ wc/guidelines.html.
9. Lee, W. and Stolfo, S. J. A Framework for Constructing Features and Models for Intrusion Detection Systems. ACM Transactions on Information and System Security, 3(4), (2000).
10. Natura Project Web site, http://natura.di.uminho.pt/.
11. Page, L., Brin, S., Motwani, R., and Winograd, T. The PageRank Citation Ranking: Bringing Order to the Web. Stanford Digital Library Technologies Project, 1998.
12. Pant, G., Srinivasan, P., and Menczer, F. Crawling the Web. In M. Levene and A. Poulovassilis (Eds.), Web Dynamics: Adapting to Change in Content, Size, Topology and Use. Springer-Verlag. 2004.
13. Pant, G., Bradshaw, S. and Menczer, F. Search engine-crawler symbiosis: Adapting to Community Interests. In Proceedings of the 7th European Conference on Research and Advanced Technology for Digital Libraries. (Trondheim, Norway, August, 2003).
14. Quinlan, J. C4.5: Programs for Machine Learning. Morgan Kaufmann. 1993.
15. Tan, P.N. and Kumar, V. Discovery of Web Robot Sessions based on their Navigational Patterns. Data Mining and Knowledge Discovery, 6(1), (2002), 9-35.
16. The Robots Exclusion Standard. http://www.robotstxt.org/wc/exclusion. htm#robotstxt.
17. The Web Robots Pages, http://www.robotstxt.org/.