Discovery of web robot sessions based on their navigational patterns

Data Mining and Knowledge Discovery

Volumn 6, Issue 1, 2002, Pages 9-35

  Tan, Pang Ning ^a   Kumar, Vipin ^a  

^a 4 174 EE CSCI Building   (United States)

Author keywords

Classification; Data mining; Web robot detection; Web usage mining

Indexed keywords

COMPUTER SOFTWARE; INFORMATION RETRIEVAL; ROBOTICS; WORLD WIDE WEB;

WEB USAGE MINING;

DATA MINING;

EID: 0036109905     PISSN: 13845810     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1013228602957     Document Type: Article

References (20)

1. Balabanovic, M. and Shoham, Y. 1995. Learning information retrieval agents: Experiments with automated web browsing. In Proc. of the AAAI Spring Symposium Series on Information Gathering from Distributed, Heterogeneous Environments, pp. 13-18.
2. Brodley, C. and Friedl, M. 1999. Identifying mislabeled training data. Journal of Artificial Intelligence Research, 11:131-167.
3. Clark, D. 2000. Shopbots become agents for business change. IEEE Computer, 33(2):18-21.
4. Cooley, R. 1999. Web Usage Mining: Discovery and Application of Interesting Patterns from Web Data. Ph.D. Thesis, University of Minnesota.
5. Cooley, R., Mobasher, B., and Srivastava, J. 1999. Data preparation for mining world wide web browsing patterns. Knowledge and Information Systems, 1(1):5-32.
6. Eichmann, D. 1995. Ethical web agents. Computer Networks and ISDN Systems, 28(1):127-136.
7. Graham, L. 2000. Keep your bots to yourself. IEEE Software, 17(6):106-107.
8. Gray, M. 1993. Measuring the growth of the web. http://www.mit.edu/people/mkgray/growth/.
9. Jackson, S. 1998. Building a better spider trap, http://www.spiderhunter.com/spidertrap.
10. Kephart, J. and Qreenwald, A. 1999. Shopbot economics. In Agents, pp. 378-379.
11. Kolar, C. Leavitt, J., and Mauldin, M. 1996. Robot exclusion standard revisited. http://www.kollar.com/robots.html.
12. Koster, M. 1994a. Guidelines for robot writers. http://info.webcrawler.com/mak/projects/robots/guidelines.html.
13. Koster, M. 1994b. A standard for robot exclusion. http://info.webcrawler.com/mak/projects/robots/norobots.html.
14. Koster, M. 1995. Robots in the web: Threat or treat. ConneXions, 9(4):2-12.
15. Lieberman, H. 1995. Letizia: An agent that assists web browsing. In Proc. of the 1995 International Joint Conference on Artificial Intelligence, Montreal, Canada, pp. 924-929.
16. Pirolli, P., Pitkow, J., and Rao, R. 1996. Silk from a sow's ear: Extracting usable structures from the web. In CHI-96, Vancouver, pp. 118-125.
17. Quinlan, J. 1993. C4.5: Programs for Machine Learning. San Mateo, CA: Morgan Kaufmann.
18. Rosenstein, M. 2000. What is actually taking place on web sites: E-commerce lessons from web server logs. In ACM Conference on Electronic Commerce, Minneapolis, MN, pp. 38-43.
19. Van Rijsbergen, C.J. 1979. Information Retrieval. London: Butterworths.
20. Yoon, M. 2000. Web robot detection, http://www.arsdigita.com/doc/robot-detection.