BotSeer: An automated information system for analyzing Web robots

Proceedings - 8th International Conference on Web Engineering, ICWE 2008

Volumn , Issue , 2008, Pages 108-114

  Sun, Yang ^a   Councill, Isaac G ^a   Giles, C Lee ^a  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER NETWORKS; INFORMATION SCIENCE; INFORMATION SERVICES; INFORMATION SYSTEMS; ROBOTS; SEARCH ENGINES; WORLD WIDE WEB;

INTERNATIONAL CONFERENCES; WEB ENGINEERING; WEB ROBOTS;

INTERNET PROTOCOLS;

EID: 51749089314     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICWE.2008.27     Document Type: Conference Paper

References (19)

1. B. Berendt and M. Spiliopoulou. Analysis of navigation behaviour in web sites integrating multiple information systems. The VLDB Journal, 9(1):56-75, 2000.
2. M. L. Boonk, D. R. A. d. Groot, F. M. T. Brazier, and A. Oskamp. Agent exclusion on websites. In Proc. of The 4th Workshop on the Law and Electronic Agents, 2005.
3. S. Chakrabarti, M. Van den Berg, , and B. Dom. Focused crawling: A new approach to topic-specific web resource discovery. In Proc. of the 8th WWW Conference, pages 545-562, 1999.
4. J. Cho, H. Garcia-Molina, and L. Page. Efficient crawling through url ordering. In Proceedings of the 7th International World Wide Web Conference, 1998.
5. R. Cooley, B. Mobasher, and J. Srivastava. Data preparation for mining world wide web browsing patterns. Knowledge and Information Systems, 1(1):5-32, 1999.
6. D. Cutting. The lucene search engine. http://lucene.apache.org/.
7. M. Drott. Indexing aids at corporate websites: The use of robots.txt and meta tags. Information Processing and Management, 38(2):209-219, 2002.
8. D. Eichmann. Ethical web agents. Computer Networks and ISDN Systems, 28(1-2):127-136, 1995.
9. C. L. Giles, K. Bollacker, and S. Lawrence. CiteSeer: An automatic citation indexing system. In I. Witten, R. Akscyn, and F. M. Shipman III, editors, The Third ACM Conference on Digital Libraries, pages 89-98, Pittsburgh, PA, 1998. ACM Press.
10. B. Kelly and I. Peacock. Webwatching uk web communities: Final report for the webwatch project. British Library Research and Innovation Report, 1999.
11. M. Koster. A method for web robots control. In the Internet Draft, The Internet Engineering Task Force (IETF), 1996.
12. B. Mobasher, R. Cooley, and J. Srivastava. Automatic personalization based on web usage mining. Commun. ACM, 43(8):142-151, 2000.
13. L. Page and S. Brin. The pagerank citation ranking: bring-ing order to the web. In Tech. report SIDL-WP-1999-0120, Stanford University, 1999.
14. G. Pant, P. Srinivasan, and F. Menczer. Crawling the Web, chapter Web Dynamics. Springer-Verlag, 2004.
15. G. Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. Commun. ACM, 18(11):613-620, 1975.
16. M. Spiliopoulou. Web usage mining for web site evaluation. Commun. ACM, 43(8):127-134, 2000.
17. J. Srivastava, R. Cooley, M. Deshpande, and P.-N. Tan. Web usage mining: discovery and applications of usage patterns from web data. SIGKDD Explor. Newsl., 1(2):12-23, 2000.
18. Y. Sun, Z. Zhuang, I. G. Councill, and C. L. Giles. Determining bias to search engines from robots.txt. In Proceedings of the 2007 IEEE/WIC International Conference on Web Intelligence, San Jose, CA, USA, 2007. IEEE Computer Society.
19. Y. Sun, Z. Zhuang, and C. L. Giles. A large-scale study of robots.txt. In Proc. of the 16th international conference on World Wide Web, 2007.