Pubcrawl: Protecting users and businesses from crawlers

Proceedings of the 21st USENIX Security Symposium

Volumn , Issue , 2012, Pages 507-522

  Jacob, Gregoire ^a   Kirda, Engin ^b   Kruegel, Christopher ^a   Vigna, Giovanni ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b NORTHEASTERN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

SOCIAL NETWORKING (ONLINE); SOCIAL SCIENCES COMPUTING; TECHNOLOGY TRANSFER;

AUTOMATED TOOLS; BLACK MARKET; HIGH-ACCURACY; LEGITIMATE USERS; PHISHING; SERVICE PROVIDER; SOCIAL NETWORKING SITES; USER TRAFFICS;

WEB CRAWLER;

EID: 84929888955     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. The R Project for Statistical Computing. http://www.r-project.org/.
2. 67th District Court, Tarrant County, Texas. Cause NO. 067-194022-02: American Airlines, Inc. vs. FareChase, Inc. http://www.fornova.net/documents/AAFareChase.pdf, 2003.
3. B. L. Bowerman, R. T. O’Connell, and A. B. Koehler. Forecasting, Time Series, and Regression – An applied approach – Fourth edition. Thomson Brook/Cole, 2005.
4. E. Burzstein, R. Bauxis, H. Paskov, D. Perito, C. Fabry, and J. Mitchell. The Failure of Noise-Based Non-Continuous Audio Captchas. In IEEE Security and Privacy, 2011.
5. P. Clark and T. Niblett. The CN2 Induction Algorithm. Machine Learning, 3(4):261–283, 1989.
6. R. B. Cleveland, W. S. Cleveland, J. E. McRae, and I. Terpenning. STL: a Seasonal-Trend Decomposition Procedure based on Loess. Journal of Official Statistics, 6(1):3–73, 1990.
7. W. S. Cleveland and S. J. Devlin. Locally Weighted Regression: an Approach to Regression Analysis by Local Fitting. J. Am. Stat. Assoc, 83:596–610, 1988.
8. M. D. Dikaiakosa, A. Stassopouloub, and L. Papageorgioua. An Investigation of Web Crawler Behavior: Characterization and Metrics. Computer Networks, 28:880–897, 2005.
9. D. Doran and S. S. Gokhale. Discovering New Trends in Web Robot Traffic through Functional Classification. In Proc. of the IEEE International Symposium on Networking Computing and Applications (NCA), pages 275–278, 2008.
10. D. Doran and S. S. Gokhale. Web Robot Detection Techniques: Overview and Limitations. Data Mining and Knowledge Discovery, 22(1-2):183–210, 2011.
11. W. Guo, S. Ju, and Y. Gu. Web Robot Detection Techniques based on Statistics of their Requested URL Resources. In Proc. of the International Conference on Computer Supported Cooperative Work in Design (CSCWD), pages 302–306, 2005.
12. R. Gusella. Characterizing the Variability of Arrival Processes with Indexes of Dispersion. ”IEEE J. Sel. Areas Commun., 9(2):203–211, 1991.
13. E. Keogh and S. Kasetty. On the Need for Time Series Data Mining Benchmarks: a Survey and Empirical Demonstration. In Proc. of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD), pages 102–111, 2002.
14. M. Koster. A Method for Web Robots Control. Technical report, RFC draft, 1996.
15. M. Lamberton, E. Levy-Abegnoli, and P. Thubert. System and Method for Enabling a Web Site Robot Trap. United States Patent No. US 6,925,465 B2, 2005.
16. T. W. Liao. Clustering of Time Series Data – a Survey. Pattern Recognition, 38(11):1857–1874, 2005.
17. X. Lin, L. Quan, and H. Wu. An Automatic Scheme to Categorize User Sessions in Modern HTTP Traffic. In Proc. of GLOBE-COM, pages 1485–1490, 2008.
18. A. G. Lourenço and O. O. Belo. Applying Clickstream Data Mining to Real-Time Web Crawler Detection and Containment using ClickTips Platform. In Advances in Data Analysis, Proc. of the 30th Annual Conference of the Gesellschaft für Klassifikation e.V., pages 351–358. Springer, 2007.
19. A. McCallum and K. Nigam. A Comparison of Event Models for Naive Bayes Text Classification. In AAAI FICML Workshop on Learning for Text Categorization, 1998.
20. Pinsent Masons. Ryanair wins German court victory in screen-scraping injunction. http://www.theregister.co.uk/2008/07/11/ryanair_screen_scraping_ victory/, 2008.
21. A. Stassopoulou and M. D. Dikaiakos. Crawler detection: A bayesian approach. In Proc. of the International Conference on Internet Surveillance and Protection (ICISP), 2006.
22. P.-N. Tan and V. Kumar. Discovery of Web Robot Sessions based on their Navigational Patterns. Data Mining and Knowledge Discovery, 6(1):9–35, 2002.
23. Tech Crunch. Hacker arrested for blackmailing StudiVZ and other social networks. http://eu.techcrunch.com/2009/10/21/ hacker-arrested-for-blackmailing-studivz- and-other-social-networks/, 2009.
24. L. von Ahn, M. Blum, N. Hopper, and J. Langford. The CAPTCHA Project. Technical report, Carnegie Mellon University, 2000.
25. P. Warden. How I got sued by Facebook. http://petewarden.typepad.com/searchbrowser/ 2010/04/how- i- got- sued- by- facebook.html, 2010.