The architecture and implementation of an extensible web crawler

Proceedings of NSDI 2010: 7th USENIX Symposium on Networked Systems Design and Implementation

Volumn , Issue , 2010, Pages 329-343

  Hsieh, Jonathan M ^a   Gribble, Steven D ^a   Levy, Henry M ^a  

^a University of Washington   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ECONOMIC AND SOCIAL EFFECTS; SOCIAL NETWORKING (ONLINE); SYSTEMS ANALYSIS; WEB SERVICES;

AND FILTERS; CLIENT APPLICATIONS; FILTER INDEXING; HIGH SELECTIVITY; LOW LATENCY; PROTOTYPE IMPLEMENTATIONS; REAL-TIME STREAMS; TRADE OFF;

WEB CRAWLER;

EID: 85076769181     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. D. J. Abadi, D. Carney, U. Çetintemel, M. Cherniack, C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. Zdonik. Aurora: a new model and architecture for data stream management. The VLDB Journal, 12(2):120-139, 2003.
2. M. K. Aguilera, R. E. Strom, D. C. Sturman, M. Astley, and T. D. Chandra. Matching events in a content-based subscription system. In Proceedings of the Eighteenth Annual ACM Symposium on Principles of Distributed Computing (PODC'99), Atlanta, GA, 1999.
3. A. V. Aho and M. J. Corasick. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333-340, 1975.
4. M. Altinel and M. J. Franklin. Efficient filtering of XML documents for selective dissemination of information. In Proceedings of the 26th International Conference on Very Large Data Bases (VLDB'00), San Francisco, CA, 2000.
5. Amazon Web Services. Alexa web search platform. http://www.amazon.com/b?ie=UTF8&node= 16265721.
6. E. Amitay, D. Carmel, M. Herscovici, R. Lempel, and A. Soffer. Trend detection through temporal link analysis. Journal of the American Society for Information Science and Technology, 55(14):1270-1281, December 2004.
7. Apache Software Foundation. Apache lucene. http://lucene.apache.org/.
8. S. Brin and L. Page. The anatomy of a large-scale hyper-textual web search engine. In Proceedings of the Seventh International Conference on the World Wide Web (WWW7), Brisbane, Australia, 1998.
9. A. Carzaniga, D. S. Rosenblum, and A. L. Wolf. Achieving scalability and expressiveness in an Internet-scale event notification service. In Proceedings of the Nineteenth Annual ACM Symposium on Principles of Distributed Computing (PODC'00), Portland, OR, 2000.
10. J. Chen, D. J. Dewitt, F. Tian, and Y. Wang. NiagaraCQ: A scalable continuous query system for Internet databases. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, Dallas, TX, May 2000.
11. M. Cherniack, H. Balakrishnan, M. Balazinska, D. Carney, U. Cetintemel, Y. Xing, and S. Zdonik. Scalable distributed stream processing. In Proceedings of the First Biennial Conference on Innovative Data Systems Research (CIDR'03), Asilomar, CA, January 2003.
12. J. Cho and S. Rajagopalan. A fast regular expression indexing engine. In Proceedings of the 18th International Conference on Data Engineering (ICDE'02), San Jose, CA, February 2002.
13. G. Copeland, W. Alexander, E. Boughter, and T. Keller. Data placement in Bubba. ACM SIGMOD Record, 17(3):99-108, 1988.
14. D. J. DeWitt, S. Ghandeharizadeh, D. Schneider, A. Bricker, H.-I. Hsiao, and R. Rasmussen. The Gamma database machine project. IEEE Transactions on Knowledge and Data Engineering, 2(1):44-62, March 1990.
15. O. Etzioni, M. Cafarella, D. Downey, S. Kok, A.-M. Popescu, T. Shaked, S. Soderland, D. S. Weld, and A. Yates. Web-scale information extraction in KnowItAll. In Proceedings of the 13th International Conference on the World Wide Web, New York, NY, May 2004.
16. F. Fabret, A. Jacobsen, F. Llirbat, J. Pereira, K. Ross, and D. Shasha. Filtering algorithms and implementation for very fast publish/subscribe systems. In Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data, Santa Barbara, CA, May 2001.
17. R. Geambasu, S. D. Gribble, and H. M. Levy. Cloud-Views: Communal data sharing in public clouds. In Proceedings of the Workshop on Hot Topics in Cloud Computing (HotCloud'09), San Diego, CA, June 1999.
18. Google. Google alerts. http://www.google.com/ alerts.
19. E. N. Hanson, C. Carnes, L. Huang, M. Konyala, L. Noronha, S. Parthasarathy, J. Park, and A. Vernon. Scalable trigger processing. In Proceedings of the 15th International Conference on Data Engineering, Sydney, Austrialia, March 1999.
20. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. World Wide Web, 2(4):219-229, 1999.
21. H.-T. Lee, D. Leonard, X. Wang, and D. Loguinov. IRLbot: Scaling to 6 billion pages and beyond. In Proceedings of the 17th International World Wide Web Conference, Beijing, China, April 2008.
22. Y. Liu and B. Plale. Survey of publish subscribe event systems. Technical Report TR574, Indiana University, May 2003.
23. A. Majumder, R. Rastogi, and S. Vanama. Scalable regular expression matching on data streams. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, Vancouver, Canada, June 2008.
24. McAfee, Inc. McAfee SiteAdvisor. http://www.siteadvisor.com/.
25. S. McCanne and V. Jacobson. The BSD packet filter: a new architecture for user-level packet capture. In Proceedings of the USENIX Winter 1993 Conference, San Diego, California, January 1993.
26. A. Moshchuk, T. Bragin, S. Gribble, and H. Levy. A crawler-based study of spyware on the Web. In Proceedings of the 13th Annual Network and Distributed Systems Security Symposium (NDSS'06), San Diego, CA, February 2006.
27. J. Pereira, F. Fabret, F. Llirbat, and D. Shasha. Efficient matching for web-based publish/subscribe systems. In Proceedings of the 7th International Conference on Cooperative Information Systems (CoopIS 2000), Eilat, Israel, September 2000.
28. N. Provos, P. Mavrommatis, M. A. Rajab, and F. Mon-rose. All your iFRAMEs point to us. In Proceedings of the 17th USENIX Security Symposium, San Jose, CA, July 2008.
29. I. Rose, R. Murty, P. Pietzuch, J. Ledlie, M. Roussopoulos, and M. Welsh. Cobra: Content-based filtering and aggregation of blogs and RSS feeds. In Proceedings of the 4th USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI 2007), Cambridge, MA, April 2007.
30. S. Rubin, S. Jha, and B. P. Miller. Protomatching network traffic for high throughput network intrusion detection. In Proceedings of the 13th ACM conference on Computer and Communications Security (CCS'06), October 2006.
31. B. Segall, D. Arnold, J. Boot, M. Henderson, and T. Phelps. Content based routing with Elvin4. In Proceedings of the 2000 AUUG Annual Conference (AUUG2K), Canberra, Australia, June 2000.
32. R. Smith, C. Estan, and S. Jha. XFA: Faster signature matching with extended automata. In Proceedings of the 2008 IEEE Symposium on Security and Privacy (Oakland'08), Oakland, CA, May 2008.
33. N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism: Exploring photo collections in 3D. In Proceedings of the 33rd International Conference an Exhibition on Computer Graphics and Interactive Techniques (SIGGRAPH'06), Boston, MA, July 2006.
34. R. Sommer and V. Paxson. Enhancing byte-level network intrusion detection signatures with context. In Proceedings of the 10th ACM conference on Computer and Communications Security (CCS'03), October 2003.
35. T. W. Yan and H. Garcia-Molina. Index structures for selective dissemination of information under the boolean model. ACM Transactions on Database Systems, 19(2):332-364, June 1994.