In-situ MapReduce for log processing

Proceedings of the 2011 USENIX Annual Technical Conference, USENIX ATC 2011

Volumn , Issue , 2019, Pages 115-129

  Logothetis, Dionysios ^a   Trezzo, Chris ^b   Webb, Kevin C ^a   Yocum, Kenneth ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b Salesforcecom Inc   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA HANDLING; NETWORK ARCHITECTURE;

CURRENT PRACTICES; CUSTOMER BEHAVIOR; DESIGN AND IMPLEMENTATIONS; DISK BANDWIDTH; GENERATION RATE; PROCESSING SYSTEMS; SECURITY MONITORING; TRADE POTENTIALS;

IN SITU PROCESSING;

EID: 85077087194     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. 1998 World Cup Web Server Logs. http://ita.ee.lbl.gov/html/traces.html.
2. Flume: Open source log collection system. http://github.com/cloudera/flume.
3. The GridMix Hadoop Workload Generator. http://hadoop.apache.org/mapreduce/docs/current/gridmix.html.
4. Windows Azure and Facebook teams. Personal communications, August 2008.
5. D. J. Abadi, Y. Ahmad, M. Balazinska, U. Cetintemel, M. Cherniack, J.-H. Hwang, W. Lindner, A. S. Maskey, A. Rasin, E. Ryvkina, N. Tatbul, Y. Xing, and S. Zdonik. The Design of the Borealis Stream Processing Engine. In Biennial Conference on Innovative Data Systems Research (CIDR), Asilomar, CA, January 2005.
6. S. Agarwal, J. Dunagan, N. Jain, S. Saroiu, and A. Wolman. Volley: Automated data placement for geo-distributed cloud services. In NSDI'10, April 2010.
7. B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and issues in data stream systems. In Symposium on Principles of Database Systems, March 2002.
8. M. Balazinska, H. Balakrishnan, S. Madden, and M. Stonebraker. Fault-tolerance in the Borealis distributed stream processing system. In SIGMOD'05, June 2005.
9. D. Carney, U. Cetintemel, M. Cherniack, C. Convey, S. Lee, G. Seidman, M. Stonebraker, N. Tatbul, and S. Zdonik. Monitoring streams-a new class of data management applications. In VLDB'02, September 2002.
10. T. Condie, N. Conway, P. Alvaro, and J. M. Hellerstein. MapReduce online. In NSDI'10, San Jose, CA, April 2010.
11. F. Dabek, R. Cox, F. Kaashoek, and R. Morris. Vivaldi: A decentralized network coordinate system. In SIGCOMM'04, August 2004.
12. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In OSDI'04, San Francisco, CA, December 2004.
13. M. J. Franklin, S. Krishnamurthy, N. Conway, A. Li, alex Russakovsky, and N. Thomre. Continuous analytics: Rethinking query processing in a network-effect world. In Proc. of CIDR, January 2009.
14. J. Gray, S. Chaudhuri, A. Bosworth, A. Layman, D. Reichart, M. Venkatrao, F. Pellow, and H. Pirahesh. Data cube: A relational aggregation operator generalizing group-by, cross-tab, and subtotals. Data Mining and Knowledge Discovery, 1(1), 1997.
15. J. Gray, S. Chaudhuri, A. Bosworth, A. Layman, D. Reichart, M. Venkatrao, F. Pellow, and H. Pirahesh. Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals. Data Mining and Knowledge Discovery, 1(1):29-53, 1997.
16. B. He, M. Yang, zhenyu Guo, R. Chen, W. Lin, B. Su, and L. Zhou. Batched stream processing: A case for data intensive distributed computing. In ACM Symposium on Cloud Computing (SOCC), June 2010.
17. T. Hoff. How Rackspace Now Uses MapReduce and Hadoop To Query Terabytes of Data, Jan 2008. http://highscalability.com/how-rackspace-now-uses-mapreduce- and-hadoop-query-terabytes-data.
18. R. Huebsch, J. M. Hellerstein, N. L. Boon, T. Loo, S. Shenker, and I. Stoica. Querying the Internet with PIER. In Proc. of VLDB, September 2003.
19. J.-H. Hwang, M. Balazinska, A. Rasin, U. Cetintemel, and S. Zdonik. High-availability algorithms for distributed stream processing. In Proc. of ICDE, April 2005.
20. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In EuroSys'07, March 2007.
21. J. Li, D. Maier, K. Tufte, V. Papdimos, and P. A. Tucker. No pane, no gain: efficient evaluation of sliding-window aggregates over data streams. SIGMOD Record, 34(1), March 2005.
22. D. Logothetis and K. Yocum. Wide-scale data stream management. In USENIX Annual Technical Conf., June 2008.
23. S. Madden, M. Franklin, J. Hellerstein, and W. Hong. TAG: a tiny aggregation service for ad-hoc sensor networks. In OSDI'02, December 2002.
24. S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis. Dremel: Interactive Analysis of Web-Scale Datasets. Proceedings of the VLDB Endowment, 3, September 2010.
25. R. N. Murty and M. Welsh. Towards a dependable architecture for Internet-scale sensing. In Second HotDep06, November 2006.
26. U. Srivastava and J. Widom. Flexible time management in data stream systems. In Proc. of PODS 2004, June 2004.
27. J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan. SALSA: analyzing logs as state machines. In 1st USENIX Workshop on Analysis of System Logs, page 6, December 2008.
28. N. Tatbul, U. Cetintemel, and S. Zdonik. Staying FIT: Efficient load shedding techniques for distributed stream processing. In VLDB'07, August 2007.
29. N. Tatbul and S. Zdonik. Window-aware load shedding for aggregation queries over data streams. In VLDB'06, August 2006.
30. W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan. Mining console logs for large-scale system problem detection. In Workshop on Tackling Computer Systems Problems with Machine Learning Techniques), Dec 2008.
31. P. Yalagandula and M. Dahlin. A scalable distributed information management system. In SIGCOMM'04, September 2004.
32. Y. Yu, P. K. Gunda, and M. Isard. Distributed aggregation for data-parallel computing: Interfaces and implementations. In SOSP'09, October 2009.