SpongeFiles: Mitigating data skew in MapReduce using distributed memory

Proceedings of the ACM SIGMOD International Conference on Management of Data

Volumn , Issue , 2014, Pages 551-562

  Elmeleegy, Khaled ^a   Olston, Christopher ^b   Reed, Benjamin ^c  

^a Platform Applied Science Infrastructure Amobee Inc   (United States)

^b GOOGLE INC   (United States)

^c FACEBOOK   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATABASE SYSTEMS;

DISK CONTENTION; DISTRIBUTED MEMORY; IMPROVING PERFORMANCE; LOCAL MEMORY; PROCESSING ENVIRONMENTS; PROCESSING PLATFORM; REMOTE MEMORY; SPARE CAPACITY;

DATA PROCESSING;

EID: 84904315886     PISSN: 07308078     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2588555.2595634     Document Type: Conference Paper

References (32)

1. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the outliers in map-reduce clusters using mantri. In Proc. OSDI, 2010.
2. HBase: the Hadoop database. http:///hbase.apache.org/.
3. Pig: Query processing system on Hadoop. http://pig.apache.org.
4. M. G. Bulmer. Principles of statistics. Courier Dover Publications, 1979.
5. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A distributed storage system for structured data. ACM Trans. Computer Systems, 26(2), 2008.
6. D. Comer and J. Griffoen. A new design for distributed systems: The remote memory model. In Proceedings of the Summer 1990 USENIX Conference, USENIX '90, 1990.
7. B. F. Cooper, R. Ramakrishnan, U. Srivastava, A. Silberstein, P. Bohannon, H.-A. Jacobsen, N. Puz, D. Weaver, and R. Yerneni. Pnuts: Yahoo!'s hosted data serving platform. Proc. VLDB Endow., 1(2):1277-1288, Aug. 2008.
8. M. D. Dahlin, O. Y. Wang, T. E. Anderson, and D. A. Patterson. Cooperative caching: Using remote client memory to improve file system performance. In In Proceedings of the First Symposium on Operating Systems Design and Implementation, pages 267-280, 1994.
9. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In Proc. OSDI, 2004.
10. D. J. DeWitt and J. Gray. Parallel database systems: The future of high performance database systems. CACM, 35(6), 1992.
11. D. J. DeWitt, J. F. Naughton, D. A. Schneider, and S. Seshadri. Practical skew handling in parallel joins. In Proc. VLDB, 1992.
12. D. J. DeWitt and M. Stonebraker. MapReduce: A major step backwards. http://databasecolumn.vertica.com/database-innovation/mapreduce-a-major-step- backwards/.
13. K. Elmeleegy. Piranha: Optimizing short jobs in hadoop. Proc. VLDB Endow., 6(11):985-996, Aug. 2013.
14. M. J. Feeley, W. E. Morgan, E. P. Pighin, A. R. Karlin, H. M. Levy, and C. A. Thekkath. Implementing global memory management in a workstation cluster. In Proceedings of the fifteenth ACM symposium on Operating systems principles, SOSP '95, pages 201-212, New York, NY, USA, 1995. ACM.
15. A. F. Gates, O. Natkovich, S. Chopra, P. Kamath, S. M. Narayanamurthy, C. Olston, B. Reed, S. Srinivasan, and U. Srivastava. Building a high-level dataow system on top of map-reduce: The Pig experience. In Proc. VLDB, 2009.
16. K. A. Hua and C. Lee. Handling data skew in multiprocessor database computers using partition tuning. In Proc. VLDB, 1991.
17. P. Hunt, M. Konar, F. P. Junqueira, and B. Reed. Zookeeper: wait-free coordination for internet-scale systems. In USENIXATC'10: Proceedings of the 2010 USENIX conference on USENIX annual technical conference, pages 11-11, 2010.
18. Y. Kwon, M. Balazinska, B. Howe, and J. Rolia. Skewtune: mitigating skew in mapreduce applications. In SIGMOD '12: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pages 25-36, New York, NY, USA, 2012. ACM.
19. J. Lin. The curse of Zipf and the limits to parallelization: A look at the stragglers problem in MapReduce. In Proc. Workshop on Large-Scale Distributed Systems for Information Retrieval, 2009.
20. E. Markatos, E. P. Markatos, G. Dramitinos, and G. Dramitinos. Implementation of a reliable remote memory pager. In In USENIX Annual Technical Conference, 1996.
21. I. McDonald. Remote paging in a single address space operating system supporting quality of service. Technical report, Department of Computing Science, University of Glasgow, Scotland, UK, 1999.
22. Memcached: A distributed memory object caching system. http://memcached.org/.
23. Microsoft. Sql server 2008 r2 documentation. http://msdn.microsoft.com/ en-us/library/ms191514.aspx.
24. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A not-so-foreign language for data processing. In Proc. ACM SIGMOD, 2008.
25. D. Ongaro, S. M. Rumble, R. Stutsman, J. Ousterhout, and M. Rosenblum. Fast crash recovery in ramcloud. In SOSP '11: Proceedings of the twentieth ACM symposium on Operating systems principles, 2011.
26. Oracle. Oracle database concepts - memory architecture, 10g release 2. http://docs.oracle.com/cd/B19306-01/server.102/b14220/memory.htm.
27. A. Pavlo, C. Curino, and S. Zdonik. Skew-aware automatic database partitioning in shared-nothing, parallel oltp systems. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12, pages 61-72, New York, NY, USA, 2012. ACM.
28. R. Ramakrishnan and J. Gehrke. Database management systems (3. ed.). McGraw-Hill, 2003.
29. C. B. Walton, A. G. Dale, and R. M. Jenevein. A taxonomy and performance model of data skew effects in parallel joins. In Proc. VLDB, 1991.
30. D. Williams, H. Jamjoom, Y.-H. Liu, and H. Weatherspoon. Overdriver: handling memory overload in an oversubscribed cloud. In Proceedings of the 7th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, VEE '11, 2011.
31. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of the 5th European conference on Computer systems, EuroSys '10, pages 265-278, New York, NY, USA, 2010. ACM.
32. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving mapreduce performance in heterogeneous environments. In OSDI'08: Proceedings of the 8th USENIX conference on Operating systems design and implementation, pages 29-42, 2008.