Sailfish: A framework for large scale data processing

Proceedings of the 3rd ACM Symposium on Cloud Computing, SoCC 2012

Volumn , Issue , 2012, Pages

  Rao, Sriram ^a   Ramakrishnan, Raghu ^a   Silberstein, Adam ^b   Ovsiannikov, Mike ^c   Reeves, Damian ^c  

^a MICROSOFT   (United States)

^b LinkedIn ^*  (United States)

^c Quantcast Corp   (United States)

Author keywords

Design; Experimentation; Performance

Indexed keywords

AUTOTUNING; BATCH DATA; DATA AGGREGATION; DATA SETS; DATA VOLUME; DISTRIBUTED FILE-SYSTEM; EXPERIMENTATION; LARGE SCALE DATA; MAP TASK; MAP-REDUCE; OPEN SOURCES; PERFORMANCE; SKEWED DISTRIBUTION; SOFTWARE COMPONENT;

CLOUD COMPUTING; DATA PROCESSING; DESIGN;

OPEN SYSTEMS;

EID: 84870534435     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2391229.2391233     Document Type: Conference Paper

References (29)

1. Apache Hadoop NextGen MapReduce (YARN). http://hadoop.apache.org/docs/r0. 23.0/hadoop-yarn/hadoop-yarn-site/YARN.html.
2. Apache Hadoop Project. http://hadoop.apache.org/.
3. HDFS. http://hadoop.apache.org/hdfs.
4. KFS. http://code.google.com/p/kosmosfs/.
5. Preemption and restart of mapreduce tasks. http://issues.apache.org/jira/ browse/MAPREDUCE-4585.
6. Sailfish. http://code.google.com/p/sailfish/.
7. Sort benchmark home page. http://sortbenchmark.org/.
8. G. Ananthanarayanan, C. Douglas, R. Ramakrishnan, S. Rao, and I. Stoica. True Elasticity in Multi-Tenant Clusters through Amoeba. In ACM Symposium on Cloud Computing, SoCC'12, October 2012.
9. J. Dean. Software engineering advice from building large-scale distributed systems. http://research.google.com/people/jeff/stanford-295-talk. pdf.
10. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. In OSDI'04: Proceedings of the 6th conference on Symposium on Operating Systems Design & Implementation, 2004.
11. J. Dean and S. Ghemawat. Mapreduce: A flexible data processing tool. Communications of the ACM, 53(1):72-77, January 2010.
12. J. Dittrich, J.-A. Quiané-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad. Hadoop++: Making a yellow elephant run like a cheetah (without it even noticing). Proc. VLDB Endow., 3(1), 2010.
13. S. Ghemawat, H. Gobioff, and S. T. Leung. The Google file system. In Proceedings of the nineteenth ACM symposium on Operating systems principles, volume 37 of SOSP '03, pages 29-43, New York, NY, USA, Oct. 2003.
14. H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu. Starfish: A self-tuning system for big data analytics. Systems Research, pages 261-272, 2011.
15. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In EuroSys '07: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, pages 59-72, 2007.
16. D. Jiang, B. C. Ooi, L. Shi, and S. Wu. The performance of mapreduce: an in-depth study. Proc. VLDB Endow., 3(1), Sept. 2010.
17. D. E. Knuth. Sorting and Searching, volume 3 of The Art of Computer Programming. Addison-Wesley Professional, second edition, May 1998.
18. Y. Kwon, M. Balazinska, B. Howe, and J. Rolia. Skew-resistant parallel processing of feature-extracting scientific user-defined functions. In Proceedings of the 1st ACM symposium on Cloud computing, SoCC '10, pages 75-86, New York, NY, USA, 2010. ACM.
19. A. Murthy. Apache hadoop: Best practices and anti-patterns. http://developer.yahoo.com/blogs/hadoop/posts/2010/08/apache/hadoop/best/ practices/a/.
20. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In SIGMOD '08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1099-1110, 2008.
21. J. Ousterhout et al. The case for ramclouds: Scalable high-performance storage entirely in dram. SIGOPS Operating Systems Review, 43(4):92-105, December 2009.
22. A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, and M. Stonebraker. A comparison of approaches to large-scale data analysis. In Proceedings of the 35th SIGMOD international conference on Management of data, SIGMOD '09, pages 165-178, New York, NY, USA, 2009. ACM.
23. A. Rasmussen, M. Conley, R. Kapoor, V. The Lam, G. Porter, and A. Vahdat. ThemisMR: An I/O Efficient MapReduce. Technical Report CS2012-0983, Department of Computer Science and Engineering, University of California at San Diego, July 2012.
24. A. Rasmussen, M. Conley, G. Porter, and A. Vahdat. Tritonsort 2011. http://sortbenchmark.org/2011/06/tritonsort.pdf.
25. A. Rasmussen, G. Porter, M. Conley, H. V. Madhyastha, R. N. Mysore, A. Pucher, and A. Vahdat. Tritonsort: a balanced large-scale sorting system. In Proceedings of the 8th USENIX conference on Networked systems design and implementation, NSDI'11, Berkeley, CA, USA, 2011.
26. S. Rao, R. Ramakrishnan, A. Silberstein, M. Ovsiannikov, D. Reeves. Sailfish: A framework for large scale data processing. Technical Report YL-2012-002, Yahoo! Labs.
27. M. Stonebraker, D. Abadi, D. J. DeWitt, S. Madden, E. Paulson, A. Pavlo, and A. Rasin. Mapreduce and parallel dbmss: friends or foes? Commun. ACM, 53(1):64-71, Jan. 2010.
28. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive: a warehousing solution over a map-reduce framework. Proc. VLDB Endow., 2(2), 2009.
29. R. Vernica, A. Balmin, K. S. Beyer, and V. Ercegovac. Adaptive MapReduce using Situation-Aware Mappers. In International Conference on Extending Database Technology (EDBT), 2012.