Scale-up vs scale-out for hadoop: Time to rethink?

Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013

Volumn , Issue , 2013, Pages

  Appuswamy, Raja ^a   Gkantsidis, Christos ^a   Narayanan, Dushyanth ^a   Hodson, Orion ^a   Rowstron, Antony ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER PROGRAMMING; COMPUTER SCIENCE;

APPLICATION CODES; CLUSTER SCALE; DATA ANALYTICS; PETA-SCALE; REAL-WORLD; SCALE OUTS; SCALING-UP; SINGLE SERVER;

CLOUD COMPUTING;

EID: 84893331360     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2523616.2523629     Document Type: Conference Paper

References (33)

1. Amazon Simple Storage Service (Amazon S3). http://aws.amazon.com/s3/. Accessed: 08/09/2011.
2. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, and I. Stoica. "PACMan: Coordinated Memory Caching for Parallel Jobs,". NSDI. 2012.
3. D. G. Andersen, J. Franklin, M. Kaminsky, A. Phanishayee, L. Tan, and V. Vasudevan. "FAWN: A Fast Array of Wimpy Nodes,". Proceedings of SOSP. 2009.
4. Apache Hadoop. http://hadoop.apache.org/. Accessed: 08/09/2011.
5. Apache Mahout. http://mahout.apache.org/. Accessed: 02/07/2013.
6. Apache Pig Wiki. http://wiki.apache.org/pig/PigPerformance. Accessed: 02/07/2013.
7. M. Bierman and L. Grimmer. How I Use the Advanced Capabilities of Btrfs. http://www.oracle.com/technetwork/articles/servers-storage-admin/ advanced-btrfs-1734952.html. Accessed: 02/07/2013. 2012.
8. J. Bonwick. ZFS End-to-End Data Integrity. https://blogs.oracle.com/ bonwick/entry/zfs-end-to-end-data. Accessed: 02/07/2013. 2005.
9. B. Calder et al. "Windows Azure Storage: a highly available cloud storage service with strong consistency,". Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles. SOSP'11. ACM, 2011, pp. 143-157.
10. R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. "SCOPE: easy and efficient parallel processing of massive data sets,". Proceedings of the VLDB Endowment 1.2 (2008), pp. 1265-1276.
11. R. Chen, H. Chen, and B. Zang. "Tiled-MapReduce: optimizing resource usages of dataparallel applications on multicore with tiling,". Proceedings of the 19th international conference on Parallel architectures and compilation techniques. PACT'10. ACM, 2010.
12. Y. Chen, S. Alspaugh, and R. H. Katz. "Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads,". PVLDB 5.12 (2012), pp. 1802-1813.
13. Cloudera. Tips and Guidelines: Improving Performance. http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/ CDH4-Installation-Guide/cdh4ig-topic-11-6.html. Accessed: 02/07/2013.
14. J. Dean and S. Ghemawat. "MapReduce: Simplified Data Processing on Large Clusters,". OSDI. 2004.
15. D. DeWitt and J. Gray. "Parallel Database Systems: The Future of High Performance Database Systems,". Communications of the ACM 35.6 (1992), pp. 85-98.
16. K. Elmeleegy. "Piranha: Optimizing Short Jobs In Hadoop,". VLDB. 2013.
17. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. Katz, S. Shenker, and I. Stoica. "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center,". Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation. NSDI'11. USENIX, 2011.
18. A. Kyrola, G. Blelloch, and C. Guestrin. "GraphChi: large-scale graph computation on just a PC,". Proceedings of the 10th USENIX conference on Operating Systems Design and Implementation. OSDI'12. USENIX Association, 2012, pp. 31-46.
19. W. Lang, J. M. Patel, and S. Shankar. "Wimpy Node Clusters: What About Non-Wimpy Workloads?," Workshop on Data Management on New Hardware (DaMon). 2010.
20. Y. Mao, R. Morris, and F. Kaashoek. Optimizing MapReduce for Multicore Architectures. Tech. rep. MIT-CSAIL-TR-2010-020. MIT CSAIL, 2010.
21. M. Michael, J. E. Moreira, D. Shiloach, and R. W. Wisniewski. "Scale-up x Scale-out: A Case Study using Nutch/Lucene,". Proceedings of the IEEE International Symposium on Parallel and Distributed Processing. IPDPS'07. IEEE, 2007, pp. 1-8.
22. S. Owen, R. Anil, T. Dunning, and E. Friedman. Mahout in Action. Manning Publications Co., 2011.
23. Panasas. Accelerating and Simplifying Apache Hadoop with Panasas ActiveStor. http://www.panasas.com/sites/default/files/uploads/docs/hadoop-wp- lr-1096.pdf. Accessed: 02/07/2013.
24. A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, and M. Stonebraker. "A comparison of approaches to large-scale data analysis,". SIGMOD'09: Proceedings of the 35th SIGMOD international conference on Management of data. ACM, 2009, pp. 165-178.
25. R. Power and J. Li. "Piccolo: Building Fast, Distributed Programs with Partitioned Tables,". USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2010.
26. C. Ranger, R. Raghuraman, A. Penmetsa, G. R. Bradski, and C. Kozyrakis. "Evaluating MapRe-duce for Multi-core and Multiprocessor Systems,". HPCA. 2007.
27. V. J. Reddi, B. C. Lee, T. M. Chilimbi, and K. Vaid. "Web search using mobile cores: Quantifying and mitigating the price of efficiency,". Proc. 37th International Symposium on Computer Architecture (37th ISCA'10). 2010, pp. 314-325.
28. A. Rowstron, D. Narayanan, A. Donnelly, G. O'Shea, and A. Douglas. "Nobody ever got fired for using Hadoop,". Workshop on Hot Topics in Cloud Data Processing (HotCDP). 2012.
29. M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, and J. Wilkes. "Omega: flexible, scalable schedulers for large compute clusters,". Proceedings of the 8th ACM European Conference on Computer Systems. EuroSys'13. ACM, 2013, pp. 351-364.
30. J. Talbot, R. M. Yoo, and C. Kozyrakis. "Phoenix++: Modular MapReduce for Shared-Memory Systems,". Second International Workshop on MapReduce and its Applications (MAPREDUCE). 2011.
31. Windows Azure Storage. http://www.microsoft.com/windowsazure/features/ storage/. Accessed: 08/09/2011.
32. R. M. Yoo, A. Romano, and C. Kozyrakis. "Phoenix Rebirth: Scalable MapReduce on a Large-Scale Shared-Memory System,". IEEE International Symposium on Workload Characterization (IISWC). 2009.
33. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing,". USENIX Symposium on Networked Systems Design and Implementation (NSDI). 2012.