No one (cluster) size fits all: Automatic cluster sizing for data-intensive analytics

Proceedings of the 2nd ACM Symposium on Cloud Computing, SOCC 2011

Volumn , Issue , 2011, Pages

  Herodotou, Herodotos ^a   Dong, Fei ^a   Babu, Shivnath ^a  

^a DUKE UNIVERSITY   (United States)

Author keywords

Cloud computing; Cluster provisioning; MapReduce

Indexed keywords

AUTOMATED TECHNIQUES; BLACK BOXES; CLUSTER PROVISIONING; CLUSTER SIZES; COMPLEX CLUSTERS; COMPREHENSIVE EVALUATION; MAPREDUCE; WHITE-BOX MODELS;

CLOUD COMPUTING; COMPUTER SIMULATION; DATA HANDLING;

CLUSTER COMPUTING;

EID: 82155186182     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2038916.2038934     Document Type: Conference Paper

References (29)

1. Amazon Elastic MapReduce. http://aws.amazon.com/elasticmapreduce.
2. S. Babu. Towards Automatic Optimization of MapReduce Programs. In SOCC, pages 137-142, 2010.
3. P. Bodik, R. Griffith, C. Sutton, A. Fox, M. Jordan, and D. Patterson. Statistical Machine Learning Makes Automatic Control Practical for Internet Datacenters. In HotCloud, 2009.
4. Facebook on Hadoop, Hive, HBase, and A/B Testing. http://tinyurl.com/ 3dsdsh4.
5. BTrace: A Dynamic Instrumentation Tool for Java. http://kenai.com/ projects/btrace.
6. B. M. Cantrill, M. W. Shapiro, and A. H. Leventhal. Dynamic Instrumentation of Production Systems. In USENIX ATEC, 2004.
7. S. Chaudhuri, P. Ganesan, and V. R. Narasayya. Primitives for Workload Summarization and Implications for SQL. In VLDB, pages 730-741, 2003.
8. N. Chohan, C. Castillo, M. Spreitzer, M. Steinder, A. Tantawi, and C. Krintz. See Spot Run: Using Spot Instances for MapReduce Workflows. In HotCloud, 2010.
9. Cloudera: 7 tips for Improving MapReduce Performance. http://www.cloudera.com/blog/2009/12/ 7-tips-for-improving-mapreduce- performance/.
10. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM, 51(1):107-113, 2008.
11. S. Duan, V. Thummala, and S. Babu. Tuning Database Configuration Parameters with iTuned. PVLDB, 2(1):1246-1257, 2009.
12. J. Hamilton. Resource Consumption Shaping. http://tinyurl.com/4m9vch.
13. H. Herodotou. Hadoop Performance Models. Technical report, Duke Univ., 2010. http://www.cs.duke.edu/starfish/files/hadoop-models.pdf.
14. H. Herodotou and S. Babu. Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs. PVLDB, 4, 2011.
15. H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu. Starfish: A Self-tuning System for Big Data Analytics. In CIDR, pages 261-272, 2011.
16. M.-Y. Iu and W. Zwaenepoel. HadoopToSQL: A MapReduce Query Optimizer. In EuroSys, pages 251-264, 2010.
17. E. Jahani, M. J. Cafarella, and C. Ré. Automatic Optimization of MapReduce Programs. PVLDB, 4:386-396, 2011.
18. K. Kambatla, A. Pathak, and H. Pucha. Towards Optimizing Hadoop Provisioning in the Cloud. In HotCloud, 2009.
19. A. Li, X. Yang, S. Kandula, and M. Zhang. CloudCmp: Shopping for a Cloud Made Easy. In HotCloud, 2010.
20. J. Lin and C. Dyer. Data-Intensive Text Processing with MapReduce. Morgan and Claypool, 2010.
21. M. Mesnier, M. Wachs, R. Sambasivan, A. Zheng, and G. Ganger. Modeling the Relative Fitness of Storage. SIGMETRICS, 35(1):37-48, 2007. (Pubitemid 350158071)
22. Mumak: Map-Reduce Simulator. https://issues.apache.org/jira/browse/ MAPREDUCE-728.
23. OpenCore Probes vs Sun BTrace. http://opencore.jinspired.com/?pagefiid= 588.
24. R. J. Quinlan. Learning with continuous classes. In 5th Australian Joint Conference on Artificial Intelligence, pages 343-348, 1992.
25. A. Qureshi, R. Weber, H. Balakrishnan, J. V. Guttag, and B. Maggs. Cutting the Electric Bill for Internet-scale Systems. In SIGCOMM, pages 123-134, 2009.
26. G. Wang, A. R. Butt, P. Pandey, and K. Gupta. A Simulation Approach to Evaluating Design Decisions in MapReduce Setups. In MASCOTS, pages 1-11, 2009.
27. T. White. Hadoop: The Definitive Guide. Yahoo Press, 2010.
28. T. Ye and S. Kalyanaraman. A Recursive Random Search Algorithm for Large-scale Network Parameter Configuration. In SIGMETRICS, pages 196-205, 2003.
29. W. Zheng, R. Bianchini, J. Janakiraman, J. R. Santos, and Y. Turner. JustRunIt: Experiment-Based Management of Virtualized Data Centers. In USENIX ATC, 2009.