Ernest: Efficient performance prediction for large-scale advanced analytics

Proceedings of the 13th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2016

Volumn , Issue , 2016, Pages 363-378

  Venkataraman, Shivaram ^a   Yang, Zongheng ^a   Franklin, Michael ^a   Recht, Benjamin ^a   Stoica, Ion ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADVANCED ANALYTICS; SYSTEMS ANALYSIS;

CLOUD COMPUTING INFRASTRUCTURES; HARDWARE CONFIGURATIONS; LARGE-SCALE ANALYTICS; OPTIMAL EXPERIMENT DESIGN; PERFORMANCE PREDICTION; RESOURCE CONFIGURATIONS; SHARED INFRASTRUCTURE; STATISTICAL TECHNIQUES;

FORECASTING;

EID: 85077040628     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (83)

1. 1000 Genomes Project and AWS. http://aws.amazon.com/1000genomes/.
2. Apache Mahout. http://mahout.apache.org/.
3. Apache Spark: Monitoring and Instrumentation. http://spark.apache.org/docs/latest/monitoring.html.
4. Apache spark, preparing for the next wave of reactive big data. http://goo.gl/FqEh94.
5. Apache Spark: Submitting Applications. http://spark.apache.org/docs/latest/submitting-applications.html.
6. Big data platform, hp haven. http://www8.hp.com/us/en/software-solutions/ big-data-platform-haven/.
7. Common crawl. http://commoncrawl.org.
8. Hadoop History Server REST APIs. http://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-yarn/hadoop-yarn-site/ HistoryServerRest.html.
9. Machine learning, microsoft azure. http://azure.microsoft.com/en-us/services/ machine-learning/.
10. MapReduce Tutorial. hadoop.apache.org/ docs/current/hadoop-mapreduce-client/ hadoop-mapreduce-client-core/ MapReduceTutorial.html.
11. V. S. Adve, J. Mellor-Crummey, M. Anderson, K. Kennedy, J.-C. Wang, and D. A. Reed. An integrated compilation and performance analysis environment for data parallel programs. In Supercomputing, 1995.
12. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, and I. Stoica. Pacman: Coordinated memory caching for parallel jobs. In USENIX NSDI, 2012.
13. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the outliers in Map-Reduce clusters using Mantri. In USENIX OSDI, 2010.
14. J. M. Anderson, L. M. Berc, J. Dean, S. Ghemawat, M. R. Henzinger, S.-T. A. Leung, R. L. Sites, M. T. Vandevoorde, C. A. Waldspurger, and W. E. Weihl. Continuous profiling: Where have all the cycles gone? ACM Transactions on Computer Systems, 15(4):357–390.
15. Apache Hadoop NextGen MapReduce (YARN). Retrieved 9/24/2013, URL: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html.
16. V. Balasundaram, G. Fox, K. Kennedy, and U. Kremer. A static performance estimator to guide data partitioning decisions. In Symposium on Principles and Practice of Parallel Programming, PPOPP’91, pages 213–223, Williamsburg, Virginia, USA.
17. G. B. Berriman, G. Juve, E. Deelman, M. Regelson, and P. Plavchan. The application of cloud computing to astronomy: A study of cost and performance. In Sixth IEEE International Conference on e-Science, 2010.
18. J. Bilmes, K. Asanovic, C.-W. Chin, and J. Demmel. Optimizing matrix multiply using PHiPAC: a portable, high-performance, ANSI C coding methodology. In Supercomputing 1997, pages 340–347.
19. I. Bird. Computing for the large hadron collider. Annual Review of Nuclear and Particle Science, 61:99–118, 2011.
20. Blaise Barney. Message Passing Interface. https://computing.llnl.gov/tutorials/mpi/.
21. G. E. Blelloch. Programming parallel algorithms. Communications of the ACM, 39(3):85–97.
22. L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems, volume 20, pages 161–168. 2008.
23. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1):1–122, 2011.
24. T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 858–867, Prague, Czech Republic, June 2007.
25. S. Browne, J. Dongarra, N. Garner, K. London, and P. Mucci. A scalable cross-platform infrastructure for application performance tuning using hardware counters. In Supercomputing 2000, Dallas, Texas, USA.
26. J. P. Campbell Jr and D. A. Reynolds. Corpora for the evaluation of speaker recognition systems. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 2, pages 829–832, 1999.
27. B. L. Chamberlain, C. Lin, S.-E. Choi, L. Snyder, E. C. Lewis, and W. D. Weathersby. Zpl’s wysiwyg performance model. In Third International Workshop on High-Level Parallel Programming Models and Supportive Environments, pages 50–61.
28. C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27, 2011.
29. S. Chaudhuri, V. Narasayya, and R. Ramamurthy. Estimating progress of execution for SQL queries. In SIGMOD 2004, pages 803–814.
30. M. Chowdhury and I. Stoica. Coflow: a networking abstraction for cluster applications. In Proceedings of the 11th ACM Workshop on Hot Topics in Networks, pages 31–36, 2012.
31. R. Clint Whaley, A. Petitet, and J. J. Dongarra. Automated empirical optimizations of software and the ATLAS project. Parallel Computing, 27(1):3–35, 2001.
32. A. Coates and A. Y. Ng. Learning feature representations with k-means. In Neural Networks: Tricks of the Trade, pages 561–580. Springer, 2012.
33. J. Dean and L. A. Barroso. The tail at scale. Communications of the ACM, 56(2):74–80, 2013.
34. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, A. Senior, P. Tucker, K. Yang, et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems 25, pages 1232–1240, 2012.
35. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Communications of the ACM, 51(1), 2008.
36. C. Delimitrou and C. Kozyrakis. Quasar: Resource-efficient and qos-aware cluster management. In ASPLOS 2014, pages 127–144.
37. J. Demmel, L. Grigori, M. Hoemmen, and J. Langou. Communication-optimal parallel and sequential QR and LU factorizations. SIAM Journal on Scientific Computing, 34(1):A206–A239, 2012.
38. B. Efron. The jackknife, the bootstrap and other resampling plans, volume 38. SIAM, 1982.
39. A. D. Ferguson, P. Bodik, S. Kandula, E. Boutin, and R. Fonseca. Jockey: guaranteed job latency in data parallel clusters. In Eurosys 2012, pages 99–112.
40. GenBase repository. https://github.com/mitdbg/genbase.
41. M. Grant and S. Boyd. Graph implementations for nonsmooth convex programs. In V. Blondel, S. Boyd, and H. Kimura, editors, Recent Advances in Learning and Control, Lecture Notes in Control and Information Sciences, pages 95–110. Springer-Verlag Limited, 2008. http://stanford.edu/~boyd/graph_dcp.html.
42. M. Grant and S. Boyd. CVX: Matlab software for disciplined convex programming, version 2.1. http://cvxr.com/cvx, Mar. 2014.
43. N. Halko, P.-G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217–288, 2011.
44. B. He, M. Yang, Z. Guo, R. Chen, W. Lin, B. Su, H. Wang, and L. Zhou. Wave computing in the cloud. In HotOS, 2009.
45. J. He, D. Nguyen, A. Arpaci-Dusseau, and R. Arpaci-Dusseau. Reducing file system tail latencies with chopper. In FAST 2015, pages 119–133, Santa Clara, CA.
46. H. Herodotou and S. Babu. Profiling, what-if analysis, and cost-based optimization of MapReduce programs. VLDB 2011, 4(11):1111–1122.
47. H. Herodotou, F. Dong, and S. Babu. No one (cluster) size fits all: automatic cluster sizing for data-intensive analytics. In SOCC 2011.
48. H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu. Starfish: A self-tuning system for big data analytics. In CIDR, volume 11, pages 261–272, 2011.
49. B. Huang, M. Boehm, Y. Tian, B. Reinwald, S. Tatikonda, and F. R. Reiss. Resource elasticity for large-scale machine learning. In SIGMOD 2015.
50. P.-S. Huang, H. Avron, T. N. Sainath, V. Sindhwani, and B. Ramabhadran. Kernel methods match deep neural networks on timit. In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, page 6, 2014.
51. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In Eurosys 2007.
52. V. Jalaparti, H. Ballani, P. Costa, T. Karagiannis, and A. Rowstron. Bridging the tenant-provider gap in cloud services. In SOCC 2012.
53. E. Jones, T. Oliphant, P. Peterson, et al. SciPy: Open source scientific tools for Python. http://www.scipy.org/, 2001–.
54. J. D. McCalpin. Memory bandwidth and machine balance in current high performance computers. IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter, pages 19–25, Dec. 1995.
55. X. Meng, J. Bradley, E. Sparks, and S. Venkataraman. ML Pipelines: A New High-Level API for MLlib. https://goo.gl/pluhq0, 2015.
56. Apache Spark MLLib. https://spark.apache.org/mllib/.
57. K. Morton, M. Balazinska, and D. Grossman. Para-timer: A progress indicator for mapreduce dags. In SIGMOD 2010, pages 507–518, Indianapolis, Indiana, USA.
58. D. G. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi. Naiad: a timely dataflow system. In SOSP 2013, pages 439–455.
59. D. Narayanan. Operating System Support for Mobile Interactive Applications. PhD thesis, CMU, 2002.
60. O. Niehorster, A. Krieger, J. Simon, and A. Brinkmann. Autonomic resource management with support vector machines. In International Conference on Grid Computing (GRID’11), pages 157–164, 2011.
61. F. A. Nothaft, M. Massie, T. Danford, Z. Zhang, U. Laserson, C. Yeksigian, J. Kottalam, A. Ahuja, J. Hammerbacher, M. Linderman, M. J. Franklin, A. D. Joseph, and D. A. Patterson. Rethinking data-intensive science using scalable analytics systems. In SIGMOD 2015, pages 631–646.
62. L. Ortiz, V. de Almeida, and M. Balazinska. Changing the Face of Database Cloud Services with Personalized Service Level Agreements. In Proceedings of the International Conference on Innovative Data Systems Research (CIDR), 2015.
63. F. Pukelsheim. Optimal design of experiments, volume 50. SIAM, 1993.
64. A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in neural information processing systems, pages 1177–1184, 2007.
65. R. Raman, M. Livny, and M. Solomon. Matchmaking: Distributed resource management for high throughput computing. In HPDC 1998.
66. B. Recht, C. Re, S. Wright, and F. Niu. HOGWILD!: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems, pages 693–701, 2011.
67. P. G. Selinger, M. M. Astrahan, D. D. Chamberlin, R. A. Lorie, and T. G. Price. Access path selection in a relational database management system. In Proceedings of the 1979 ACM SIGMOD international conference on Management of data, pages 23–34, 1979.
68. J. Shi, J. Zou, J. Lu, Z. Cao, S. Li, and C. Wang. MRTuner: A Toolkit to Enable Holistic Optimization for MapReduce Jobs. VLDB 2014, 7(13).
69. P. Shivam, V. Marupadi, J. Chase, T. Subramaniam, and S. Babu. Cutting corners: workbench automation for server benchmarking. In USENIX ATC 2008, pages 241–254.
70. S. Sonnenburg and V. Franc. Coffin: A computational framework for linear svms. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 999–1006, 2010.
71. E. Sparks. Announcing KeystoneML. https://amplab.cs.berkeley.edu/ announcing-keystoneml, 2015.
72. L. D. Stein et al. The case for cloud computing in genome informatics. Genome Biology, 11(5):207, 2010.
73. R. Taft, M. Vartak, N. R. Satish, N. Sundaram, S. Madden, and M. Stonebraker. Genbase: A complex analytics genomics benchmark. In SIGMOD 2014, pages 177–188.
74. J. Uszkoreit, J. M. Ponte, A. C. Popat, and M. Dubiner. Large scale parallel document mining for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1101–1109. Association for Computational Linguistics, 2010.
75. R. A. Van De Geijn and J. Watts. SUMMA: Scalable universal matrix multiplication algorithm. Concurrency-Practice and Experience, 9(4):255–274, 1997.
76. S. Venkataraman, A. Panda, G. Ananthanarayanan, M. J. Franklin, and I. Stoica. The power of choice in data-aware cluster scheduling. In OSDI 2014, pages 301–316.
77. A. Verma, L. Cherkasova, and R. H. Campbell. Aria: Automatic resource inference and allocation for mapreduce environments. In ICAC 2011, pages 235–244, Karlsruhe, Germany.
78. A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes. Large-scale cluster management at google with borg. In Eurosys 2015.
79. W. Wang, L. Xu, and I. Gupta. Scale up Vs. Scale out in Cloud Storage and Graph Processing System. In Proceedings of the 2nd IEEE Workshop on Cloud Analytics, 2015.
80. N. J. Yadwadkar, G. Ananthanarayanan, and R. Katz. Wrangler: Predictable and faster jobs using fewer resources. In SOCC 2014.
81. W. Yoo, K. Larson, L. Baugh, S. Kim, and R. H. Campbell. Adp: Automated diagnosis of performance pathologies using hardware events. In International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS 2012, pages 283–294, London, England, UK.
82. H.-F. Yu, H.-Y. Lo, H.-P. Hsieh, J.-K. Lou, T. G. McKenzie, J.-W. Chou, P.-H. Chung, C.-H. Ho, C.F. Chang, Y.-H. Wei, et al. Feature engineering and classifier ensemble for KDD Cup 2010. KDD Cup 2010.
83. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI 2012.