Improving mapreduce performance using smart speculative execution strategy

IEEE Transactions on Computers

Volumn 63, Issue 4, 2014, Pages 954-967

  Chen, Qi ^a   Liu, Cheng ^a   Xiao, Zhen ^a  

^a PEKING UNIVERSITY   (China)

Author keywords

cluster throughput; cost performance; MapReduce; speculative execution; straggler

Indexed keywords

DATA PROCESSING; PARALLEL ARCHITECTURES; THROUGHPUT;

COMPUTING FRAMEWORKS; COST PERFORMANCE; EXPONENTIALLY WEIGHTED MOVING AVERAGE; LARGE-SCALE DATA PROCESSING; MAP-REDUCE; PERFORMANCE METRICS; SPECULATIVE EXECUTION; STRAGGLER;

MULTIPROCESSING SYSTEMS;

EID: 84897130212     PISSN: 00189340     EISSN: None     Source Type: Journal    
DOI: 10.1109/TC.2013.15     Document Type: Article

References (31)

1. J. Dean and S. Ghemawat, "Mapreduce: Simplified Data Processing on Large Clusters, " Comm. ACM, vol. 51, pp. 107-113, Jan. 2008
2. Apache hadoop, http://hadoop.apache.org/2013.
3. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks, " Proc. Second ACM SIGOPS/EuroSys European Conf. Computer Systems (EuroSys '07), 2007.
4. M. Zaharia, A. Konwinski, A.D. Joseph, R. Katz, and I. Stoica, "Improving Mapreduce Performance in Heterogeneous Environments, " Proc. Eighth USENIX Conf. Operating Systems Design and Implementation, (OSDI '08), 2008.
5. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris, "Reining in the Outliers in Map-Reduce Clusters Using Mantri, " Proc. Ninth USENIX Conf. Operating Systems Design and Implementation, (OSDI '10), 2010.
6. "Amazon Elastic Compute Cloud (EC2), " http://aws.amazon. com/ec2/, 2013.
7. Y. Kwon, M. Balazinska, and B. Howe, "A Study of Skew in Map-reduce Applications, " Proc. Fifth Open Cirrus Summit, 2011.
8. P.H. Ellaway, "Cumulative Sum Technique and Its Application to the Analysis of Peristimulus Time Histograms, " Electroencephalog-raphy and Clinical Neurophysiology, vol. 45, no. 2, pp. 302-304, 1978. (Pubitemid 9025359)
9. "Open Stack Cloud Operating System, " http://www.openstack.- org/, 2013.
10. K. Avi, K. Yaniv, L. Dor, L. Uri, and L. Anthony, "Kvm: The Linux Virtual Machine Monitor, " Proc. Linux Symp., 2007.
11. R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou, "Scope: Easy and Efficient Parallel Process ing of Massive Data Sets, " Proc. VLDB Endowment, vol. 1, pp. 1265-1276, Aug. 2008.
12. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig Latin: A Not-So-Foreign Language for Data Processing, " Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '08), 2008.
13. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P.K. Gunda, and J. Currey, "Dryadlinq: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language, " Proc. Eighth USENIX Conf. Operating Systems Design and Implementation, (OSDI '08), 2008.
14. H. Herodotou and S. Babu, "Profiling, Whatif Analysis, and Costbased Optimization of Mapreduce Programs, " Proc. VLDB Endow., vol. 3, pp. 1111-1122, Sept. 2011.
15. S. Babu, "Towards Automatic Optimization of Mapreduce Programs, " Proc. First ACM Symp. Cloud Computing (SoCC '10), 2010.
16. X. Pan, J. Tan, S. Kavulya, R. Gandhi, and P. Narasimhan, "Ganesha: Blackbox Diagnosis of Mapreduce Systems, " ACM SIG-METRICS Performance Evaluation Rev., vol. 37, pp. 8-13, Jan. 2010.
17. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears, "Mapreduce Online, " Proc. Seventh USENIX Conf. Networked Systems Design and Implementation (NSDI '10), 2010.
18. H.-c. Yang, A. Dasdan, R.-L. Hsiao, and D.S. Parker, "Map-Reduce- Merge: Simplified Relational Data Processing on Large Clusters, " Proc. ACM SIGMOD Int'l Conf. Management of Data, (SIGMOD '07), 2007.
19. I. Roy, S.T.V. Setty, A. Kilzer, V. Shmatikov, and E. Witchel, "Airavat: Security and Privacy for Mapreduce, " Proc. Seventh USENIX Conf. Networked Systems Design and Implementation (NSDI '10), 2010.
20. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris, "Scarlett: Coping with Skewed Content Popularity in Mapreduce Clusters, " Proc. Sixth Conf. Computer Systems (EuroSys '11), 2011.
21. B. Nicolae, D. Moise, G. Antoniu, L. Bouge, and M. Dorier, "Blobseer: Bringing High Throughput under Heavy Concurrency to Hadoop Map-Reduce Applications, " Proc. IEEE Int'l Symp. Parallel Distributed Processing (IPDPS), Apr. 2010.
22. J. Leverich and C. Kozyrakis, "On the Energy (In)Efficiency of Hadoop Clusters, " ACM SIGOPS Operating Systems Rev., vol. 44, pp. 61-65, Mar. 2010.
23. J.-A. Quiane-Ruiz, C. Pinkel, J. Schad, and J. Dittrich, "Rafting Mapreduce: Fast Recovery on the Raft, " Proc. IEEE 27th Int'l Conf. Data Eng. (ICDE), Apr. 2011.
24. T. Sandholm and K. Lai, "Mapreduce Optimization Using Regulated Dynamic Prioritization, " Proc. 11th Int'l Joint Conf. Measurement and Modeling of Computer Systems, (SIGMETRICS '09), 2009.
25. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quincy: Fair Scheduling for Distributed Computing Clusters, " Proc. ACM SIGOPS 22nd Symp. Operating Systems Principles (SOSP '09), 2009.
26. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling, " Proc. Fifth European Conf. Computer Systems (EuroSys '10), 2010.
27. "Apache hadoop 2.0, " http://hadoop.apache.org/docs/r2.0.0- alpha/, 2013.
28. A. Faraj, X. Yuan, and D. Lowenthal, "Star-Mpi: Self Tuned Adaptive Routines for Mpi Collective Operations, " Proc. 20th Ann. Int'l Conf. Supercomputing (ICS '06), 2006.
29. J. Bernardin, P. Lee, and J. Lewis, "Using Execution Statistics to Select Tasks for Redundant Assignment in a Distributed Computing Platform, " Patent 7 093 004, Aug. 2006.
30. B. Ucar, C. Aykanat, K. Kaya, and M. Ikinci, "Task Assignment in Heterogeneous Computing Systems, " J. Parallel and Distributed Computing, vol. 66, pp. 32-46, Jan. 2006. (Pubitemid 41827946)
31. S. Manoharan, "Effect of Task Duplication on the Assignment of Dependency Graphs, " Parallel Computing, vol. 27, pp. 257-268, Feb. 2001. (Pubitemid 32264542)