PerfXplain: Debugging MapReduce job performance

Proceedings of the VLDB Endowment

Volumn 5, Issue 7, 2012, Pages 598-609

  Khoussainova, Nodira ^a   Balazinska, Magdalena ^a   Suciu, Dan ^a  

^a University of Washington   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA MINING; DECISION TREES; QUERY LANGUAGES; TREES (MATHEMATICS);

GENERATION ALGORITHM; GENERATION METHOD; JOB EXECUTION; JOB PERFORMANCE; LARGE-SCALE DATA ANALYSIS; PARALLEL COMPUTATION; PERFORMANCE CHARACTERISTICS; RELATIVE PERFORMANCE;

QUALITY CONTROL;

EID: 84863739408     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: 10.14778/2180912.2180913     Document Type: Article

References (52)

1. Amazon EC2. http://aws.amazon.com/ec2/.
2. Ganglia Monitoring System. http://www.ganglia.sourceforge.net.
3. MySQL Query Analyzer. http://www.mysql.com/products/enterprise/query.html.
4. PostgreSQL Tuning Wizard. http://pgfoundry.org/projects/pgtune.
5. Tuning Your PostgreSQL Server. http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server.
6. S. Agrawal, S. Chaudhuri, L. Kollar, A. Marathe, V. Narasayya, and M. Syamala. Database Tuning Advisor for Microsoft SQL Server demo. In Proc. of the SIGMOD Conf., pages 930-932, 2005.
7. S. Agrawal, S. Chaudhuri, L. Kollár, A. P. Marathe, V. R. Narasayya, and M. Syamala. Database Tuning Advisor for Microsoft SQL Server 2005. In Proc. of the 30th VLDB Conf., pages 1110-1121, 2004.
8. S. Agrawal, S. Chaudhuri, and V. R. Narasayya. Automated Selection of Materialized Views and Indexes in SQL Databases. In Proc. of the 26th VLDB Conf., pages 496-505, 2000.
9. ASTERIX: A Highly Scalable Parallel Platform for Semi-structured Data Management and Analysis. http://asterix.ics.uci.edu/.
10. S. Babu. Towards Automatic Optimization of MapReduce Programs. In Proc. of the 1st ACM symposium on Cloud computing (SOCC), pages 137-142, 2010.
11. D. Becker and P. A. Barsch. Strike it Rich: Application Tuning Helps Companies Save Money through Query Optimization. Teradata magazine online. http://www.teradata.com/tdmo/v07n04/FactsAndFun/Services/StrikeItRich.aspx.
12. N. Borisov, S. Uttamchandani, R. Routray, and A. Singh. Why Did My Query Slow Down? In Proc. of the Fourth CIDR Conf., 2009.
13. R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. In Proc. of the 34th VLDB Conf., pages 1265-1276, 2008.
14. C. Chambers, A. Raniwala, F. Perry, S. Adams, R. R. Henry, R. Bradshaw, and N. Weizenbaum. FlumeJava: Easy, Efficient Data-parallel Pipelines. In PLDI '10: Proceedings of the 2010 ACM SIGPLAN conference on Programming language design and implementation, pages 363-375, 2010.
15. S. Chaudhuri and V. Narasayya. AutoAdmin What-if Index Analysis Utility. SIGMOD Rec., 27:367-378, June 1998.
16. S. Chaudhuri and V. R. Narasayya. An Efficient Cost-Driven Index Selection Tool for Microsoft SQL Server. In Proc. of the 23rd VLDB Conf., pages 146-155, 1997.
17. S. Chaudhuri and V. R. Narasayya. Self-Tuning Database Systems: A Decade of Progress. In Proc. of the 33rd VLDB Conf., pages 3-14, 2007.
18. B. Dageville, D. Das, K. Dias, K. Yagoub, M. Zäit, and M. Ziauddin. Automatic SQL Tuning in Oracle 10g. In Proc. of the 30th VLDB Conf., pages 1098-1109, 2004.
19. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Proc. of the 6th OSDI Symp., pages 137-149, 2004.
20. J. Dittrich, J.-A. Quiané-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad. Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing). Proc. VLDB Endow., 3:515-529, September 2010.
21. S. Duan, S. Babu, and K. Munagala. Fa: A System for Automating Failure Diagnosis. Proc. of the 25th ICDE Conf., pages 1012-1023, 2009.
22. S. Duan, V. Thummala, and S. Babu. Tuning Database Configuration Parameters with iTuned. PVLDB, 2(1):1246-1257, 2009.
23. My Excite. http://www.excite.com/.
24. A. Ganapathi, Y. Chen, A. Fox, R. H. Katz, and D. A. Patterson. Statistics-driven Workload Modeling for the Cloud. In SMDB, pages 87-92, 2010.
25. A. Ganapathi, K. Datta, A. Fox, and D. Patterson. A Case for Machine Learning to Optimize Multicore Performance. In HotPar, 2009.
26. A. Ganapathi, H. Kuno, U. Dayal, J. Wiener, A. Fox, M. Jordan, and D. Patterson. Predicting Multiple Performance Metrics for Queries: Better Decisions Enabled by Machine Learning. In Proc. of the 25th ICDE Conf., pages 592-603, 2009.
27. Greenplum Database. http://www.greenplum.com/.
28. Hadoop. http://hadoop.apache.org/.
29. H. Herodotou and S. Babu. Xplus: a SQL-Tuning-Aware Query Optimizer. Proc. VLDB Endow., 3:1149-1160, 2010.
30. H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu. Starfish: A Self-tuning System for Big Data Analytics. In Proc. of the Fifth CIDR Conf., 2011.
31. Hive. http://hadoop.apache.org/hive/.
32. IBM. DB2 Performance Tuning using the DB2 Configuration Advisor. http://www.ibm.com/developerworks/data/library/techarticle/dm-0605shastry/index.html.
33. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks. In Proc. of the European Conference on Computer Systems (EuroSys), pages 59-72, 2007.
34. E. Jahani, M. Cafarella, and C. Re. Automatic Optimization for MapReduce Programs. PVLDB, 4(6):385-396, 2011.
35. D. Jiang, B. C. Ooi, L. Shi, and S. Wu. The Performance of MapReduce: An In-depth Study. PVLDB, 3:472-483, 2010.
36. MATLAB. version 7.10.0 (R2010a). The MathWorks Inc., 2010.
37. K. Morton, M. Balazinska, and D. Grossman. ParaTimer: A Progress Indicator for MapReduce DAGs. In Proc. of the SIGMOD Conf., pages 507-518, 2010.
38. K. Morton, A. Friesen, M. Balazinska, and D. Grossman. Estimating the Progress of MapReduce Pipelines. In Proc. of the 26th ICDE Conf., pages 681-684, 2010.
39. Netezza, inc. http://www.netezza.com/.
40. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: a Not-So-Foreign Language for Data Processing. In Proc. of the SIGMOD Conf., pages 1099-1110, 2008.
41. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the Data: Parallel Analysis with Sawzall. Scientific Programming, 13(4), 2005.
42. J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993.
43. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2011.
44. M. Robnik-Sikonja and I. Kononenko. An Adaptation of Relief for Attribute Estimation in Regression. In D. H. Fisher, editor, Fourteenth International Conference on Machine Learning, pages 296-304. Morgan Kaufmann, 1997.
45. J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz. Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance. PVLDB, 3:460-471, 2010.
46. StratoSphere: Above the Clouds. http://www.stratosphere.eu/.
47. Teradata, Inc. http://www.teradata.com/.
48. V. Thummala and S. Babu. iTuned: a Tool for Configuring and Visualizing Database Parameters. In Proc. of the SIGMOD Conf., pages 1231-1234, 2010.
49. T. White. Hadoop: The Definitive Guide. MapReduce for the Cloud. O'Reilly Media, 2009.
50. I. H. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques. Second edition, 2005.
51. W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan. Online System Problem Detection by Mining Patterns of Console Logs. In ICDM '09: Proceedings of the 2009 Ninth IEEE International Conference on Data Mining, pages 588-597, 2009.
52. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In Proc. of the 8th OSDI Symp., pages 1-14, 2008.