Automatic optimization for MapReduce programs

Proceedings of the VLDB Endowment

Volumn 4, Issue 6, 2011, Pages 385-396

  Jahani, Eaman ^a   Cafarella, Michael J ^a   Ré, Christopher ^b  

^a UNIVERSITY OF MICHIGAN   (United States)

^b UNIVERSITY OF WISCONSIN MADISON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

QUERY PROCESSING; SEARCH ENGINES; SEMANTICS; TREES (MATHEMATICS);

AUTOMATIC OPTIMIZATION; DATA OPERATIONS; DATA SEMANTICS; DATABASE QUERIES; DISTRIBUTED PROGRAMMING; EXISTING SYSTEMS; PROGRAM LOGIC; RELATIONAL DATABASE;

COMPUTER SOFTWARE;

EID: 84863535860     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: 10.14778/1978665.1978670     Document Type: Article

References (30)

1. D. J. Abadi, S. Madden, and M. Ferreira. Integrating compression and execution in column-oriented database systems. In SIGMOD Conference, pages 671-682, 2006.
2. A. Abouzeid, K. Bajda-Pawlikowski, D. J. Abadi, A. Rasin, and A. Silberschatz. HadoopDB: An architectural hybrid of mapreduce and dbms technologies for analytical workloads. PVLDB, 2(1):922-933, 2009.
3. F. Afrati and J. Ullman. Optimizing joins in a map-reduce environment. In EDBT, 2010.
4. A. V. Aho, M. Lam, R. Sethi, and J. D. Ullman. Compilers: Principles, Techniques, and Tools: Second Edition. Addison-Wesley, 2007.
5. E. Anderson and J. Tucek. Efficiency matters. In SIGOPS Workshop, pages 40-45, 2010.
6. Apache Avro. http://avro.apache.org/, 2011.
7. Asm. http://asm.ow2.org/, 2010.
8. P. A. Boncz, S. Manegold, and M. L. Kersten. Database architecture optimized for the new bottleneck: Memory access. In VLDB, pages 54-65, 1999.
9. Y. Bu, B. Howe, M. Balazinska, and M. Ernst. HaLoop: Efficient iterative data processing on large clusters. PVLDB, 3(1):285-296, 2010.
10. M. J. Cafarella and C. Ré. Manimal: Relational optimization for data-intensive programs. In WebDB, 2010.
11. T. M. Chilimbi. Efficient representations and abstractions for quantifying and exploiting data reference locality. In PLDI, pages 191-202, 2001.
12. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In OSDI, pages 137-150, 2004.
13. J. Dittrich, J.-A. Quiané-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad. Hadoop++: Making a yellow elephant run like a cheetah (without it even noticing). PVLDB, 3(1):518-529, 2010.
14. C. Douglas and H. Tang. Gridmix3: Emulating production io workload for apache hadoop. In USENIX FAST '10: Work-in-Progress Reports, 2010.
15. A. Floratou, J. M. Patel, E. J. Shekita, and S. Tata. Speeding up hadoop using column-store techniques. To appear, PVLDB.
16. A. Guttman. R-trees: A dynamic index structure for spatial searching. In SIGMOD Conference, pages 47-57, 1984.
17. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: fair scheduling for distributed computing clusters. In SOSP, pages 261-276, 2009.
18. Karmasphere corporation: A study of hadoop developers. http://karmasphere.com., 2010.
19. K. Kim, K. Jeon, H. Han, S. G. Kim, H. Jung, and H. Y. Yeom. Mrbench: A benchmark for mapreduce framework. In ICPADS, pages 11-18, 2008.
20. E. Meijer, B. Beckman, and G. M. Bierman. LINQ: reconciling object, relations and xml in the .net framework. In SIGMOD Conference, page 706, 2006.
21. A. C. Murthy. Apache hadoop: Best practices and anti-patterns. http://developer.yahoo.com/blogs/hadoop/posts/2010/08/apache_hadoop_best_practices_a/, 2010.
22. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: a not-so-foreign language for data processing. In SIGMOD Conference, pages 1099-1110, 2008.
23. A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, and M. Stonebraker. A comparison of approaches to large-scale data analysis. In SIGMOD Conference, pages 165-178, 2009.
24. Personal communication, Amr Awadallah, Cloudera corp., 2010.
25. C. Re, J. Siméon, and M. F. Fernández. A complete and efficient algebraic compiler for xquery. In ICDE, page 14, 2006.
26. M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. Madden, E. J. O'Neil, P. E. O'Neil, A. Rasin, N. Tran, and S. B. Zdonik. C-store: A column-oriented dbms. In VLDB, pages 553-564, 2005.
27. C. Yang, C. Yen, C. Tan, and S. Madden. Osprey: Implementing mapreduce-style fault tolerance in a shared-nothing distributed database. In ICDE, pages 657-668, 2010.
28. H.-C. Yang, A. Dasdan, R.-L. Hsiao, and D.S. Parker Jr. Map-reduce-merge: simplified relational data processing on large clusters. In SIGMOD Conference, pages 1029-1040, 2007.
29. M. Zaharia, A. Konwinski, A. D. Joseph, R. H. Katz, and I. Stoica. Improving mapreduce performance in heterogeneous environments. In OSDI, pages 29-42, 2008.
30. J. Zawodny. Yahoo! launches world's largest hadoop production application. http://developer.yahoo.com/blogs/hadoop/posts/2008/02/yahoo-worlds-largest-production-hadoop/, 2008