Optimizing multiway joins in a map-reduce environment

IEEE Transactions on Knowledge and Data Engineering

Volumn 23, Issue 9, 2011, Pages 1282-1298

  Afrati, Foto N ^a   Ullman, Jeffrey D ^b  

^a NATIONAL TECHNICAL UNIVERSITY OF ATHENS   (Greece)

^b Stanford University   (United States)

Author keywords

joins; Map reduce; parallel computing; query optimization

Indexed keywords

DIMENSION TABLES; FIXED NUMBERS; JOINS; MAP PROCESS; MAP-REDUCE; MULTI-WAY JOIN; QUERY OPTIMIZATION; SOCIAL NETWORKS; STAR JOIN; VERY LARGE DATUM;

PARALLEL ARCHITECTURES; USER INTERFACES;

OPTIMIZATION;

EID: 79960930577     PISSN: 10414347     EISSN: None     Source Type: Journal    
DOI: 10.1109/TKDE.2011.47     Document Type: Article

References (33)

1. F.N. Afrati and J.D. Ullman, "Optimizing Joins in a Map-Reduce Environment," Proc. 13th Int'l Conf. Extending Database Technology (EDBT '10), 2010.
2. Apache, "Hadoop," http://hadoop.apache.org/, 2006.
3. Apache, "Hive," http://wiki.apache.org/hadoop/Hive, 2008.
4. R. Avnur and J.M. Hellerstein, "Eddies: Continuously Adaptive Query Processing," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '00), pp. 261-272, 2000.
5. S. Babu, K. Munagala, J. Widom, and R. Motwani, "Adaptive Caching for Continuous Queries," Proc. 21st Int'l Conf. Data Eng. (ICDE '05), pp. 118-129, 2005. (Pubitemid 41731120)
6. S. Babu and J. Widom, "Streamon: An Adaptive Engine for Stream Query Processing," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '04), pp. 931-932, 2004.
7. D. Battré, S. Ewen, F. Hueske, O. Kao, V. Markl, and D. Warneke, "Nephele/Pacts: A Programming Model and Execution Framework for Web-Scale Analytical Processing," Proc. First ACM Symp. Cloud Computing (SoCC '10), pp. 119-130, 2010.
8. V. Borkar, M. Carey, R. Grover, N. Onose, and R. Vernica, "Hyracks: A Flexible and Extensible Foundation for Data-Intensive Computing," Proc. IEEE 27th Int'l Conf. Data Eng. (ICDE), 2011.
9. S. Brin and L. Page, "The Anatomy of a Large-Scale Hypertextual Web Search Engine," Computer Networks and ISDN Systems, vol. 30, nos. 1-7, pp. 107-117, 1998.
10. Y. Bu, B. Howe, M. Balazinska, and M. Ernst, "Haloop: Efficient Iterative Data Processing on Large Clusters," Proc. VLDB Endowment, vol. 3, no. 1/2, pp. 285-296, 2010.
11. R. Chaiken, B. Jenkins, P. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou, "Scope: Easy and Efficient Parallel Processing of Massive Data Sets," Proc. VLDB Endowment, vol. 1, no. 2, pp. 1265-1276, 2008.
12. F. Chang, J. Dean, S. Ghemawat, W.C. Hsieh, D.A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R.E. Gruber, "Bigtable: A Distributed Storage System for Structured Data," ACM Trans. Computer Systems, vol. 26, no. 2, pp. 1-26, 2008.
13. B.F. Cooper, R. Ramakrishnan, U. Srivastava, A. Silberstein, P. Bohannon, H.-A. Jacobsen, N. Puz, D. Weaver, and R. Yerneni, "Pnuts: Yahoo!'s Hosted Data Serving Platform," Proc. VLDB Endowment, vol. 1, no. 2, pp. 1277-1288, 2008.
14. J. Dean and S. Ghemawat, "Mapreduce: Simplified Data Processing on Large Clusters," Comm. ACM, vol. 51, no. 1, pp. 107-113, 2008.
15. A. Deshpande and L. Hellerstein, "Flow Algorithms for Parallel Query Optimization," Proc. IEEE 24th Int'l Conf. Data Eng. (ICDE), pp. 754-763, 2008.
16. D.J. DeWitt, J.F. Naughton, D.A. Schneider, and S. Seshadri, "Practical Skew Handling in Parallel Joins," Proc. 18th Int'l Conf. Very Large Data Bases (VLDB '92), pp. 27-40, 1992.
17. D.J. DeWitt, E. Paulson, E. Robinson, J.F. Naughton, J. Royalty, S. Shankar, and A. Krioukov, "Clustera: An Integrated Computation and Data Management System," Proc. VLDB Endowment, vol. 1, no. 1, pp. 28-41, 2008.
18. S. Ganguly, A. Silberschatz, and S. Tsur, "A Framework for the Parallel Processing of Datalog Queries," ACM SIGMOD Record, vol. 19, pp. 143-152, June 1990.
19. H. Garcia-Molina, J.D. Ullman, and J. Widom, Database Systems: The Complete Book. Pearson Prentice Hall, 2008.
20. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The Google File System," Proc. 19th ACM Symp. Operating Systems Principles, 2003.
21. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks," ACM SIGOPS Operating Systems Rev., vol. 41, no. 3, pp. 59-72, 2007. (Pubitemid 47281575)
22. Z.G. Ives, D. Florescu, M. Friedman, A.Y. Levy, and D.S. Weld, "An Adaptive Query Execution System for Data Integration," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '99), pp. 299-310, 1999. (Pubitemid 129597344)
23. H. Jakobsson, "Mixed-Approach Algorithms for Transitive Closure," Proc. 10th ACM SIGACT-SIGMOD-SIGART Symp. Principles of Database Systems (PODS '91), pp. 199-205, 1991.
24. H. Jakobsson, "On Tree-Based Techniques for Query Evaluation," Proc. 11th ACM SIGACT-SIGMOD-SIGART Symp. Principles of Database Systems (PODS '92), pp. 380-392, 1992. (Pubitemid 23608183)
25. H. Jakobsson, "Tree-Based Techniques for Query Evaluation," PhD thesis, Dept. of CS, Stanford Univ., STAN-CS-93-1492, 1993.
26. J.M. Kleinberg, "Authoritative Sources in a Hyperlinked Environment," J. ACM, vol. 46, pp. 668-677, 1999.
27. S. Madden, M.A. Shah, J.M. Hellerstein, and V. Raman, "Continuously Adaptive Continuous Queries over Streams," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '02), pp. 49-60, 2002. (Pubitemid 34985535)
28. G. Malewicz, M. Austern, A. Bik, J. Dehnert, I. horn, N. Leiser, and G. Czajkowski, "Pregel: A System for Large-Scale Graph Processing," Proc. Int'l Conf. Management of Data (SIGMOD '10), 2010.
29. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig Latin: A Not-so-Foreign Language for Data Processing," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '08), pp. 1099-1110, 2008.
30. K.A. Ross and J. Cieslewicz, "Optimal Splitters for Database Partitioning with Size Bounds," Proc. 12th Int'l Conf. Database Theory (ICDT '09), pp. 98-110, 2009.
31. U. Srivastava, K. Munagala, J. Widom, and R. Motwani, "Query Optimization over Web Services," Proc. 32nd Int'l Conf. Very Large Data Bases (VLDB '06), pp. 355-366, 2006.
32. K.-L. Tan and H. Lu, "A Note on the Strategy Space of Multiway Join Query Optimization Problem in Parallel Systems," ACM SIGMOD Record, vol. 20, no. 4, pp. 81-82, 1991.
33. S.D. Viglas, J.F. Naughton, and J. Burger, "Maximizing the Output Rate of Multi-Way Join Queries over Streaming Information Sources," Proc. 29th Int'l Conf. Very Large Data Bases (VLDB '03), pp. 285-296, 2003.