Skew-resistant parallel processing of feature-extracting scientific user-defined functions

Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC '10

Volumn , Issue , 2010, Pages 75-86

  Kwon, YongChul ^a   Balazinska, Magdalena ^a   Howe, Bill ^a   Rolia, Jerome ^b  

^a UNIVERSITY OF WASHINGTON   (United States)

^b HEWLETT PACKARD LABORATORIES   (United States)

Author keywords

Algorithms; Design; Experimentation

Indexed keywords

EXECUTION MODEL; EXECUTION TIME; INPUT DATAS; INPUT SIZE; MULTI-DIMENSIONAL SPACE; MULTIDIMENSIONAL ARRAYS; NEW SYSTEM; OPTIMIZERS; PARALLEL DATA PROCESSING; PARALLEL PROCESSING; PARAMETERIZED; RUNTIMES; USER DEFINED FUNCTIONS;

ALGORITHMS; COST FUNCTIONS; DISTRIBUTED COMPUTER SYSTEMS; EXPERIMENTS; FEATURE EXTRACTION;

DATA PROCESSING;

EID: 77954913432     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1807128.1807140     Document Type: Conference Paper

References (41)

1. Amazon Elastic Compute Cloud (Amazon EC2). http://www.amazon.com/gp/ browse.html?node=201590011.
2. Oceanic remote chemical analyzer (ORCA). http://armbrustlab.ocean. washington.edu/.
3. M. Berger and S. Bokhari. A partitioning strategy for nonuniform problems on multiprocessors. Computers, IEEE Transactions on, C-36(5), 1987.
4. J. Boulos and K. Ono. Cost estimation of user-defined methods in object-relational database systems. SIGMOD Record, 28(3), 1999.
5. M. Davis, G. Efstathiou, C. S. Frenk, and S. D. M. White. The evolution of large-scale structure in a universe dominated by cold dark matter. Astroph. J., 292:371-394, May 1985.
6. J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. In Proc. of the 6th OSDI Symp., 2004.
7. A. Deshpande, Z. Ives, and V. Raman. Adaptive query processing. Foundations and Trends in Databases, 1(1):139, 2007.
8. K. Devine, E. Boman, R. Heapby, B. Hendrickson, and C. Vaughan. Zoltan data management service for parallel dynamic applications. Computing in Science and Engg., 4(2), 2002.
9. K. Devine, E. Boman, and G. Karypis. Partitioning and load balancing for emerging parallel applications and architectures, chapter 6. 2006.
10. D. DeWitt and J. Gray. Parallel database systems: the future of high performance database systems. CACM, 35(6), 1992.
11. D. J. DeWitt, J. F. Naughton, D. A. Schneider, and S. Seshadri. Practical Skew Handling in Parallel Joins. In Proc. of the 18th VLDB Conf., 1992.
12. DeWitt et. al. Clustera: an integrated computation and data management system. In Proc. of the 34th VLDB Conf., 2008.
13. J. M. Gelb and E. Bertschinger. Cold dark matter. 1: The formation of dark halos. Astroph. J., 436:467-490, Dec. 1994. (Pubitemid 24974105)
14. R. L. Graham. Bounds on multiprocessing timing anomalies. SIAM Journal on Applied Mathematics, 17(2):416-429, 1969.
15. Hadoop. http://hadoop.apache.org/.
16. Hive. http://hadoop.apache.org/hive/.
17. B. Howe, D. Maier, and L. Bright. Smoothing the roi curve for scientific data management applications. In Proc. of the Third CIDR Conf., 2007.
18. K. A. Hua and C. Lee. Handling data skew in multiprocessor database computers using partition tuning. In Proc. of the 17th VLDB Conf., 1991.
19. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In Proc. of the EuroSys Conf., 2007.
20. S. R. Knollmann and A. Knebe. AHF: Amiga's Halo Finder. Astroph. J. Suppl., 182:608-624, June 2009.
21. Kwon et. al. Scalable clustering algorithm for N-body simulations in a shared-nothing cluster. Technical Report UW-CSE-09-06-01, Dept. of Comp. Sci., Univ. of Washington, 2009.
22. J. Lin. The Curse of Zipf and Limits to Parallelization: A Look at the Stragglers Problem in MapReduce. In 7th Workshop on Large-Scale Distributed Systems for Information Retrieval, volume i, 2009.
23. R. P. Mount. The office of science data-management challenge. Technical report, Department of Energy, 2004.
24. L. Oliker and R. Biswas. Plum: parallel load balancing for adaptive unstructured meshes. J. Parallel Distrib. Comput., 52(2), 1998.
25. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In Proc. of the SIGMOD Conf., 2008.
26. Oracle. http://www.oracle.com/database/.
27. M. Parashar, H. Liu, Z. Li, V. Matossian, C. Schmidt, G. Zhang, and S. Hariri. AutoMate: Enabling Autonomic Applications on the Grid. Cluster Computing, 9(2), 2006.
28. Pavlo et. al. A comparison of approaches to large-scale data analysis. In Proc. of the SIGMOD Conf., 2009.
29. X. Qiu, J. Ekanayake, S. Beason, T. Gunarathne, G. Fox, R. Barga, and D. Gannon. Cloud technologies for bioinformatics applications. In MTAGS '09: Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers, pages 1-10, 2009.
30. A. Shatdal and J. Naughton. Adaptive parallel aggregation algorithms. In Proc. of the SIGMOD Conf., 1995.
31. Springel et. al. Simulations of the formation, evolution and clustering of galaxies and quasars. NATURE, 435:629-636, June 2005.
32. Sql server. http://www.microsoft.com/sqlserver/.
33. J. G. Stadel. Cosmological N-body simulations and their analysis. PhD thesis, University of Washington, 2001.
34. A. Szalay and J. Gray. 2020 computing: Science in an exponential world. Nature, 440:413-414, 2006.
35. C. B. Walton, A. G. Dale, and R. M. Jenevein. A Taxonomy and Performance Model of Data Skew Effects in Parallel Joins. In Proc. of the 17th VLDB Conf., 1991.
36. R. S. Wei Li, D Gao. Skew handling techniques in sort-merge join. In Proc. of the SIGMOD Conf., 2002.
37. D. H. Weinberg, L. Hernquist, and N. Katz. Photoionization, Numerical Resolution, and Galaxy Formation. Astroph. J., 477:8-+, Mar. 1997.
38. Y. Xu and P. Kostamaa. Efficient outer join data skew handling in parallel DBMS. In VLDB, 2009.
39. Y. Xu, P. Kostamaa, X. Zhou, and L. Chen. Handling data skew in parallel joins in shared-nothing systems. In Proc. of the SIGMOD Conf., 2008.
40. Y. Yu, P. K. Gunda, and M. Isard. Distributed aggregation for data-parallel computing: interfaces and implementations. In Proc. of the 22nd SOSP Symp., 2009.
41. Yu et. al. DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language. In Proc. of the 8th OSDI Symp., 2008.