Optimizing data partitioning for data-parallel computing

13th Workshop on Hot Topics in Operating Systems, HotOS 2011

Volumn , Issue , 2011, Pages

  Ke, Qifa ^a   Prabhakaran, Vijayan ^a   Xie, Yinglian ^a   Yu, Yuan ^a   Wu, Jingyue ^b   Yang, Junfeng ^b  

^a MICROSOFT RESEARCH   (United States)

^b Columbia University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA PARALLEL COMPUTING; DATA PARTITIONING; DATABASE COMMUNITY; DRYADLINQ; MAP-REDUCE; OPTIMAL DATA; STATE-OF-THE-ART SYSTEM; USER DEFINED FUNCTIONS;

DATA HANDLING;

EID: 85092653290     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. The Hadoop Project. http://hadoop.apache.org.
2. The HIVE Project. http://hadoop.apache.org/hive/.
3. Vertica. http://www.vertica.com.
4. K. Beyer, P. J. Haas, B. Reinwald, Y. Sismanis, and R. Gemulla. On synopses for distinct-value estimation under multiset operations. In SIGMOD 2007.
5. S. Boyd and L. Vandenberghe. Convex optimization. Cambridge University Press, 2004.
6. S. Chaudhuri, R. Motwani, and V. R. Narasayya. Random sampling for histogram construction: How much is enough? In SIGMOD 1998.
7. G. Cormode and M. Hadjieleftheriou. Finding frequent items in data streams. In VLDB 2008.
8. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In OSDI 2004.
9. D. J. DeWitt, J. F. Naughton, D. A. Schneider, and S. Seshadri. Practical skew handling in parallel joins. In VLDB 1992.
10. S. F. Goldsmith, A. S.Aiken, and D. S.Wilkerson. Measuring empirical computational complexity. In ESEC/FSE 2007.
11. S. Gulwani, K. Mehra, and T. Chilimbi. SPEED: Precise and efficient static estimation of program computational complexity. In POPL 2009.
12. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In Eurosys 2007.
13. Y. Kwon, M. Balazinska, B. Howe, and J. Rolia. Skew-resistant parallel processing of feature-extracting scientific user-defined functions. In SoCC 2010.
14. D. Narayanan and M. Satyanarayanan. Predictive resource management for wearable computing. In MobiSys 2003.
15. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: A not-so-foreign language for data processing. In SIGMOD 2008.
16. A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. Dewitt, S. Madden, and M. Stonebraker. A comparison of approaches to large-scale data analysis. In SIGMOD 2009.
17. L. Wasserman. All of statistics. Spring-Verlag, New York, page 408, 2004.
18. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language. In OSDI 2008.
19. Y. Zhao, Y. Xie, F. Yu, Q. Ke, Y. Yu, Y. Chen, and E. Gillum. Bot-Graph: Large scale spamming botnet detection. In NSDI 2009.