Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language

Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2008

Volumn , Issue , 2019, Pages 1-14

  Yu, Yuan ^a   Isard, Michael ^a   Fetterly, Dennis ^a   Budiu, Mihai ^a   Erlingsson, Úlfar ^b   Gunda, Pradeep Kumar ^a   Currey, Jon ^a  

^a MICROSOFT RESEARCH   (United States)

^b REYKJAVIK UNIVERSITY   (Iceland)

Author keywords

[No Author keywords available]

Indexed keywords

DISTRIBUTED COMPUTER SYSTEMS; HIGH LEVEL LANGUAGES; SYSTEMS ANALYSIS;

ABSOLUTE PERFORMANCE; CONTINUOUS OPERATION; EXECUTION ENVIRONMENTS; EXECUTION PLATFORMS; HIGH-LEVEL PROGRAMMING LANGUAGE; LANGUAGE EXTENSIONS; LARGE SCALE DISTRIBUTED COMPUTING; SEQUENTIAL PROGRAMS;

PROGRAM DEBUGGING;

EID: 85076882757     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (39)

1. The DryadLINQ project. http://research.microsoft.com/research/sv/DryadLINQ/.
2. The LINQ project. http://msdn.microsoft.com/netframework/future/linq/.
3. Sort benchmark. http://research.microsoft.com/barc/SortBenchmark/.
4. BARU, C. K., FECTEAU, G., GOYAL, A., HSIAO, H., JHINGRAN, A., PADMANABHAN, S., COPELAND, G. P., AND WILSON, W. G. DB2 parallel edition. IBM Systems Journal 34, 2, 1995.
5. BECK, M., DONGARRA, J., AND PLANK, J. S. NetSolve/D: A massively parallel grid execution system for scalable data intensive collaboration. In International Parallel and Distributed Processing Symposium (IPDPS), 2005.
6. BLELLOCH, G. E. Programming parallel algorithms. Communications of the ACM (CACM) 39, 3, 1996.
7. BLUMOFE, R. D., JOERG, C. F., KUSZMAUL, B., LEISERSON, C. E., RANDALL, K. H., AND ZHOU, Y. Cilk: An efficient multithreaded runtime system. In ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), 1995.
8. BORAL, H., ALEXANDER, W., CLAY, L., COPELAND, G., DANFORTH, S., FRANKLIN, M., HART, B., SMITH, M., AND VALDURIEZ, P. Prototyping Bubba, a highly parallel database system. IEEE Trans. on Knowl. and Data Eng. 2, 1, 1990.
9. CARNAHAN, J., AND DECOSTE, D. Pipelets: A framework for distributed computation. In W4: Learning in Web Search, 2005.
10. CHAIKEN, R., JENKINS, B., LARSON, P.-Å., RAMSEY, B., SHAKIB, D., WEAVER, S., AND ZHOU, J. SCOPE: Easy and efficient parallel processing of massive data sets. In International Conference of Very Large Data Bases (VLDB), 2008.
11. CHANG, F., DEAN, J., GHEMAWAT, S., HSIEH, W. C., WALLACH, D. A., BURROWS, M., CHANDRA, T., FIKES, A., AND GRUBER, R. E. BigTable: A distributed storage system for structured data. In Symposium on Operating System Design and Implementation (OSDI), 2006.
12. CHIH YANG, H., DASDAN, A., HSIAO, R.-L., AND PARKER, D. S. Map-reduce-merge: simplified relational data processing on large clusters. In SIGMOD international conference on Management of data, 2007.
13. CHU, L., TANG, H., YANG, T., AND SHEN, K. Optimizing data aggregation for cluster-based internet services. In Symposium on Principles and practice of parallel programming (PPoPP), 2003.
14. CRUANES, T., DAGEVILLE, B., AND GHOSH, B. Parallel SQL execution in Oracle 10g. In ACM SIGMOD, 2004.
15. DEAN, J., AND GHEMAWAT, S. MapReduce: Simplified data processing on large clusters. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI), 2004.
16. DESHPANDE, A., IVES, Z., AND RAMAN, V. Adaptive query processing. Foundations and Trends in Databases 1, 1, 2007.
17. DEWITT, D., GHANDEHARIZADEH, S., SCHNEIDER, D., HSIAO, H., BRICKER, A., AND RASMUSSEN, R. The Gamma database machine project. IEEE Transactions on Knowledge and Data Engineering 2, 1, 1990.
18. DEWITT, D., AND GRAY, J. Parallel database systems: The future of high performance database processing. Communications of the ACM 36, 6, 1992.
19. DUFFY, J. A query language for data parallel programming. In Proceedings of the 2007 workshop on Declarative aspects of multicore programming, 2007.
20. ENGLERT, S., GLASSTONE, R., AND HASAN, W. Parallelism and its price: A case study of nonstop SQL/MP. In Sigmod Record, 1995.
21. FENG, L., LU, H., TAY, Y. C., AND TUNG, A. K. H. Buffer management in distributed database systems: A data mining based approach. In International Conference on Extending Database Technology, 1998, H.-J. Schek, F. Saltor, I. Ramos, and G. Alonso, Eds., vol. 1377 of Lecture Notes in Computer Science.
22. GRAEFE, G. Encapsulation of parallelism in the Volcano query processing system. In SIGMOD International Conference on Management of data, 1990.
23. GRAY, J., SZALAY, A., THAKAR, A., KUNSZT, P., STOUGHTON, C., SLUTZ, D., AND VANDENBERG, J. Data mining the SDSS SkyServer database. In Distributed Data and Structures 4: Records of the 4th International Meeting, 2002.
24. HASAN, W., FLORESCU, D., AND VALDURIEZ, P. Open issues in parallel query optimization. SIGMOD Rec. 25, 3, 1996.
25. HELLERSTEIN, J. M., STONEBRAKER, M., AND HAMILTON, J. Architecture of a database system. Foundations and Trends in Databases 1, 2, 2007.
26. ISARD, M., BUDIU, M., YU, Y., BIRRELL, A., AND FETTERLY, D. Dryad: Distributed data-parallel programs from sequential building blocks. In Proceedings of European Conference on Computer Systems (EuroSys), 2007.
27. KABRA, N., AND DEWITT, D. J. Efficient mid-query re-optimization of sub-optimal query execution plans. In SIGMOD International Conference on Management of Data, 1998.
28. KOSSMANN, D. The state of the art in distributed query processing. ACM Comput. Surv. 32, 4, 2000.
29. MORVAN, F., AND HAMEURLAIN, A. Dynamic memory allocation strategies for parallel query execution. In Symposium on Applied computing (SAC), 2002.
30. OINN, T., GREENWOOD, M., ADDIS, M., FERRIS, J., GLOVER, K., GOBLE, C., HULL, D., MARVIN, D., LI, P., LORD, P., POCOCK, M. R., SENGER, M., WIPAT, A., AND WROE, C. Taverna: Lessons in creating a workflow environment for the life sciences. Concurrency and Computation: Practice and Experience 18, 10, 2005.
31. OLSTON, C., REED, B., SRIVASTAVA, U., KUMAR, R., AND TOMKINS, A. Pig Latin: A not-so-foreign language for data processing. In International Conference on Management of Data (Industrial Track) (SIGMOD), 2008.
32. PIKE, R., DORWARD, S., GRIESEMER, R., AND QUINLAN, S. Interpreting the data: Parallel analysis with Sawzall. Scientific Programming 13, 4, 2005.
33. SIMMA, A., GOLDSZMIDT, M., MACCORMICK, J., BARHAM, P., BLACK, R., ISAACS, R., AND MORTIER, R. CT-NOR: representing and reasoning about events in continuous time. In International Conference on Uncertainty in Artificial Intelligence, 2008.
34. STONEBRAKER, M., BEAR, C., ÇETINTEMEL, U., CHERNIACK, M., GE, T., HACHEM, N., HARIZOPOULOS, S., LIFTER, J., ROGERS, J., AND ZDONIK, S. One size fits all? Part 2: Benchmarking results. In Conference on Innovative Data Systems Research (CIDR), 2005.
35. STONEBRAKER, M., MADDEN, S., ABADI, D. J., HARIZOPOULOS, S., HACHEM, N., AND HELLAND, P. The end of an architectural era (it's time for a complete rewrite). In International Conference of Very Large Data Bases (VLDB), 2007.
36. TAYLOR, I., SHIELDS, M., WANG, I., AND HARRISON, A. Workflows for e-Science. 2007, ch. The Triana Workflow Environment: Architecture and Applications, pp. 320-339.
37. TRINDER, P., LOIDL, H.-W., AND POINTON, R. Parallel and distributed Haskells. Journal of Functional Programming 12, (4&5), 2002.
38. YU, Y., ISARD, M., FETTERLY, D., BUDIU, M., ERLINGSSON, Ú., GUNDA, P. K., CURREY, J., MCSHERRY, F., AND ACHAN, K. Some sample programs written in DryadLINQ. Tech. Rep. MSR-TR-2008-74, Microsoft Research, 2008.
39. ZHAO, Y., HATEGAN, M., CLIFFORD, B., FOSTER, I., VON LASZEWSKI, G., NEFEDOVA, V., RAICU, I., STEF-PRAUN, T., AND WILDE, M. Swift: Fast, reliable, loosely coupled parallel computation. IEEE Congress on Services, 2007.