NeCTAR: Automatic management of data and computation in datacenters

Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2010

Volumn , Issue , 2019, Pages 75-88

  Gunda, Pradeep Kumar ^a   Ravindranath, Lenin ^a,b   Thekkath, Chandramohan A ^a   Yu, Yuan ^a   Zhuang, Li ^a  

^a MICROSOFT RESEARCH   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DIGITAL STORAGE; SYSTEMS ANALYSIS;

AUTOMATIC MANAGEMENT; CACHING SERVICES; DATACENTER MANAGEMENTS; DESIGN AND IMPLEMENTATIONS; LOST OPPORTUNITIES; MANUAL MANAGEMENT; RESOURCE UTILIZATIONS; WASTEFUL CONSUMPTION;

INFORMATION MANAGEMENT;

EID: 85076910718     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. AGRAWAL, P., KIFER, D., AND OLSTON, C. Scheduling shared scans of large data files. Proc. VLDB Endow. 1, 1 (2008), 958-969.
2. AGRAWAL, S., CHAUDHURI, S., AND NARASAYYA, V. R. Automated selection of materialized views and indexes in SQL databases. In VLDB (2000), pp. 496-505.
3. ALVARO, P., CONDIE, T., CONWAY, N., ELMELEEGY, K., HELLERSTEIN, J. M., AND SEARS, R. Boom analytics: exploring data-centric, declarative programming for the cloud. In EuroSys'10: Proceedings of the 5th European conference on Computer systems (2010), pp. 223-236.
4. BRODER, A. Z. Some applications of Rabins´ fingerprinting method. In Sequences II: Methods in Communications, Security, and Computer Science (1993), Springer-Verlag, pp. 143-152.
5. CERI, S., AND WIDOM, J. Deriving production rules for incremental view maintenance. In VLDB'91: Proceedings of the 17th International Conference on Very Large Data Bases (1991), pp. 577-589.
6. CHAIKEN, R., JENKINS, B., LARSON, P.-A., RAMSEY, B., SHAKIB, D., WEAVER, S., AND ZHOU, J. SCOPE: easy and efficient parallel processing of massive data sets. Proc. VLDB Endow. 1, 2 (2008), 1265-1276.
7. DEAN, J., AND GHEMAWAT, S. Mapreduce: simplified data processing on large clusters. Commun. ACM 51, 1 (2008), 107-113.
8. FETTERLY, D., HARIDASAN, M., ISARD, M., AND SUNDARARAMAN, S. TidyFS: A simple and small distributed filesystem. Tech. Rep. MSR-TR-2010-124, Microsoft Research, October 2010.
9. GOLUMBIC, M. C. Algorithmic Graph Theory and Perfect Graphs (Annals of Discrete Mathematics, Vol. 57). North-Holland Publishing Co., Amsterdam, The Netherlands, The Netherlands, 2004.
10. GRAY, J., CHAUDHURI, S., BOSWORTH, A., LAYMAN, A., REICHART, D., VENKATRAO, M., PELLOW, F., AND PIRAHESH, H. Data cube: A relational aggregation operator generalizing group-by, cross-tab, and sub-totals. Data Mining and Knowledge Discovery 1, 1 (1997).
11. GRAY, J., SZALAY, A., THAKAR, A., KUNSZT, P., STOUGHTON, C., SLUTZ, D., AND VANDENBERG, J. Data mining the SDSS SkyServer database. In Distributed Data and Structures 4: Records of the 4th International Meeting (Paris, France, March 2002), Carleton Scientific, pp. 189-210. Also available as MSR-TR-2002-01.
12. The Hadoop project. http://hadoop.apache.org/.
13. HALEVY, A. Y. Answering Queries Using Views: A Survey. VLDB J. 10, 4 (2001), 270-294.
14. HE, B., YANG, M., GUO, Z., CHEN, R., SU, B., LIN, W., AND ZHOU, L. Comet: batched stream processing for data intensive distributed computing. In ACM Symposium on Cloud Computing (SOCC) (2010), pp. 63-74.
15. HEYDON, A., LEVIN, R., MANN, T., AND YU, Y. Software Configuration Management Using Vesta. Springer-Verlag, 2006.
16. HEYDON, A., LEVIN, R., AND YU, Y. Caching function calls using precise dependencies. In PLDI'00: Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation (New York, NY, USA, 2000), ACM, pp. 311-320.
17. The HIVE project. http://hadoop.apache.org/hive/.
18. ISARD, M., BUDIU, M., YU, Y., BIRRELL, A., AND FETTERLY, D. Dryad: distributed data-parallel programs from sequential building blocks. In EuroSys'07: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 (2007), pp. 59-72.
19. LEE, K. Y., SON, J. H., AND KIM, M. H. Efficient incremental view maintenance in data warehouses. In CIKM'01: Proceedings of the tenth international conference on Information and knowledge management (2001), pp. 349-356.
20. The LINQ project. http://msdn.microsoft.com/netframework/future/linq/.
21. LIU, Y. A., STOLLER, S. D., AND TEITELBAUM, T. Static caching for incremental computation. ACM Trans. Program. Lang. Syst. 20, 3 (1998), 546-585.
22. LOGOTHETIS, D., OLSTON, C., REED, B., WEBB, K., AND YOCUM, K. Stateful bulk processing for incremental algorithms. In ACM Symposium on Cloud Computing (SOCC) (2010).
23. OLSTON, C., REED, B., SILBERSTEIN, A., AND SRIVASTAVA, U. Automatic optimization of parallel dataflow programs. In ATC'08: USENIX 2008 Annual Technical Conference on Annual Technical Conference (2008), pp. 267-273.
24. OLSTON, C., REED, B., SRIVASTAVA, U., KUMAR, R., AND TOMKINS, A. Pig latin: a not-so-foreign language for data processing. In SIGMOD'08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data (2008), pp. 1099-1110.
25. PIKE, R., DORWARD, S., GRIESEMER, R., AND QUINLAN, S. Interpreting the data: Parallel analysis with Sawzall. Scientific Programming 13, 4 (2005).
26. POPA, L., BUDIU, M., YU, Y., AND ISARD, M. DryadInc: Reusing work in large-scale computations. In Workshop on Hot Topics in Cloud Computing (HotCloud) (San Diego, CA, June 15 2009).
27. PUGH, W., AND TEITELBAUM, T. Incremental computation via function caching. In POPL'89: Proceedings of the 16th ACM SIGPLAN-SIGACT symposium on Principles of programming languages (1989), pp. 315-328.
28. YU, Y., GUNDA, P. K., AND ISARD, M. Distributed aggregation for data-parallel computing: interfaces and implementations. In SOSP'09: Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles (2009), pp. 247-260.
29. YU, Y., ISARD, M., FETTERLY, D., BUDIU, M., ERLINGSSON, Ú., GUNDA, P. K., AND CURREY, J. DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language. In Proceedings of the 8th Symposium on Operating Systems Design and Implementation (OSDI) (2008), pp. 1-14.