Occupy the cloud: Distributed computing for the 99%

SoCC 2017 - Proceedings of the 2017 Symposium on Cloud Computing

Volumn , Issue , 2017, Pages 445-451

  Jonas, Eric ^a   Pu, Qifan ^a   Venkataraman, Shivaram ^a   Stoica, Ion ^a   Recht, Benjamin ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

AWS lambda; Distributed computing; PyWren; Serverless

Indexed keywords

CLOUD COMPUTING; DATA HANDLING; DIGITAL STORAGE; OPEN SYSTEMS;

AWS LAMBDA; COMPUTING ENVIRONMENTS; DISTRIBUTED COMPUTATION FRAMEWORK; DISTRIBUTED COMPUTING MODELS; OPEN SOURCE PLATFORMS; PROTOTYPE IMPLEMENTATIONS; PYWREN; SERVERLESS;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 85032457554     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/3127479.3128601     Document Type: Conference Paper

References (49)

1. ABADI, M., BARHAM, P., CHEN, J., CHEN, Z., DAVIS, A., DEAN, J., DEVIN, M., GHEMAWAT, S., IRVING, G., ISARD, M., ET AL. Tensorflow: A system for large-scale machine learning. In OSDI (2016).
2. ARMBRUST, M., FOX, A., GRIFFITH, R., JOSEPH, A. D., KATZ, R., KONWINSKI, A., LEE, G., PATTERSON, D., RABKIN, A., STOICA, I., ET AL. A view of cloud computing. CACM 53, 4 (2010), 50-58.
3. ASANOVIC, K., AND PATTERSON, D. Firebox: A hardware building block for 2020 warehouse-scale computers. In FAST (2014).
4. Serverless Reference Architecture: MapReduce. https://github.com/awslabs/lambda-refarch-mapreduce.
5. CANNY, J., AND ZHAO, H. Big data analytics with small footprint: Squaring the cloud. In KDD (2013).
6. CARRIERO, N., AND GELERNTER, D. Linda in context. CACM 32, 4 (Apr. 1989).
7. cloudpickle: Extended pickling support for python objects. https://github.com/cloudpipe/cloudpickle.
8. DOUZE, M., JÉGOU, H., SANDHAWALIA, H., AMSALEG, L., AND SCHMID, C. Evaluation of gist descriptors for web-scale image search. In ACM International Conference on Image and Video Retrieval (2009).
9. IEEE P802.3ba, 40Gb/s and 100Gb/s Ethernet Task Force. http://www.ieee802. org/3/ba/.
10. FANG, L., NGUYEN, K., XU, G., DEMSKY, B., AND LU, S. Interruptible tasks: Treating memory pressure as interrupts for highly scalable data-parallel programs. In SOSP (2015).
11. FOULADI, S., WAHBY, R. S., SHACKLETT, B., BALASUBRAMANIAM, K. V., ZENG, W., BHALERAO, R., SIVARAMAN, A., PORTER, G., AND WINSTEIN, K. Encoding, Fast and Slow: Low-Latency Video Processing Using Thousands of Tiny Threads. In NSDI (2017).
12. G. ANANTHANARAYANAN, A. GHODSI, S. SHENKER, I. STOICA. Disk-Locality in Datacenter Computing Considered Irrelevant. In Proc. HotOS (2011).
13. GAO, P. X., NARAYAN, A., KARANDIKAR, S., CARREIRA, J., HAN, S., AGARWAL, R., RATNASAMY, S., AND SHENKER, S. Network requirements for resource disaggregation. In OSDI (2016).
14. HAN, S., EGI, N., PANDA, A., RATNASAMY, S., SHI, G., AND SHENKER, S. Network support for resource disaggregation in next-generation datacenters. In HotNets (2013).
15. HAN, S., AND RATNASAMY, S. Large-scale computation not at the cost of expressiveness. In HotOS (2013).
16. HENDRICKSON, S., STURDEVANT, S., HARTER, T., VENKATARAMANI, V., ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEAU, R. H. Serverless computation with OpenLambda. In HotCloud (2016).
17. HERODOTOU, H., LIM, H., LUO, G., BORISOV, N., DONG, L., CETIN, F. B., AND BABU, S. Starfish: A self-tuning system for big data analytics. In CIDR (2011).
18. HETTRICK, S., ANTONIOLETTI, M., CARR, L., CHUE HONG, N., CROUCH, S., DE ROURE, D., EMSLEY, I., GOBLE, C., HAY, A., INUPAKUTIKA, D., JACKSON, M., NENADIC, A., PARKINSON, T., PARSONS, M. I., PAWLIK, A., PERU, G., PROEME, A., ROBINSON, J., AND SUFI, S. Uk research software survey 2014. https://doi.org/10.5281/zenodo.14809, Dec. 2014.
19. HINDMAN, B., KONWINSKI, A., ZAHARIA, M., GHODSI, A., JOSEPH, A., KATZ, R., SHENKER, S., AND STOICA, I. Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. In Proc. NSDI (2011).
20. HP The Machine: Our vision for the Future of Computing. https://www.labs.hpe. com/the-machine.
21. ISARD, M., PRABHAKARAN, V., CURREY, J., WIEDER, U., TALWAR, K., AND GOLDBERG, A. Quincy: Fair Scheduling for Distributed Computing Clusters. In Proc. SOSP (2009), pp. 261-276.
22. LAGAR-CAVILLA, H. A.,WHITNEY, J. A., SCANNELL, A. M., PATCHIN, P., RUMBLE, S. M., DE LARA, E., BRUDNO, M., AND SATYANARAYANAN, M. Snowflock: Rapid virtual machine cloning for cloud computing. In EuroSys (2009).
23. LI, M., ANDERSEN, D. G., PARK, J. W., SMOLA, A. J., AHMED, A., JOSIFOVSKI, V., LONG, J., SHEKITA, E. J., AND SU, B.-Y. Scaling distributed machine learning with the parameter server. In OSDI (2014).
24. MCAULEY, J., TARGETT, C., SHI, Q., AND VAN DEN HENGEL, A. Image-based recommendations on styles and substitutes. In SIGIR (2015).
25. MCSHERRY, F., ISARD, M., AND MURRAY, D. G. Scalability! but at what COST? In HotOS (2015).
26. MOMCHEVA, I., AND TOLLERUD, E. Software Use in Astronomy: an Informal Survey. arXiv 1507.03989 (2015).
27. NIGHTINGALE, E. B., ELSON, J., FAN, J., HOFMANN, O., HOWELL, J., AND SUZUE, Y. Flat datacenter storage. In OSDI (2012).
28. NIU, F., RECHT, B., RE, C., AND WRIGHT, S. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In NIPS (2011).
29. OLIVA, A., AND TORRALBA, A. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of computer vision 42, 3 (2001), 145-175.
30. O'MALLEY, O. TeraByte Sort on Apache Hadoop. http://sortbenchmark.org/YahooHadoop.pdf.
31. OpenWhisk. https://developer.ibm.com/openwhisk/.
32. OUSTERHOUT, K., PANDA, A., ROSEN, J., VENKATARAMAN, S., XIN, R., RATNASAMY, S., SHENKER, S., AND STOICA, I. The case for tiny tasks in compute clusters. In HotOS (2013).
33. OUSTERHOUT, K., WENDELL, P., ZAHARIA, M., AND STOICA, I. Sparrow: distributed, low latency scheduling. In SOSP (2013).
34. PENG, D., AND DABEK, F. Large-scale incremental processing using distributed transactions and notifications. In OSDI (2010).
35. POWER, R., AND LI, J. Piccolo: Building fast, distributed programs with partitioned tables. In OSDI (2010).
36. Redis server side scripting. https://redis.io/commands/eval.
37. Redis benchmarks. https://redis.io/topics/benchmarks.
38. RUMBLE, S. M., ONGARO, D., STUTSMAN, R., ROSENBLUM, M., AND OUSTERHOUT, J. K. It's Time for Low Latency. In Proc. HotOS (2011).
39. RUSSAKOVSKY, O., DENG, J., SU, H., KRAUSE, J., SATHEESH, S., MA, S., HUANG, Z., KARPATHY, A., KHOSLA, A., BERNSTEIN, M., BERG, A. C., AND LI, F.-F. ImageNet Large Scale Visual Recognition Challenge. IJCV 115, 3 (2015), 211-252.
40. SCHWARZKOPF, M., KONWINSKI, A., ABD-EL-MALEK, M., AND WILKES, J. Omega: flexible, scalable schedulers for large compute clusters. In Proc. EuroSys (2013).
41. SCOTT, C. Latency trends. http://colin-scott.github.io/blog/2012/12/24/latency-trends/.
42. SHVACHKO, K., KUANG, H., RADIA, S., AND CHANSLER, R. The Hadoop Distributed File System. In Mass storage systems and technologies (MSST) (2010).
43. Sort Benchmark. http://sortbenchmark.org.
44. Tuning Java Garbage Collection for Apache Spark Applications. https://goo.gl/SIWlqx.
45. Tuning Spark. https://spark.apache.org/docs/latest/tuning.html# garbage-collection-tuning.
46. VAVILAPALLI, V. K., MURTHY, A. C., DOUGLAS, C., AGARWAL, S., KONAR, M., EVANS, R., GRAVES, T., LOWE, J., SHAH, H., SETH, S., ET AL. Apache Hadoop YARN: Yet another resource negotiator. In SoCC (2013).
47. VENKATARAMAN, S., YANG, Z., FRANKLIN, M., RECHT, B., AND STOICA, I. Ernest: Efficient performance prediction for large-scale advanced analytics. In NSDI (2016).
48. X1 instances. https://aws.amazon.com/ec2/instance-types/x1/.
49. ZAHARIA, M., CHOWDHURY, M., DAS, T., DAVE, A., MA, J., MCCAULEY, M., FRANKLIN, M., SHENKER, S., AND STOICA, I. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In Proc. NSDI (2011).