Interruptible tasks: Treating memory pressure as interrupts for highly scalable data-parallel programs

SOSP 2015 - Proceedings of the 25th ACM Symposium on Operating Systems Principles

Volumn , Issue , 2015, Pages 394-409

  Fang, Lu ^a   Nguyen, Khanh ^a   Xu, Guoqing ^a   Demsky, Brian ^a   Lu, Shan ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

SCALABILITY;

DATA PARALLEL PROGRAMS; DIFFERENT SIZES; MEMORY PRESSURE; MEMORY PROBLEMS; OUT-OF-MEMORY ERRORS; PERFORMANCE AND SCALABILITIES; PROGRAMMING MODELS; RUNTIME SYSTEMS;

EMBEDDED SYSTEMS;

EID: 84957942544     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2815400.2815407     Document Type: Conference Paper

References (62)

1. AsterixDB. https://asterixdb.ics.uci.edu/.
2. Cascading. http://www.cascading.org.
3. Hadoop YARN. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/.
4. Hyracks. http://hyracks.org/.
5. Out of memory error due to appending values to string-builder. http://stackoverflow.com/questions/12831076/.
6. Out of memory error due to large spill buffer. http://stackoverflow.com/questions/8464048/.
7. Out of memory error in a web parser. http://stackoverflow.com/questions/17707883/.
8. Out of memory error in building inverted index. http://stackoverflow.com/questions/17980491/.
9. Out of memory error in computing frequencies of attribute values. http://stackoverflow.com/questions/23042829/.
10. Out of memory error in customer review processing. http://stackoverflow.com/questions/20247185/.
11. Out of memory error in efficient sharded positional indexer. http://www.cs.cmu.edu/~lezhao/TA/2010/HW2/.
12. Out of memory error in hash join using distributed-cache. http://stackoverflow.com/questions/15316539/.
13. Out of memory error in map-side aggregation. http://stackoverflow.com/questions/16684712/.
14. Out of memory error in processing a text file as a record. http://stackoverflow.com/questions/12466527/.
15. Out of memory error in word cooccurrence matrix stripes builder. http://stackoverflow.com/questions/12831076/.
16. The performance comparison between in-mapper combiner and regular combiner. http://stackoverflow.com/questions/10925840/.
17. Reducer hange at the merge step. http://stackoverflow.com/questions/15541900/.
18. Tuning Spark. http://spark.apache.org/docs/latest/tuning.html.
19. AHMAD, F., and CHAKRADHAR, S. T., RAGHUNATHAN, A., and VIJAYKUMAR, T. N. Shufflewatcher: Shuffle-aware scheduling in multi-tenant mapreduce clusters. In USENIX ATC (2014), pp. 1-12.
20. ALSUBAIEE, S., ALTOWIM, Y., ALTWAIJRY, H., BEHM, A., and BORKAR, V. R., BU, Y., CAREY, M. J., CETINDIL, I., CHEELANGI, M., FARAAZ, K., GABRIELOVA, E., GROVER, R., HEILBRON, Z., KIM, Y., LI, C., LI, G., OK, J. M., ONOSE, N., PIRZADEH, P., TSOTRAS, V. J., VERNICA, R., WEN, J., AND WESTMANN, T. Asterixdb: A scalable, open source BDMS. PVLDB 7, 14 (2014), 1905-1916.
21. Hadoop: Open-source implementation of MapReduce. http://hadoop.apache.org.
22. BEHM, A., and BORKAR, V. R., CAREY, M. J., GROVER, R., LI, C., ONOSE, N., VERNICA, R., DEUTSCH, A., PAPAKONSTANTINOU, Y., and TSOTRAS, V. J. ASTERIX: Towards a scalable, semistructured data platform for evolving-world models. Distributed and Parallel Databases 29 (2011), 185-216.
23. BOND, M. D., and MCKINLEY, K. S. Leak pruning. In ASPLOS (2009), pp. 277-288.
24. BORKAR, V. R., and CAREY, M. J., GROVER, R., ONOSE, N., and VERNICA, R. Hyracks: A flexible and extensible foundation for data-intensive computing. In ICDE (2011), pp. 1151-1162.
25. BORKAR, V. R., and CAREY, M. J., and LI, C. Inside "Big Data Management": Ogres, Onions, or Parfaits? In EDBT (2012), pp. 3-14.
26. BU, Y., BORKAR, V., XU, G., and CAREY, M. J. A bloat-aware design for big data applications. In ISMM (2013), pp. 119-130.
27. BU, Y., and BORKAR, V. R., JIA, J., CAREY, M. J., and CONDIE, T. Pregelix: Big(ger) graph analytics on a dataflow engine. PVLDB 8, 2 (2014), 161-172.
28. CHAIKEN, R., JENKINS, B., LARSON, P., RAMSEY, B., SHAKIB, D., WEAVER, S., and ZHOU, J. SCOPE: easy and efficient parallel processing of massive data sets. PVLDB 1, 2 (2008), 1265-1276.
29. CHAMBERS, C., RANIWALA, A., PERRY, F., ADAMS, S., and HENRY, R. R., BRADSHAW, R., and WEIZENBAUM, N. FlumeJava: Easy, efficient data-parallel pipelines. In PLDI (2010), pp. 363-375.
30. CHANG, F., DEAN, J., GHEMAWAT, S., HSIEH, W. C., and WALLACH, D. A., BURROWS, M., CHANDRA, T., FIKES, A., and GRUBER, R. E. Bigtable: A distributed storage system for structured data. ACM Trans. Comput. Syst. 26, 2 (2008), 4:1-4:26.
31. CHU, C. T., and KIM, S. K., LIN, Y. A., YU, Y., BRADSKI, G. R., and NG, A. Y., and OLUKOTUN, K. Map-reduce for machine learning on multicore. In NIPS (2006), pp. 281-288.
32. CONDIE, T., CONWAY, N., ALVARO, P., HELLERSTEIN, J. M., ELMELEEGY, K., and SEARS, R. MapReduce online. In NSDI (2010), pp. 313-328.
33. DEAN, J., and GHEMAWAT, S. MapReduce: Simplified data processing on large clusters. In OSDI (2004), pp. 137-150.
34. GOG, I., GICEVA, J., SCHWARZKOPF, M., VASWANI, K., VYTINIOTIS, D., RAMALINGAM, G., COSTA, M., and MURRAY, D. G., HAND, S., and ISARD, M. Broom: Sweeping out garbage collection from big data systems. In HotOS (2015).
35. GUO, Z., FAN, X., CHEN, R., ZHANG, J., ZHOU, H., MCDIRMID, S., LIU, C., LIN, W., ZHOU, J., and ZHOU, L. Spotting code optimizations in data-parallel pipelines through periscope. In OSDI (2012), pp. 121-133.
36. HERODOTOU, H., LIM, H., LUO, G., BORISOV, N., DONG, L., CETIN, F. B., and BABU, S. Starfish: A self-tuning system for big data analytics. In CIDR (2011), pp. 261-272.
37. HINDMAN, B., KONWINSKI, A., ZAHARIA, M., GHODSI, A., and JOSEPH, A. D., KATZ, R., SHENKER, S., and STOICA, I. Mesos: A platform for fine-grained resource sharing in the data center. In NSDI (2011), pp. 295-308.
38. ISARD, M., BUDIU, M., YU, Y., BIRRELL, A., and FETTERLY. D. Dryad: Distributed data-parallel programs from sequential building blocks. In EuroSys (2007), pp. 59-72.
39. Kryo. https://github.com/EsotericSoftware/kryo.
40. KWON, Y., REN, K., BALAZINSKA, M., and HOWE, B. Managing skew in hadoop. IEEE Data Eng. Bull. 36, 1 (2013), 24-33.
41. KYROLA, A., BLELLOCH, G., and GUESTRIN, C. GraphChi: Large-Scale Graph Computation on Just a PC. In OSDI (2012), pp. 31-46.
42. LIU, J., RAVI, N., CHAKRADHAR, S., and KANDEMIR, M. Panacea: Towards holistic optimization of MapReduce applications. In CGO (2012), pp. 33-43.
43. LOW, Y., GONZALEZ, J., KYROLA, A., BICKSON, D., GUESTRIN, C., and HELLERSTEIN, J. M. Distributed GraphLab: A framework for machine learning in the cloud. PVLDB 5, 8 (2012), 716-727.
44. MITCHELL, N., SCHONBERG, E., and SEVITSKY, G. Four trends leading to java runtime bloat. IEEE Software 27, 1 (2010), 56-63.
45. MITCHELL, N., and SEVITSKY, G. The causes of bloat, the limits of health. In OOPSLA (2007), pp. 245-260.
46. MURRAY, D. G., ISARD, M., and YU, Y. Steno: Automatic optimization of declarative queries. In PLDI (2011), pp. 121-131.
47. NGUYEN, K., WANG, K., BU, Y., FANG, L., HU, J., and XU, G. FACADE: A compiler and runtime for (almost) object-bounded big data applications. In ASPLOS (2015), pp. 675-690.
48. OLSTON, C., REED, B., SRIVASTAVA, U., KUMAR, R., and TOMKINS, A. Pig latin: A not-so-foreign language for data processing. In SIGMOD (2008), pp. 1099-1110.
49. PIKE, R., DORWARD, S., GRIESEMER, R., and QUINLAN, S. Interpreting the data: Parallel analysis with sawzall. Scientific Programming 13, 4 (2005), 277-298.
50. TANG, Y., GAO, Q., and QIN, F. LeakSurvivor: Towards safely tolerating memory leaks for garbage-collected languages. In USENIX ATC (2008), pp. 307-320.
51. THUSOO, A., and SARMA, J. S., JAIN, N., SHAO, Z., CHAKKA, P., ANTHONY, S., LIU, H., WYCKOFF, P., and MURTHY, R. Hive - A warehousing solution over a map-reduce framework. PVLDB 2, 2 (2009), 1626-1629.
52. The TPC Benchmark(TM)H (TPC-H). http://www.tpc.org/tpch.
53. XU, G., MITCHELL, N., ARNOLD, M., ROUNTEV, A., and SEVITSKY, G. Software bloat analysis: Finding, removing, and preventing performance problems in modern large-scale object-oriented applications. In FoSER (2010), pp. 421-426.
54. XU, G. H., MITCHELL, N., ARNOLD, M., ROUNTEV, A., SCHONBERG, E., and SEVITSKY, G. Scalable runtime bloat detection using abstract dynamic slicing. TOSEM 23, 3 (2014), 23.
55. Yahoo! Webscope Program. http://webscope.sandbox.yahoo.com/.
56. YANG, H.-C., DASDAN, A., HSIAO, R.-L., and PARKER, D. S. Map-reduce-merge: Simplified relational data processing on large clusters. In SIGMOD (2007), pp. 1029-1040.
57. YU, Y., and GUNDA, P. K., and ISARD, M. Distributed aggregation for data-parallel computing: Interfaces and implementations. In SOSP (2009), pp. 247-260.
58. YU, Y., ISARD, M., FETTERLY, D., BUDIU, M., ERLINGSSON, U., GUNDA, P. K., and CURREY, J. DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language. In OSDI (2008), pp. 1-14.
59. ZAHARIA, M., CHOWDHURY, M., DAS, T., DAVE, A., MA, J., MCCAULY, M., FRANKLIN, M. J., SHENKER, S., and STOICA, I. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI (2012), pp. 15-28.
60. ZAHARIA, M., CHOWDHURY, M., FRANKLIN, M. J., SHENKER, S., and STOICA, I. Spark: Cluster computing with working sets. In HotCloud (2010).
61. ZHANG, J., ZHOU, H., CHEN, R., FAN, X., GUO, Z., LIN, H., LI, J. Y., LIN, W., ZHOU, J., and ZHOU, L. Optimizing data shuffling in data-parallel computation by understanding user-defined functions. In NSDI (2012), pp. 22-22.
62. ZHOU, J., LARSON, P.-Å., AND CHAIKEN, R. Incorporating partitioning and parallel plans into the SCOPE optimizer. In ICDE (2010), pp. 1060-1071.