Making state explicit for imperative big data processing

Proceedings of the 2014 USENIX Annual Technical Conference, USENIX ATC 2014

Volumn , Issue , 2014, Pages 49-60

  Fernandez, Raul Castro ^a   Migliavacca, Matteo ^b   Kalyvianaki, Evangelia ^c   Pietzuch, Peter ^a  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF KENT   (United Kingdom)

^c NONE

Author keywords

[No Author keywords available]

Indexed keywords

BIG DATA; DATA FLOW ANALYSIS; DATA HANDLING; FAULT TOLERANCE; JAVA PROGRAMMING LANGUAGE; LEARNING ALGORITHMS; MACHINE LEARNING; MATLAB; SEMANTICS;

DATA PARALLEL; HIGH-LOW; HIGH-THROUGHPUT; IMPERATIVE LANGUAGES; JAVA PROGRAM; LOW LATENCY; LOW OVERHEAD; MACHINE LEARNING ALGORITHMS; PARALLEL PROCESSING; PERFORMANCE AND SCALABILITIES;

SCALABILITY;

EID: 85077469444     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (40)

1. AKIDAU, T., BALIKOV, A., ET AL. MillWheel: Fault-Tolerant Stream Processing at Internet Scale. In VLDB (2013).
2. ALVARO, P., CONWAY, N., ET AL. Blazes: Coordination Analysis for Distributed Programs. In ICDE (2014).
3. BECK, M., AND PINGALI, K. From Control Flow to Dataflow. In ICPP (1990).
4. BHATOTIA, P., WIEDER, A., ET AL. Incoop: MapReduce for Incremental Computations. In SOCC (2011).
5. BU, Y., HOWE, B., ET AL. HaLoop: Efficient Iterative Data Processing on Large Clusters. In VLDB (2010).
6. CHAIKEN, R., JENKINS, B., ET AL. SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. In VLDB (2008).
7. CHAMBERS, C., RANIWALA, A., ET AL. FlumeJava: Easy, Efficient Data-Parallel Pipelines. In PLDI (2010).
8. DEAN, J., AND GHEMAWAT, S. MapReduce: Simplified Data Processing on Large Clusters. In CACM (2008).
9. EWEN, S., TZOUMAS, K., ET AL. Spinning Fast Iterative Data Flows. In VLDB (2012).
10. FERNANDEZ, R. C., MIGLIAVACCA, M., ET AL. Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management. In SIGMOD (2013).
11. GUNDA, P. K., RAVINDRANATH, L., ET AL. Nectar: Automatic Management of Data and Comp. in Datacenters. In OSDI (2010).
12. HE, B., YANG, M., ET AL. Comet: Batched Stream Processing for Data Intensive Distributed Computing. In SOCC (2010).
13. HUESKE, F., PETERS, M., ET AL. Opening the Black Boxes in Data Flow Optimization. In VLDB (2012).
14. HWANG, J.-H., BALAZINSKA, M., ET AL. High-Availability Algorithms for Distributed Stream Processing. In ICDE (2005).
15. ISARD, M., BUDIU, M., ET AL. Dryad: Dist. Data-Parallel Programs from Sequential Building Blocks. In EuroSys (2007).
16. JOHNSTON, W. M., HANNA, J., ET AL. Advances in Dataflow Programming Languages. In CSUR (2004).
17. KDNUGGETS ANNUAL SOFTWARE POLL. RapidMiner and R vie for the First Place. http://goo.gl/OLikb, 2013.
18. LI, F., POP, A., ET AL. Automatic Extraction of Coarse-Grained Data-Flow Threads from Imperative Programs. In Micro (2012).
19. LOGOTHETHIS, D., OLSON, C., ET AL. Stateful Bulk Processing for Incremental Analytics. In SOCC (2010).
20. LOW, Y., BICKSON, D., ET AL. Dist. GraphLab: A Framework for ML and Data Mining in the Cloud. In VLDB (2012).
21. MA, J., SAUL, L. K., ET AL. Identifying Suspicious URLs: an Application of Large-Scale Online Learning. In ICML (2009).
22. MALEWICZ, G., AUSTERN, M. H., ET AL. Pregel: A System for Large-scale Graph Processing. In SIGMOD (2010).
23. MISHNE, G., DALTON, J., ET AL. Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture. In SIGMOD (2013).
24. MITCHELL, C., POWER, R., ET AL. Oolong: Asynchronous Distributed Applications Made Easy. In APSYS (2012).
25. MURRAY, D., SCHWARZKOPF, M., ET AL. CIEL: A Universal Exec. Engine for Distributed Data-Flow Comp. In NSDI (2011).
26. MURRAY, D. G., MCSHERRY, F., ET AL. Naiad: A Timely Dataflow System. In SOSP (2013).
27. NIKHIL, R. S., ET AL. Executing a Program on the MIT Tagged-Token Dataflow Architecture. In TC (1990).
28. OLSTON, C., REED, B., ET AL. Pig Latin: A Not-So-Foreign Language for Data Processing. In SIGMOD (2008).
29. ONGARO, D., RUMBLE, S. M., ET AL. Fast Crash Recovery in RAMcloud. In SOSP (2011).
30. POWER, R., AND LI, J. Piccolo: Building Fast, Distributed Programs with Partitioned Tables. In OSDI (2010).
31. SCHELTER, S., EWEN, S., ET AL. All Roads Lead to Rome: Optimistic Recovery for Distributed Iterative Data Processing. In CIKM (2013).
32. SUMBALY, R., KREPS, J., ET AL. The Big Data Ecosystem at LinkedIn. In SIGMOD (2013).
33. VALLÉE-RAI, R., HENDREN, L., ET AL. Soot: A Java Optimization Framework. In CASCON (1999).
34. VANDIERENDONCK, H., RUL, S., ET AL. The Paralax Infrastructure: Automatic Parallelization with a Helping Hand. In PACT (2010).
35. VENKATARAMAN, S., BODZSAR, E., ET AL. Presto: Dist. ML and Graph Processing with Sparse Matrices. In EuroSys (2013).
36. XIN, R. S., ROSEN, J., ET AL. Shark: SQL and Rich Analytics at Scale. In SIGMOD (2013).
37. YU, Y., ISARD, M., ET AL. DryadLINQ: a System for General-Purpose Distributed Data-Parallel Computing using a High-Level Language. In OSDI (2008).
38. ZAHARIA, M., CHOWDHURY, M., ET AL. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In NSDI (2012).
39. ZAHARIA, M., DAS, T., ET AL. Discretized Streams: Fault-tolerant Streaming Computation at Scale. In SOSP (2013).
40. ZAHARIA, M., KONWINSKI, A., ET AL. Improving MapReduce Performance in Heterogeneous Environments. In OSDI (2008).