Pixida: Optimizing data parallel jobs in wide-area data analytics

Proceedings of the VLDB Endowment

Volumn 9, Issue 2, 2016, Pages 72-83

  Kloudas, Konstantinos ^b   Mamede, Margarida ^a   Preguiça, Nuno ^a   Rodrigues, Rodrigo ^b  

^a UNIVERSIDADE NOVA DE LISBOA   (Portugal)

^b UNIVERSITY OF LISBON   (Portugal)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; GRAPH THEORY; SCHEDULING;

ANALYTICAL QUERIES; DATA ANALYTICS; DATA MOVEMENTS; DATA PARALLEL; GRAPH PARTITIONING PROBLEMS; MULTIPLE DATA; NOVEL ALGORITHM; TRAFFIC REDUCTION;

BIG DATA;

EID: 84975862704     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: None     Document Type: Chapter

References (23)

1. Amazon EC2 Pricing. http://aws.amazon.com/ec2/pricing/.
2. S. Agarwal, S. Kandula, N. Bruno, M.-C. Wu, I. Stoica, and J. Zhou. Re-optimizing Data-parallel Computing. In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation, NSDI'12, pages 281-294, San Jose, CA, USA, 2012. USENIX Association.
3. T. Akidau, A. Balikov, K. Bekiroğlu, S. Chernyak, J. Haberman, R. Lax, S. McVeety, D. Mills, P. Nordstrom, and S. Whittle. MillWheel: Fault-tolerant Stream Processing at Internet Scale. PVLDB, 6(11):1033-1044, 2013.
4. R. Ananthanarayanan, V. Basker, S. Das, A. Gupta, H. Jiang, T. Qiu, A. Reznichenko, D. Ryabkov, M. Singh, and S. Venkataraman. Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD '13, pages 577-588, New York, NY, USA, 2013. ACM.
5. A. Bessani, M. Correia, B. Quaresma, F. Andŕe, and P. Sousa. DepSky: Dependable and Secure Storage in a Cloud-of-clouds. In Proceedings of the 6th ACM European Conference on Computer Systems, EuroSys '11, pages 31-46, Salzburg, Austria, 2011. ACM.
6. N. Bruno, S. Agarwal, S. Kandula, B. Shi, M.-C. Wu, and J. Zhou. Recurring Job Optimization in Scope. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12, pages 805-806, Scottsdale, Arizona, USA, 2012. ACM.
7. J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, W. Hsieh, S. Kanthak, E. Kogan, H. Li, A. Lloyd, S. Melnik, D. Mwaura, D. Nagle, S. Quinlan, R. Rao, L. Rolig, Y. Saito, M. Szymaniak, C. Taylor, R. Wang, and D. Woodford. Spanner: Google's Globally-Distributed Database. In Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation, OSDI'12, pages 251-264, Hollywood, CA, 2012. USENIX Association.
8. T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to Algorithms. McGraw-Hill Higher Education, 3rd edition, 2009.
9. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of the 6th USENIX Symposium on Operating Systems Design and Implementation, OSDI'04, pages 137-149, San Francisco, CA, USA, 2004. USENIX Association.
10. A. Ene, J. Vondŕak, and Y. Wu. Local Distribution and the Symmetry Gap: Approximability of Multiway Partitioning Problems. In Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '13, pages 306-325. SIAM, 2013.
11. A. D. Ferguson, P. Bodik, S. Kandula, E. Boutin, and R. Fonseca. Jockey: Guaranteed Job Latency in Data Parallel Clusters. In Proceedings of the 7th ACM European Conference on Computer Systems, EuroSys '12, pages 99-112, Bern, Switzerland, 2012. ACM.
12. A. Gupta, F. Yang, J. Govig, A. Kirsch, K. Chan, K. Lai, S. Wu, S. G. Dhoot, A. R. Kumar, A. Agiwal, S. Bhansali, M. Hong, J. Cameron, M. Siddiqi, D. Jones, J. Shute, A. Gubarev, S. Venkataraman, and D. Agrawal. Mesa: Geo-Replicated, Near Real-Time, Scalable Data Warehousing. PVLDB, 7(12):1259-1270, 2014.
13. K. Kloudas, M. Mamede, N. Preguiça, and R. Rodrigues. The Generalized Min-Cut Problem. NOVA LINCS Tech. Report. Available at: http://novasys.di.fct.unl.pt/~kkloudas/tr/techreport.pdf.
14. P. Pietzuch, J. Ledlie, J. Shneidman, M. Roussopoulos, M. Welsh, and M. Seltzer. Network-Aware Operator Placement for Stream-Processing Systems. In Proceedings of the 22nd International Conference on Data Engineering, ICDE'06, Atlanta, GA, USA, 2006. IEEE Computer Society.
15. Q. Pu, G. Ananthanarayanan, P. Bodik, S. Kandula, A. Akella, P. Bahl, and I. Stoica. Low Latency Geo-distributed Data Analytics. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, SIGCOMM '15, pages 421-434, London, United Kingdom, 2015. ACM.
16. A. Rabkin, M. Arye, S. Sen, V. S. Pai, and M. J. Freedman. Aggregation and Degradation in JetStream: Streaming Analytics in the Wide Area. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI'14, pages 275-288, Seattle, WA, USA, 2014. USENIX Association.
17. V. V. Vazirani. Approximation Algorithms. Springer-Verlag New York, Inc., New York, NY, USA, 2001.
18. A. Vulimiri, C. Curino, B. Godfrey, K. Karanasos, and G. Varghese. WANalytics: Analytics for a Geo-Distributed Data-Intensive World. CIDR 2015, January 2015.
19. A. Vulimiri, C. Curino, P. B. Godfrey, T. Jungblut, J. Padhye, and G. Varghese. Global Analytics in the Face of Bandwidth and Regulatory Constraints. In Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation, NSDI'15, pages 323-336, Oakland, CA, USA, 2015. USENIX Association.
20. Z. Wu, M. Butkiewicz, D. Perkins, E. Katz-Bassett, and H. V. Madhyastha. SPANStore: Cost-effective Geo-replicated Storage Spanning Multiple Cloud Services. In Proceedings of the 24th ACM Symposium on Operating Systems Principles, SOSP'13, pages 292-308, Pennsylvania, PA, USA, 2013. ACM.
21. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling. In Proceedings of the 5th ACM European Conference on Computer Systems, EuroSys '10, pages 265-278, Paris, France, 2010. ACM.
22. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauly, M. J. Franklin, S. Shenker, and I. Stoica. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation, NSDI'12, pages 15-28, San Jose, CA, USA, 2012. USENIX.
23. M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, and I. Stoica. Discretized Streams: Fault-tolerant Streaming Computation at Scale. In Proceedings of the 24th ACM Symposium on Operating Systems Principles, SOSP'13, pages 423-438, Pennsylvania, PA, USA, 2013. ACM.