Low latency geo-distributed data analytics

SIGCOMM 2015 - Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

Volumn , Issue , 2015, Pages 421-434

  Pu, Qifan ^a,b   Ananthanarayanan, Ganesh ^a   Kandula, Peter Bodik ^a   Kandula, Srikanth ^a   Akella, Aditya ^c   Bahl, Paramvir ^a   Stoica, Ion ^b  

^a MICROSOFT RESEARCH   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF WISCONSIN MADISON   (United States)

Author keywords

Data analytics; Geo distributed; Low latency; Network aware; Wan analytics

Indexed keywords

BUDGET CONTROL; CONVOLUTIONAL CODES; IRIDIUM; OPTIMIZATION;

DATA ANALYTICS; DISTRIBUTED DATA ANALYTICS; GEO-DISTRIBUTED; LOW LATENCY; NETWORK BOTTLENECKS; NETWORK-AWARE; PLACEMENT OPTIMIZATION; VARIABLE CAPACITY;

WIDE AREA NETWORKS;

EID: 84962258940     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2829988.2787505     Document Type: Conference Paper

References (62)

1. Amazon EC2 Instance Types. http://aws.amazon.com/ec2/instance-types/.
2. Amazon Web Services. http://aws.amazon.com/about-aws/global-infrastructure/.
3. Apache Calcite. http://optiq.incubator.apache.org/.
4. Big Data Benchmark. https://amplab.cs.berkeley.edu/benchmark/.
5. EC2 Pricing. http://aws.amazon.com/ec2/pricing/.
6. Google Datacenter Locations. http://www.google.com/about/datacenters/inside/locations/.
7. Gurobi Optimization. http://www.gurobi.com/.
8. Hadoop Distributed File System. http://hadoop.apache.org/docs/r1.2.1/hdfsdesign.html.
9. How Map and Reduce operations are actually carried out. http://wiki.apache.org/hadoop/HadoopMapReduce.
10. Linux Traffic Control. http://lartc.org/manpages/tc.txt.
11. Microsoft Datacenters. http://www.microsoft.com/en-us/server-cloud/cloud-os/global-datacenters.aspx.
12. TPC Decision Support Benchmark. http://www.tpc.org/tpcds/.
13. Measuring Internet Congestion: A preliminary report. https://ipp.mit.edu/sites/default/-les/documents/Congestion-handout-final.pdf, 2014.
14. S. Agarwal, S. Kandula, N. Bruno, M.-C. Wu, I. Stoica, and J. Zhou. Re-optimizing Data-Parallel Computing. In USENIX NSDI, 2012.
15. S. Agarwal, B. Mozafari, A. Panda, M. H., S. Madden, and I. Stoica. Blink DB: Queries with Bounded Errors and Bounded Response Times on Very Large Data. In ACM EuroSys, 2013.
16. G. Ananthanarayanan, A. Ghodsi, S. Shenker, and I. Stoica. Effiective Straggler Mitigation: Attack of the Clones. In USENIX NSDI, 2013.
17. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, and I. Stoica. PACMan: Coordinated memory caching for parallel jobs. In USENIX NSDI, 2012.
18. G. Ananthanarayanan, M. C.-C. Hung, X. Ren, I. Stoica, A. Wierman, and M. Yu. GRASS: Trimming Stragglers in Approximation Analytics. USENIX NSDI, 2014.
19. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the Outliers in Map-Reduce Clusters using Mantri. In USENIX OSDI, 2010.
20. Apache Hadoop NextGen MapReduce (YARN). Retrieved 9/24/2013, URL: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html.
21. A. Balachandran, V. Sekar, A. Akella, S. Seshan, I. Stoica, and H. Zhang. Developing a Predictive Model of Quality of Experience for Internet Video. In ACM SIGCOMM, 2013.
22. P. A. Bernstein, N. Goodman, E. Wong, C. L. Reeve, and J. B. Rothnie Jr. Query Processing in a System for Distributed Databases (SDD-1). ACM Transactions on Database Systems, 1981.
23. E. Boutin, J. Ekanayake, W. Lin, B. Shi, J. Zhou, Z. Qian, M. Wu, and L. Zhou. Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing. In USENIX OSDI, 2014.
24. M. Calder, X. Fan, Z. Hu, E. Katz-Bassett, J. Heidemann, and R. Govindan. Mapping the Expansion of Google's Serving Infrastructure. In ACM IMC, 2013.
25. M. Chowdhury, M. Zaharia, J. Ma, M. I. Jordan, and I. Stoica. Managing Data Transfers in Computer Clusters with Orchestra. In ACM SIGCOMM, 2011.
26. M. Chowdhury, Y. Zhong, and I. Stoica. Efficient Coow Scheduling with Varys. In ACM SIGCOMM, 2013.
27. W. W. Chu and P. Hurley. Optimal Query Processing for Distributed Database Systems. IEEE Transactions on Computers, 1982.
28. J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, W. Hsieh, S. Kanthak, E. Kogan, H. Li, A. Lloyd, S. Melnik, D. Mwaura, D. Nagle, S. Quinlan, R. Rao, L. Rolig, Y. Saito, M. Szymaniak, C. Taylor, R. Wang, and D. Woodford. Spanner: Google's Globally-distributed Database. In USENIX OSDI, 2012.
29. J. Dean and S. Ghemawat. Map Reduce: Simplified Data Processing on Large Clusters. Communications of the ACM, 2008.
30. F. R. Dogar, T. Karagiannis, H. Ballani, and A. Rowstron. Decentralized Task-aware Scheduling for Data Center Networks. In ACM SIGCOMM, 2014.
31. A. Elwalid, C. Jin, S. Low, and I. Widjaja. MATE: Multipath Adaptive Traffic Engineering. Computer Networks, 2002.
32. B. Fortz, J. Rexford, and M. Thorup. Traffic Engineering with Traditional IP Routing Protocols. Communications Magazine, IEEE, 2002.
33. R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao, and A. Akella. Multi-Resource Packing for Cluster Schedulers. In ACM SIGCOMM, 2014.
34. A. a. Gupta et al. Mesa: Geo-Replicated, Near Real-Time, Scalable Data Warehousing. In VLDB, 2014.
35. C.-Y. Hong, M. Caesar, and B. Godfrey. Finishing OWS quickly with preemptive scheduling. ACM SIGCOMM, 2012.
36. C.-Y. Hong, S. Kandula, R. Mahajan, M. Zhang, V. Gill, M. Nanduri, and R. Wattenhofer. Achieving High Utilization with Software-Driven WAN. In ACM SIGCOMM, 2013.
37. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: Fair Scheduling for Distributed Computing Clusters. In ACM SOSP, 2009.
38. S. Jain, A. Kumar, S. Mandal, J. Ong, L. Poutievski, A. Singh, S. Venkata, J. Wanderer, J. Zhou, M. Zhu, J. Zolla, U. Holzle, S. Stuart, and A. Vahdat. B4: Experience with a Globally-deployed Software De-ned Wan. ACM SIGCOMM, 2013.
39. S. Kandula, D. Katabi, B. Davie, and A. Charny. Walking the Tightrope: Responsive Yet Stable Trac Engineering. ACM SIGCOMM, 2005.
40. D. Kossmann. The State of the Art in Distributed Query Processing. ACM Computer Survey, 2000.
41. N. Laoutaris, M. Sirivianos, X. Yang, and P. Rodriguez. Inter-datacenter Bulk Transfers with Netstitcher. ACM SIGCOMM, 2011.
42. P. Mohan, A. Thakurta, E. Shi, D. Song, and D. Culler. GUPT: Privacy Preserving Data Analysis Made Easy. In ACM SIGMOD, 2012.
43. E. Nygren, R. Sitaraman, and J. Sun. The Akamai Network: A Platform for High-Performance Internet Applications. In ACM SIGOPS OSR, 2010.
44. M. T. Ozsu and P. Valduriez. Principles of Distributed Database Systems. 2011.
45. A. Pavlo, E. Paulson, A. Rasin, d. Abadi, Daniel and DeWitt, S. Madden, and M. Stonebraker. A Comparison of Approaches to Large-Scale Data Analysis. In ACM SIGMOD, 2009.
46. A. Rabkin, M. Arye, S. Sen, V. Pai, and M. Freedman. Aggregation and Degradation in JetStream: Streaming Analytics in the Wide Area. In USENIX NSDI, 2014.
47. R. Sitaraman, M. Kasbekar, W. Lichtenstein, and M. Jain. Overlay Networks: An Akamai Perspective. In Advanced Content Delivery, Streaming, and Cloud Services, 2014.
48. S. Sundaresan, W. d. Donato, N. Feamster, R. Teixeira, S. Crawford, and A. Pescape. Broadband Internet Performance: A View From the Gateway. In ACM SIGCOMM, 2011.
49. A. Thusoo, J. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Antony, H. Liu, and R. Murthy. Hive-A Petabyte Scale Data Warehouse using Hadoop. In ICDE, 2010.
50. S. Traverso, K. Huguenin, I. Trestian, V. Erramilli, N. Laoutaris, and K. Papagiannaki. TailGate: Handling Long-tail Content with a Little Help from Friends. In WWW, 2012.
51. B. Vamanan, J. Hasan, and T. N. Vijaykumar. Deadline-Aware Datacenter TCP (D2TCP). In Proceedings of the ACM SIGCOMM, 2012.
52. S. Venkataraman, A. Panda, G. Ananthanarayanan, M. Franklin, and I. Stoica. The Power of Choice in Data-Aware Cluster Scheduling. In USENIX OSDI, 2014.
53. A. Vulimiri, C. Curino, B. Godfrey, T. Jungblut, J. Padhye, and G. Varghese. Global Analytics in the Face of Bandwidth and Regulatory Constraints. In USENIX NSDI, 2015.
54. A. Vulimiri, C. Curino, B. Godfrey, K. Karanasos, and G. Varghese. WAN alytics: Analytics for a Geo-distributed Data-intensive World. In CIDR, 2015.
55. C. Wilson, H. Ballani, T. Karagiannis, and A. Rowtron. Better Never Than Late: Meeting Deadlines in Datacenter Networks. ACM SIGCOMM, 2011.
56. Z. Wu, M. Butkiewicz, D. Perkins, E. Katz-Bassett, and H. Madhyastha. SPAN Store: Cost-e-ective Geo-replicated Storage Spanning Multiple Cloud Services. In ACM SOSP, 2013.
57. Y. Yu, P. K. Gunda, and M. Isard. Distributed Aggregation for Data-Parallel Computing: Interfaces and Implementations. In ACM SOSP, 2009.
58. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In ACM EuroSys, 2010.
59. M. Zaharia, M. Chowdhury, M. Franklin, S. Shenker, and I. Stoica. Spark: Cluster Computing with Working Sets. In USENIX HotCloud, 2010.
60. M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, and I. Stoica. Discretized Streams: Fault-Tolerant Streaming Computation at Scale. In ACM SOSP, 2013.
61. M. Zaharia, A. Konwinski, A. Joseph, R. Katz, and I. Stoica. Improving MapReduce performance in heterogeneous environments. In USENIX OSDI, 2008.
62. D. Zats, T. Das, P. Mohan, D. Borthakur, and R. Katz. DeTail: Reducing the Flow Completion Time Tail in Datacenter Networks. ACM SIGCOMM, 2012.