CLARINET: WAN-aware optimization for analytics queries

Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016

Volumn , Issue , 2016, Pages 435-450

  Viswanathan, Raajay ^a   Ananthanarayanan, Ganesh ^b   Akella, Aditya ^a  

^a UNIVERSITY OF WISCONSIN MADISON   (United States)

^b MICROSOFT   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DIGITAL STORAGE; OPTIMIZATION; RESPONSE TIME (COMPUTER SYSTEMS); SCHEDULING; SYSTEMS ANALYSIS;

DATA CENTER NETWORKS; DISTRIBUTED DATA; DISTRIBUTED SITES; HEURISTIC SOLUTIONS; LARGE SCALE SIMULATIONS; MANAGEMENT DECISIONS; PRODUCTION WORKLOADS; QUERY EXECUTION PLAN;

WIDE AREA NETWORKS;

EID: 85076983760     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (49)

1. Amazon datacenter locations. https://aws.amazon.com/about-aws/global-infrastructure/.
2. Apache Calcite - a dynamic data management framework. http://calcite.incubator.apache.org. Accessed 04-27-2015.
3. Apache Hive. http://hive.apache.org.
4. Apache Tez. http://tez.apache.org.
5. Google datacenter locations. http://www.google.com/about/datacenters/inside/locations/.
6. Microsoft datacenters. http://www.microsoft.com/en-us/server-cloud/cloud-os/global-datacenters.aspx.
7. Spark SQL. https://spark.apache.org/sql.
8. TPC Benchmark DS (TPC-DS). http://www.tpc.org/tpcds.
9. AGARWAL, S., KANDULA, S., BRUNO, N., WU, M.-C., STOICA, I., and ZHOU, J. Reoptimizing data parallel computing. In NSDI (2012).
10. ARMBRUST, M., and XIN, R. S., LIAN, C., HUAI, Y., LIU, D., and BRADLEY, J. K., MENG, X., KAFTAN, T., FRANKLIN, M. J., GHODSI, A., and ZAHARIA, M. Spark SQL: Relational data processing in Spark. In SIGMOD (2015).
11. AVNUR, R., and HELLERSTEIN, J. M. Eddies: Continuously adaptive query processing. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (New York, NY, USA, 2000), SIGMOD '00, ACM, pp. 261-272.
12. BERNSTEIN, P. A., and CHIU, D.-M. W. Using semi-joins to solve relational queries. Journal of the ACM 28, 1 (1981), 25-40.
13. CALDER, M., FAN, X., HU, Z., KATZ-BASSETT, E., HEIDEMANN, J., and GOVINDAN, R. Mapping the expansion of Google's serving infrastructure. In IMC (2013).
14. DEWITT, D. J., GHANDEHARIZADEH, S., SCHNEIDER, D., BRICKER, A., HSIAO, H.-I., RASMUSSEN, R., ET AL The Gamma database machine project. IEEE Transactions on Knowledge and Data Engineering 2, 1 (1990), 44-62.
15. GANJAM, A., SIDDIQUI, F., ZHAN, J., LIU, X., STOICA, I., JIANG, J., SEKAR, V., and ZHANG, H. C3: Internet-scale control plane for video quality optimization. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15) (Oakland, CA, May 2015), USENIX Association, pp. 131-144.
16. GRAEFE, G. Volcano: An extensible and parallel query evaluation system. IEEE Trans. on Knowl. and Data Eng. 6, 1 (Feb. 1994), 120-135.
17. GRAEFE, G. The cascades framework for query optimization. Data Engineering Bulletin 18 (1995).
18. GRANDL, R., ANANTHANARAYANAN, G., KANDULA, S., RAO, S., and AKELLA, A. Multi-resource packing for cluster schedulers. In SIGCOMM (2014).
19. GUPTA, A., SUDARSHAN, S., and VISHWANATHAN, S. Query scheduling in multi query optimization. In Database Engineering and Applications, 2001 International Symposium on. (2001), IEEE, pp. 11-19.
20. HONG, C.-Y., KANDULA, S., MAHAJAN, R., ZHANG, M., GILL, V., NANDURI, M., and WATTENHOFER, R. Achieving high utilization with software-driven WAN. In SIGCOMM (2013).
21. HUNG, C.-C., GOLUBCHIK, L., and YU, M. Scheduling jobs across geo-distributed datacenters. In SoCC (2015).
22. ISARD, M., PRABHAKARAN, V., CURREY, J., WIEDER, U., TALWAR, K., and GOLDBERG, A. Quincy: Fair scheduling for distributed computing clusters. In SOSP (2009).
23. JAIN, S., KUMAR, A., MANDAL, S., ONG, J., POUTIEVSKI, L., SINGH, A., VENKATA, S., WANDERER, J., ZHOU, J., ZHU, M., ZOLLA, J., HÖLZLE, U., STUART, S., and VAHDAT, A. B4: Experience with a globally-deployed software defined WAN. In SIGCOMM (2013).
24. JIANG, J., DAS, R., ANANTHANARAYANAN, G., CHOU, P., PADMANABHAN, V., SEKAR, V., DOMINIQUE, E., GOLISZEWSKI, M., KUKOLECA, D., VAFIN, R., and ZHANG, H. Via: Improving internet telephony call quality using predictive relay selection. In SIGCOMM (2015).
25. KITSUREGAWA, M., TANAKA, H., and MOTOOKA, T. Application of hash to data base machine and its architecture. New Generation Computing 1, 1 (1983), 63-74.
26. KUMAR, A., JAIN, S., NAIK, U., RAGHURAMAN, A., KASINADHUNI, N., ZERMENO, E. C., and GUNN, C. S., AI, J., CARLIN, B., AMARANDEI-STAVILA, M., ROBIN, M., SIGANPORIA, A., STUART, S., and VAHDAT, A. BwE: Flexible, hierarchical bandwidth allocation for WAN distributed computing. In SIGCOMM (2015).
27. LI, H., GHODSI, A., ZAHARIA, M., SHENKER, S., and STOICA, I. Tachyon: Reliable, memory speed storage for cluster computing frameworks. In Proceedings of the ACM Symposium on Cloud Computing (New York, NY, USA, 2014), SOCC '14, ACM, pp. 6:1-6:15.
28. ∗ optimizer validation and performance evaluation for distributed queries. In PVLDB (1986).
29. MARKL, V., RAMAN, V., SIMMEN, D., LOHMAN, G., PIRAHESH, H., and CILIMDZIC, M. Robust query processing through progressive optimization. In Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data (New York, NY, USA, 2004), SIGMOD '04, ACM, pp. 659-670.
30. MASTROLILLI, M., and SVENSSON, O. (acyclic) job shops are hard to approximate. In FOCS (2008).
31. MONALDO, M., and OLA, S. Improved bounds for flow shop scheduling. In ICALP (2009).
32. MULLIN, J. K. Optimal semijoins for distributed database systems. IEEE Transactions on Software Engineering 16, 5 (1990), 558-560.
33. OLSTON, C., REED, B., SRIVASTAVA, U., KUMAR, R., and TOMKINS, A. Pig latin: A not-so-foreign language for data processing. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (New York, NY, USA, 2008), SIGMOD '08, ACM, pp. 1099-1110.
34. POLYCHRONIOU, O., SEN, R., and ROSS, K. A. Track join: distributed joins with minimal network traffic. In SIGMOD (2014).
35. PU, Q., ANANTHANARAYANAN, G., BODIK, P., KANDULA, S., AKELLA, A., BAHL, V., and STOICA, I. Low latency geo-distributed data analytics. In SIGCOMM (2015).
36. RABKIN, A., ARYE, M., SEN, S., PAI, V. S., and FREEDMAN, M. J. Aggregation and degradation in JetStream: Streaming analytics in the wide area. In NSDI (2014).
37. REN, X., ANANTHANARAYANAN, G., WIERMAN, A., and YU, M. Hopper: Decentralized speculation-aware cluster scheduling at scale. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication (New York, NY, USA, 2015), SIGCOMM '15, ACM, pp. 379-392.
38. RODIGER, W., MUHLBAUER, T., UNTERBRUNNER, P., REISER, A., KEMPER, A., and NEUMANN, T. Locality-sensitive operators for parallel main-memory database clusters. In ICDE (2014).
39. ROY, P., SESHADRI, S., SUDARSHAN, S., and BHOBE, S. Efficient and extensible algorithms for multi query optimization. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (New York, NY, USA, 2000), SIGMOD '00, ACM, pp. 249-260.
40. SELLIS, T. K. Multiple-query optimization. ACM Trans. Database Syst. 13, 1 (Mar. 1988), 23-52.
41. URHAN, T., and FRANKLIN, M. J. XJoin: A reactively-scheduled pipelined join operator. IEEE Data Engineering Bulletin (2000), 27-33.
42. VISWANATHAN, R., ANANTHANARAYANAN, G., and AKELLA, A. Clarinet: Wan-aware optimization for analytics queries. Tech. Rep. TR1841, University of Wisconsin-Madison, 2016.
43. VULIMIRI, A., CURINO, C., GODFREY, B., PADHYE, J., and VARGHESE, G. Global analytics in the face of bandwidth and regulatory constraints. In NSDI (2015).
44. WANG, X., OLSTON, C., SARMA, A. D., and BURNS, R. Coscan: Cooperative scan sharing in the cloud. In Proceedings of the 2Nd ACM Symposium on Cloud Computing (2011), SOCC '11.
45. XIAO, X., HANNAN, A., BAILEY, B., and NI, L. M. Traffic engineering with mpls in the internet. Network, IEEE 14, 2 (2000), 28-33.
46. XIN, R. S., ROSEN, J., ZAHARIA, M., FRANKLIN, M. J., SHENKER, S., and STOICA, I. Shark: SQL and rich analytics at scale. In SIGMOD (2013).
47. XIONG, P., HACIGUMUS, H., and NAUGHTON, J. F. A software-defined networking based approach for performance management of analytical queries on distributed data stores. In SIGMOD (2014).
48. ZAHARIA, M., CHOWDHURY, M., DAS, T., DAVE, A., MA, J., MCCAULEY, M., FRANKLIN, M., SHENKER, S., and STOICA, I. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI (2012).
49. ZAMANIAN, E., BINNIG, C., and SALAMA, A. Locality-aware partitioning in parallel database systems. In SIGMOD (2015).