Apache tez: A unifying framework for modeling and building data processing applications

Proceedings of the ACM SIGMOD International Conference on Management of Data

Volumn 2015-May, Issue , 2015, Pages 1357-1369

  Saha, Bikas ^a   Shah, Hitesh ^a   Seth, Siddharth ^a   Vijayaraghavan, Gopal ^a   Murthy, Arun ^a   Curino, Carlo ^b  

^a Hortonworks ^*  (United States)

^b MICROSOFT   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA TRANSFER; ENGINES; FAULT TOLERANCE; SCAFFOLDS; WINDOWS OPERATING SYSTEM; WOOL; YARN;

DATA PROCESSING APPLICATIONS; EXPERIMENTAL EVIDENCE; FUNDAMENTAL FEATURES; OPEN SOURCE FRAMEWORKS; OPEN SOURCE IMPLEMENTATION; PRODUCTION WORKLOADS; RESOURCE MANAGEMENT; RUNTIME OPTIMIZATION;

DATA HANDLING;

EID: 84952798078     PISSN: 07308078     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2723372.2742790     Document Type: Conference Paper

References (38)

1. Understanding just-in-time compilation and optimization. https://docs.oracle.com/cd/E15289-01/doc.40/e15058/underst-jit.htm.
2. Apache tez project website, 2014. http://tez.apache.org.
3. Cascading integration with tez project, 2014. http://www.cascading.org/2014/09/17/cascading-3-0-wip-now-supports-apache-tez.
4. Cascading: Project website, 2014. http://www.cascading.org.
5. Flink: apache incubator project (formerly stratosphere), 2014. http://flink.incubator.apache.org/.
6. Flink integration with tez project jira, 2014. http://issues.apache.org/jira/browse/FLINK-972.
7. Hive integration with tez project jira, 2014. http://issues.apache.org/jira/browse/HIVE-4660.
8. Pig integration with tez project jira, 2014. http://issues.apache.org/jira/browse/PIG-3446.
9. Presto: Distributed sql query engine for big data, 2014. http:/prestodb.io.
10. What's new in the hadoop cluster versions provided by hdinsight?, 2014. http://azure.microsoft.com/en-us/documentation/articles/hdinsight-component-versioning/.
11. New syncsort big data software removes major barriers to mainstream apache hadoop adoption, 2015. http://www.syncsort.com/en/About/News-Center/Press-Release/New-Syncsort-Big-Data-Software.
12. A. Agarwal. https://issues.apache.org/jira/browse/hdfs-5851, 2014. https://issues.apache.org/jira/browse/HDFS-5851.
13. S. Babu. Towards automatic optimization of mapreduce programs. In Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC '10, pages 137-142, New York, NY, USA, 2010. ACM.
14. D. Battré, S. Ewen, F. Hueske, O. Kao, V. Markl, and D. Warneke. Nephele/PACTs: A programming model and execution framework for web-scale analytical processing. In SOCC, 2010.
15. V. Borkar, M. Carey, R. Grover, N. Onose, and R. Vernica. Hyracks: A flexible and extensible foundation for data-intensive computing. In ICDE, 2011.
16. D. Borthakur. Hdfs architecture guide. HADOOP APACHE PROJECT http://hadoop. apache.org/common/docs/current/hdfs design. pdf, 2008.
17. Calcite developer community (formerly Optiq). Calcite: Apache incubator project, 2014. https://github.com/apache/incubator-calcite.
18. H. Choi, J. Son, H. Yang, H. Ryu, B. Lim, S. Kim, and Y. D. Chung. Tajo: A distributed data warehouse system on large clusters. In Data Engineering (ICDE), 2013 IEEE 29th International Conference on, pages 1320-1323. IEEE, 2013.
19. B.-G. Chun, T. Condie, C. Curino, C. Douglas, S. Matusevych, B. Myers, S. Narayanamurthy, R. Ramakrishnan, S. Rao, J. Rosen, et al. Reef: Retainable evaluator execution framework. Proceedings of the VLDB Endowment, 6(12):1370-1373, 2013.
20. Daniel Dai and Rohini Palaniswamy. Pig on Tez: Low latency ETL, 2014. http://www.slideshare.net/Hadoop-Summit/pig-on-tez-low-latency-etl-with-big-data/25.
21. J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. Commun. ACM, 51, 2008.
22. Gopal Vijayaraghavan and Oleg Zhurakousky. Improving Spark for Data Pipelines with Native YARN Integration, 2014. http://bit.ly/1IkZVvP.
23. M. Hausenblas and J. Nadeau. Apache drill: interactive ad-hoc analysis at scale. Big Data, 1(2):100-104, 2013.
24. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. SIGOPS, 2007.
25. M. Kornacker, A. Behm, V. Bittorf, and et al. Impala: A modern, open-source sql engine for hadoop. In CIDR, 2015.
26. S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis. Dremel: Interactive analysis of web-scale datasets. Proc. VLDB Endow., 3(1-2):330-339, Sept. 2010.
27. A. C. Murthy, C. Douglas, M. Konar, O. O'Malley, S. Radia, S. Agarwal, and V. KV. Architecture of next generation apache hadoop mapreduce framework. Technical report, Tech. rep., Apache Hadoop, 2011.
28. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, SIGMOD '08, pages 1099-1110, New York, NY, USA, 2008. ACM.
29. M. Stonebraker, D. Abadi, D. J. DeWitt, S. Madden, E. Paulson, A. Pavlo, and A. Rasin. Mapreduce and parallel dbmss: Friends or foes? Commun. ACM, 53(1):64-71, Jan. 2010.
30. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive - a warehousing solution over a map-reduce framework. In PVLDB, 2009.
31. Twill developer community. Twill: Apache incubator project, 2014. http://twill.incubator.apache.org/.
32. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O'Malley, S. Radia, B. Reed, and E. Baldeschwieler. Apache hadoop yarn: Yet another resource negotiator. In SOCC, 2013.
33. P. Voss. The challenge to choosing the "right" execution engine, 2014. http://bit.ly/1wXP9Wp.
34. A. Vulimiri, C. Curino, B. Godfrey, K. Karanasos, and G. Varghese. Wanalytics: Analytics for a geo-distributed data-intensive world. In CIDR, 2015.
35. Yahoo Hadoop Platforms Team. Yahoo Betting on Apache Hive, Tez, and YARN, 2014. http://yahoodevelopers.tumblr.com/post/85930551108/yahoo-betting-on-apache-hive-tez-and-yarn.
36. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of the 5th European conference on Computer systems, pages 265-278. ACM, 2010.
37. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, pages 2-2. USENIX Association, 2012.
38. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: Cluster computing with working sets. In Proceedings of the 2Nd USENIX Conference on Hot Topics in Cloud Computing, HotCloud'10, pages 10-10, Berkeley, CA, USA, 2010. USENIX Association.