Oozie: Towards a scalable workflow management system for hadoop

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages

  Islam, Mohammad ^a   Huang, Angelo K ^a   Battisha, Mohamed ^a   Chiang, Michelle ^a   Srinivasan, Santhosh ^a   Peters, Craig ^a   Neumann, Andreas ^a   Abdelnur, Alejandro ^b  

^a YAHOO INC   (United States)

^b Cloudera Inc   (United States)

Author keywords

Hadoop; Oozie; Scalability; Workflow management

Indexed keywords

HADOOP; MULTI TENANCIES; OOZIE; PARALLEL COMPUTATION; SCALABILITY AND PERFORMANCE; SCHEDULING SYSTEMS; WORKFLOW MANAGEMENT SYSTEMS; WORKFLOW MANAGEMENTS;

WORK SIMPLIFICATION;

SCALABILITY;

EID: 84875178858     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2443416.2443420     Document Type: Conference Paper

References (26)

1. Apache Hadoop. (n.d.). Retrieved February 24, 2012 from http://hadoop.apache.org/.
2. Apache Oozie. (n.d.). Retrieved February 24, 2012 from http://incubator.apache.org/oozie/.
3. Oracle Big Data Appliance. (n.d.). Retrieved February 10, 2012 from http://www.oracle.com/us/products/database/bigdata-appliance/overview/index. html.
4. Cloudera's Distribution Including Hadoop. (n.d.). Retrieved February 10, 2012 from http://www.cloudera.com/hadoopdetails/.
5. IBM InfoSphere BigInsights Information Center. (n.d.). Retieved February 10, 2012 from http://publib.boulder.ibm.com/infocenter/bigins/v1r3/index.jsp.
6. Deelman, E., Gannon, D., Shields, M., & Taylor, I. May 2009. Workflows and e-Science: An overview of workflow system features and capabilities, Future Generation Computer Systems, Volume 25, Issue 5, May 2009, Pages 528-540.
7. Kepler. (n.d.). Retrieved February 10, 2012 from https://kepler-project. org/.
8. Taverna (n.d.). Retrieved February 10, 2012 from http://edutechwiki. unige.ch/en/Taverna-workbench.
9. Jianwu Wang, Daniel Crawl, Ilkay Altintas. Kepler + Hadoop : A General Architecture Facilitating Data-Intensive Applications in Scientific Workflow Systems. In Proceedings of the 4th Workshop on Workflows in Support of Large- Scale Science (WORKS09) at Supercomputing 2009 (SC2009) Conference, Portland, Oregon, USA, November 14-20, 2009.
10. Gurmeet Singh, Mei-Hui Su, Karan Vahi, Ewa Deelman, Bruce Berriman, John Good, Daniel S. Katz, and Gaurang Mehta. Workflow Task Clustering for Best Effort Systems with Pegasus. Mardi Gras Conference, Baton Rouge, LA, January 2008.
11. Dan Gunter, Ewa Deelman, Taghrid Samak, Christopher Brooks, Monte Goode, Gideon Juve, Gaurang Mehta, Priscilla Moraes, Fabio Silva, Martin Swany, Karan Vahi. Online Workflow Management and Performance Analysis with Stampede, 7th International Conference on Network and Service Management (CNSM-2011), Paris, France, October 2011.
12. Weiwei Chen, Ewa Deelman, Workflow Overhead Analysis and Optimizations, 6th Workshop on Workflows in Support of Large-Scale Science (WORKS 11), Seattle, Washington, November 14th, 2011.
13. The Azkaban Project. (n.d.). Retrieved February 24, 2012 from http://github.com/azkaban/azkaban.
14. Apache Pig. (n.d.). Retrieved February 24, 2012 from http://pig.apache. org.
15. Gates, A. F., Natkovich, O., Chopra, S., Kamath, P., Narayanamurthy, S. M., Olston, C., Reed, B., Srinivasan, S. & Srivastava, U. 2009. Building a high-level dataflow system on top of map-reduce: The Pig experience. In Proc. VLDB.
16. Thusoo, A., Sarma, J., Jain, N., Shao, Z., Chakka, P., Anthony, S., Liu, H., Wyckoff, P. & Murthy, R. 2009. Hive- A Warehousing Solution Over a Map-Reduce Framework, IN VLDB '09: PROCEEDINGS OF THE VLDB ENDOWMENT.
17. The Hive Project. (n.d.). Retrieved February 24, 2012 from http://hadoop.apache.org/hive/.
18. Hunt, P., Konar, M., Junqueira, F. P., & Reed, B. 2010. ZooKeeper: Wait-free coordination for internet-scale systems. In Proc. USENIX Annual Technical Conference.
19. Google App Engine Multitenancy. (n.d.). Retrieved February 10, 2012 from http://code.google.com/appengine/docs/java/multitenancy/overview.html.
20. Chong, F., Carraro, G., & Wolter, R. 2006. Multi-Tenant Data Architecture. Retrieved February 10, 2012 from http://msdn.microsoft.com/en-us/ library/aa479086.aspx.
21. Wang, C. (2009, November 18). Cloud Security Front and Center [Web log comment]. Forrester Research. Retrieved from http://blogs.forrester.com/srm/ 2009/11/cloud-securityfront-and-center.html.
22. Brodkin, J. 2008, July 02. Gartner: Seven cloud-computing security risks. InfoWorld. Retrieved from http://www.infoworld.com/d/security-central/gartner- sevencloud-computing-security-risks-853.
23. Security Guidance for Critical Areas of Focus in Cloud Computing. (n.d.). Cloud Security Alliance. Retrieved February 24, 2012 from https:// cloudsecurityalliance.org/research/projects/securityguidance-for-critical-areas- of-focus-in-cloud-computing/.
24. Zhang, K. 2009. Adding user and service-to-service authentication to Hadoop. Retrieved February 24, 2012 from https://issues.apache.org/jira/browse/ HADOOP.
25. O'Malley, O. 2010. Integrating Kerberos into Apache Hadoop [PDF of PowerPoint slides]. Retrieved from the 2010 Kerberos Conference: Agenda and Slides site http://www.kerberos.org/events/2010conf/2010slides/2010kerberos- owen-omalley.pdf.
26. Secure Impersonation using UserGroupInformation.doAs 2011. (n.d.). Retrieved February 24, 2012 from http://hadoop.apache.org/common/docs/stable/ Secure-Impersonation.html.