PigOut: Making multiple Hadoop clusters work together

Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014

Volumn , Issue , 2014, Pages 100-109

  Jeon, Kyungho ^a   Chandrashekhara, Sharath ^a   Shen, Feng ^a   Mehra, Shikhar ^a   Kennedy, Oliver ^a   Ko, Steven Y ^a  

^a UNIVERSITY AT BUFFALO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BENCHMARKING; DATA HANDLING; QUERY LANGUAGES;

AUTOMATED APPROACH; AUTOMATICALLY GENERATED; DATA ANALYSTS; HADOOP MAPREDUCE; HIGH-LEVEL LANGUAGE; HIGHER-LEVEL LANGUAGES; PERFORMANCE; PIG LATINS; SINGLE USERS; WORK-FLOWS;

HIGH LEVEL LANGUAGES;

EID: 84921806676     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/BigData.2014.7004218     Document Type: Conference Paper

References (18)

1. Apache, "Apache Hadoop," http: //hadoop.apache.org, accessed May 27 2014.
2. "Altior's AltraSTAR-Hadoop storage accelerator and optimizer now certified on CDH4 (Cloudera's distribution including Apache Hadoop version 4)," http: //goo.gl/QIUeoX, 2012, retrieved on June 26, 2014.
3. Apache, "Apache Pig: high-level dataflow system for Hadoop," http: //pig.apache.org, accessed May 27, 2014.
4. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig latin: a not-so-foreign language for data processing," in SIGMOD '08.
5. Apache, "Apache Oozie workflow scheduler for Hadoop," http: //oozie. apache.org, accessed May 27, 2014.
6. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz, "Pegasus: a framework for mapping complex scientific workflows onto distributed systems," Sci. Program. vol. 13, no. 3, pp. 219-237, Jul. 2005. [Online]. Available: http: //dl.acm.org/citation.cfm?id=1239649.1239653
7. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones E. A. Lee, J. Tao, and Y. Zhao, "Scientific workflow management and the kepler system: Research articles," Concurr. Comput.: Pract. Exper., vol. 18, no. 10, pp. 1039-1065, Aug. 2006. [Online]. Available: http: //dx.doi.org/10.1002/cpe.v18: 10
8. A. Gates, O. Natkovich, S. Chopra, P. Kamath, S. Narayanam, C. Olston B. Reed, S. Srinivasan, and U. Srivastava, "Building a high level dataflow system on top of mapreduce: The pig experience," PVLDB 2009.
9. Apache, "PigMix," https: //cwiki.apache.org/confluence/display/PIG/ PigMix, accessed June 24, 2014.
10. M. Blasgen, M. Astrahan, D. Chamberlin, J. Gray, W. F. King, B. Lindsay R. Lorie, J. W. Mehl, T. Price, G. R. Putzolu, M. Schkolnick P. Selinger, D. R. Slutz, H. R. Strong, I. L. Traiger, B. Wade, and R. Yost, "System R: An architectural overview," IBM Systems Journal vol. 20, no. 1, pp. 41-62, 1981.
11. Y. Chen, S. Alspaugh, and R. Katz, "Interactive analytical processing in big data systems: A cross-industry study of mapreduce workloads," Proc. VLDB Endow., vol. 5, no. 12, pp. 1802-1813, Aug. 2012. [Online]. Available: http: //dx.doi.org/10.14778/2367502.2367519
12. A. P. Sheth and J. A. Larson, "Federated database systems for managing distributed, heterogeneous, and autonomous databases," ACM Comput. Surv., vol. 22, no. 3, pp. 183-236, Sep. 1990. [Online]. Available: http: //doi.acm.org/10.1145/96602.96604
13. J. Wang, D. Crawl, and I. Altintas, "Kepler + hadoop: A general architecture facilitating data-intensive applications in scientific workflow systems," in WORKS. New York, NY, USA: ACM, 2009. [Online]. Available: http: //doi.acm.org/10.1145/1645164.1645176
14. L. Wang, J. Tao, H. Marten, A. Streit, S. Khan, J. Kolodziej, and D. Chen, "Mapreduce across distributed clusters for data-intensive applications," in IPDPSW PhD Forum, May 2012.
15. P. Kondikoppa, C.-H. Chiu, C. Cui, L. Xue, and S.-J. Park, "Networkaware scheduling of mapreduce framework on distributed clusters over high speed networks," in Proceedings of the 2012 Workshop on Cloud Services, Federation, and the 8th Open Cirrus Summit, ser. FederatedClouds '12. New York, NY, USA: ACM, 2012, pp. 39-44. [Online]. Available: http: //doi.acm.org/10.1145/2378975.2378985
16. S. Y. Ko, K. Jeon, and R. Morales, "The hybrex model for confidentiality and privacy in cloud computing," in Proceedings of the 3rd USENIX Conference on Hot Topics in Cloud Computing, ser. HotCloud'11. Berkeley, CA, USA: USENIX Association, 2011, pp. 8-8. [Online]. Available: http: //dl.acm.org/citation.cfm?id=2170444. 2170452
17. K. Zhang, X. Zhou, Y. Chen, X. Wang, and Y. Ruan, "Sedic: Privacyaware data intensive computing on hybrid clouds," in Proceedings of the 18th ACM Conference on Computer and Communications Security ser. CCS '11. New York, NY, USA: ACM, 2011, pp. 515-526. [Online]. Available: http: //doi.acm.org/10.1145/2046707.2046767
18. C. Zhang, E.-C. Chang, and R. H. Yap, "Tagged-mapreduce: A general framework for secure computing with mixed-sensitivity data on hybrid clouds," in Cluster, Cloud and Grid Computing (CCGrid), 2014 14th IEEE/ACM International Symposium on, May 2014, pp. X-X