Φsched: A heterogeneity-aware hadoop workflow scheduler

Proceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS

Volumn 2015-February, Issue February, 2015, Pages 255-264

  Krish, K R ^a   Anwar, Ali ^a   Butt, Ali R ^a  

^a VIRGINIA POLYTECHNIC INSTITUTE AND STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER HARDWARE; FILE ORGANIZATION;

DESIGN AND IMPLEMENTS; HADOOP DISTRIBUTED FILE SYSTEM (HDFS); HARDWARE CONFIGURATIONS; OPTIMIZED PLACEMENTS; QUANTITATIVE APPROACH; RESOURCE CONSUMPTION; SCHEDULING CONSTRAINTS; TEST APPLICATIONS;

SCHEDULING;

EID: 84937856338     PISSN: 15267539     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/MASCOTS.2014.40     Document Type: Conference Paper

References (40)

1. E. Amazon. Amazon elastic compute cloud (amazon ec2). 2010.
2. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, and I. Stoica. Pacman: Coordinated memory caching for parallel jobs. In Proc. USENIX NSDI, 2012.
3. Apache Software Foundation. Hadoop, 2011. http://hadoop.apache.org/core/.
4. J. Blythe, S. Jain, E. Deelman, Y. Gil, K. Vahi, A. Mandal, and K. Kennedy. Task scheduling strategies for workflow-based applications in grids. In Proc. IEEE CCGrid, 2005.
5. D. Borthakur. The hadoop distributed file system: Architecture and design. Hadoop Project Website, 2007.
6. D. Borthakur. Facebook has the world's largest hadoop cluster!, 2010. http://hadoopblog.blogspot.com/2010/05/facebook-has-worlds-largest-hadoop.html.
7. D. e. Borthakur. Apache hadoop goes realtime at Facebook. Proc. ACM SIGMOD, 2011.
8. Q. Chen, L. Wang, and Z. S hang. Mrgis: A mapreduce-enabled high performance workflow system for gis. In Proc. IEEE International Conference on eScience, 2008.
9. Y. Chen, S. Alspaugh, and R. Katz. Interactive analytical proces sing in big data systems: A cross-industry study of mapreduce workloads. Proc. VLDB Endowment, 5(12):1802-1813, 2012.
10. E. S. Chung, J. D. Davis, and J. Lee. Linqits: Big data on little clients. In Proc. ACM ISCA, 2013.
11. F. Corona. Facebook, 2012. https://gigaom.com/2012/11/08/facebookopen-sources-corona-a-better-way-to-do-webscale-hadoop.
12. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Communications of the ACM, 51(1):107-113, 2008.
13. D. J. DeWitt, E. Paulson, E. Robinson, J. Naughton, J. Royalty, S. Shankar, and A. Krioukov. Clustera: An integrated computation and data management system. Proc. VLDB Endowment, 1(1):28-41, 2008.
14. F. Dong. Extending Starfish to Support the Growing Hadoop Ecosystem. PhD thesis, Duke University, 2012.
15. F. Dong S. G. Akl. Pfas: A resource-performance-fluctuation-aware workflow scheduling algorithm for grid computing. Proc. IEEE IPDPS. 2007.
16. M. Y. Eltabakh, Y. Tian, F. Ö zcan, R. Gemulla, A. Krettek, and J. McPherson. Cohadoop: Flexible data placement and its exploitation in hadoop. Proc. VLDB Endowment, 4(9):575-585, 2011.
17. C. Esterbrook. Using mix-ins with python. Linux Journal, 2001(84es):7, 2001.
18. H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu. Starfish: A self-tuning system for big data analytics. In Proc. CIDR, 2011.
19. S. Huang, J. Huang, Y. Liu, L. Yi, and J. Dai. Hibench: A representative and comprehensive hadoop benchmark suite. In Proc. ICDE Workshops, 2010.
20. M. Islam, A. K. Huang, M. Battisha, M. Chiang, S. Srinivasan, C. Peters, A. Neumann, and A. Abdelnur. Oozie: Towards a scalable workflow management system for hadoop. In Proc. ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies, 2012.
21. Linux man page. sar(1 ), 2013. http://linux.die.net/man/1/sar.
22. N. Maheshwari, R. Nanduri, and V. Varma. Dynamic energy efficient data placement and cluster reconfiguration algorithm for mapreduce framework. Future Generation Computer Systems, 28(1):119-127, 2012.
23. A. Mahout. Mahout, 2010.
24. R. Mantri, R. Ingle, and P. Patil. Scdp: Scalable, cost-effective, distributed and parallel computing model for academics. In Proc. ICECT, 2011.
25. M. Mihailescu, G. Soundararajan, and C. Amza. Mixapart: Decoupled analytics for shared storage systems. In Proc. USENIX HotStorage, 2012.
26. A. Nutch. Nutch, 2010. http://nutch.apache.org.
27. C. Olston, G. Chiou, L. Chitnis, F. Liu, Y. Han, M. Larsson, A. Neumann, V. B. Rao, V. Sankarasubramanian, S. Seth, et al. Nova: Continuous pig/hadoop workflows. In Proc. ACM SIGMOD, 2011.
28. D. Peng and F. Dabek. Large-scale incremental processing using distributed transactions and notifications. In Proc. USENIX OSDI, 2010.
29. S. Radia and S. Srinivas. Scaling hdfs cluster using namenode federation. HDFS-1052, August, 2010.
30. S. Rao, B. Reed, and A. Silberstein. Hotrod: Man aging grid storage with on-demand replication. In Proc. IEEE ICDEW, 2013.
31. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive: A warehousing solution over a mapreduce framework. Proc. VLDB Endowment, 2(2):1626-1629, 2009.
32. C. Wang, X. Li, X. Zhou, Y. Chen, and R. C. Cheung. Big data genome sequencing on zynq based clusters. In Proc. ACM SIGDA, 2014.
33. J. Wang, D. Crawl, and I. Altintas. Kepler + hadoop: A general architecture facilitating data-intensive applications in scientific workflow systems. Proc. WORKS, 2009.
34. Q. Wei, B. Veeravalli, B. Gong, L. Zeng, and D. Feng. Cdrm: A cost-effective dynamic replication management scheme for cloud storage cluster. In Proc. IEEE CLUSTER, 2010.
35. C.Wensel. Cascading: Defining and executing complex and fault tolerant data processing workflows on a hadoop cluster, 2008.
36. T. White. Hadoop The Definitive Guide: The Definitive Guide. O'Reilly Media 2009.
37. J. Xie, S. Yin, X. Ruan, Z. Ding, Y. Tian, J. Majors, A. Manzanares, and X. Qin. Improving mapreduce performance through data placement in heterogeneous hadoop clusters. In Proc. IEEE IPDPSW, 2010.
38. J. Yu and R. Buyya. A taxonomy of workflow management systems for grid computing. Journal of Grid Computing, 3(3-4):171-200, 2005.
39. M. Zaharia, A. Konwinski, A. D. Joseph, R. H. Katz, and I. Stoica. Improving mapreduce performance in heterogeneous environments. In Proc. USENIX OSDI, 2008.
40. C. Zhang and H. De Sterck. Cloudwf: A computational workflow system for clouds based on hadoop. In Cloud Computing, pages 393-404. Springer, 2009.