DARE: Adaptive data replication for efficient cluster scheduling

Proceedings - IEEE International Conference on Cluster Computing, ICCC

Volumn , Issue , 2011, Pages 159-168

  Abad, Cristina L ^a,b   Lu, Yi ^a   Campbell, Roy H ^a  

^a University of Illinois at Urbana Champaign   (United States)

^b ESCUELA SUPERIOR POLITÉCNICA DEL LITORAL   (Ecuador)

Author keywords

locality; MapReduce; replication; scheduling

Indexed keywords

CLUSTER SCHEDULING; DATA LOCALITY; DATA REPLICATION; DATA REPLICATION ALGORITHM; DATA-INTENSIVE SYSTEMS; DELAY SCHEDULING; LOCALITY; MAPREDUCE; NETWORK USAGE; PROBABILISTIC SAMPLING; PRODUCTION SYSTEM; REMOTE DATA ACCESS; REPLICATION;

ADAPTIVE ALGORITHMS; TURNAROUND TIME;

CLUSTER COMPUTING;

EID: 80955123462     PISSN: 15525244     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLUSTER.2011.26     Document Type: Conference Paper

References (33)

1. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in Proc. USENIX Symp. Operating Syst. Design and Implementation (OSDI), 2004, pp. 137-150.
2. "Apache Hadoop," Jun. 2011. [Online]. Available: http://hadoop. apache.org
3. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed data-parallel programs from sequential building blocks,"in Proc. Eur. Conf. Comput. Syst. (EuroSys), 2007, pp. 59-72. (Pubitemid 47281575)
4. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber, "Bigtable: A distributed storage system for structured data," ACM Trans. Comput. Syst., vol. 26, no. 2, 2008.
5. A. Lakshman and P. Malik, "Cassandra: A decentralized structured storage system," SIGOPS Operating Syst. Rev., vol. 44, no. 2, 2010.
6. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris, "Scarlett: Coping with skewed popularity content in MapReduce clusters," in Proc. Eur. Conf. Comput. Syst. (EuroSys), 2011.
7. J. Dixon, "Pentaho, Hadoop, and data lakes," blog, Oct. 2010. [Online]. Available: http://jamesdixon.wordpress.com/2010/ 10/14/pentaho-hadoop- and-data-lakes
8. J. Zuanich, "Twitter analytics lead, Kevin Weil, interviewed,"Cloudera Blog, Sep. 2010. [Online]. Available: http://www.cloudera. com/blog/2010/09
9. "Powered-by - Hadoop Wiki," Jul. 2011. [Online]. Available: http://wiki.apache.org/hadoop/PoweredBy
10. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling," in Proc. Eur. Conf. Comput. Syst. (EuroSys), 2010.
11. M. Satyanarayanan, "A survey of distributed file systems," Annu. Rev. of Comput. Sci., vol. 4, pp. 73-104, 1990.
12. Q. Wei, B. Veeravalli, B. Gong, L. Zeng, and D. Feng, "CDRM: A cost-effective dynamic replication management scheme for cloud storage cluster," in Proc. IEEE Int'l Conf. Cluster Computing (CLUSTER), 2010, pp. 188-196.
13. J. Xiong, J. Li, R. Tang, and Y. Hu, "Improving data availability for a cluster file system through replication," in Proc. IEEE Int'l Symp. Parallel and Distrib. Processing (IPDPS), 2008.
14. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The Hadoop Distributed File System," in Proc. IEEE Symp. Mass Storage Syst. and Technologies (MSST), 2010.
15. D. Ford, F. Labelle, F. Popovici, M. Stokely, V.-A. Truong, L. Barroso, C. Grimes, and S. Quinlan, "Availability in globally distributed storage systems," in Proc. USENIX Symp. Operating Syst. Design and Implementation (OSDI), 2010.
16. J. Terrace and M. J. Freedman, "Object storage on CRAQ: High-throughput chain replication for read-mostly workloads," in Proc. USENIX Annu. Tech. Conf. (USENIX), 2009.
17. M. Tang, B.-S. Lee, X. Tang, and C.-K. Yeo, "The impact of data replication on job scheduling performance in the data grid," Future Generation Comput. Syst., vol. 22, no. 3, 2006.
18. G. Ananthanarayanan, A. Ghodsi, S. Shenker, and I. Stoica, "Disk-locality in datacenter computing considered irrelevant," in Proc. USENIX Workshop on Hot Topics in Operating Syst. (HotOS), 2011.
19. T. Hey, S. Tansley, and K. Tolle, Eds., The Fourth Paradigm: Data- Intensive Scientific Discovery. Microsoft Research, 2009.
20. Y. Lu, B. Prabhakar, and F. Bonomi, "ElephantTrap: A low cost device for identifying large flows," in Proc. IEEE Symp. High-Performance Interconnects, ser. HOTI, 2007.
21. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The Google File System,"in Proc. ACM Symp. Operating Syst. Principles (SOSP), 2003, pp. 29-43.
22. "CCT: Illinois cloud computing testbed," (Last accessed: Jul 7, 2011). [Online]. Available: http://cloud.cs.illinois.edu
23. G. Wang and T. S. E. Ng, "The impact of virtualization on network performance of Amazon EC2 data center," in Proc. Conf. Inf. Commun. (INFOCOM), 2010.
24. "Yahoo! Webscope dataset ydata-hdfs-audit-logs-v1-0," direct distribution, Feb. 2011, http://research.yahoo.com/Academic-Relations.
25. B. Fan, W. Tantisiriroj, L. Xiao, and G. Gibson, "DiskReduce: RAID for data-intensive scalable computing," in Proc. Annu. Workshop Petascale Data Storage (PDSW), 2009, pp. 6-10.
26. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica, "Improving mapreduce performance in heterogeneous environments,"in Proc. USENIX Symp. Operating Syst. Design and Implementation (OSDI), 2008, pp. 29-42.
27. Y. Chen, A. Ganapathi, R. Griffith, and R. Katz, "The case for evaluating MapReduce performance using workload suites," in Proc. IEEE Int'l Symp. Modeling, Analysis & Simulation of Comput. Telecomm. Syst. (MASCOTS), 2011.
28. W. Cirne and F. Berman, "When the herd is smart: Aggregate behavior in the selection of job request," IEEE Trans. Parallel Distrib. Syst., vol. 14, no. 2, 2003.
29. D. Feitelson and L. Rudolph, "Metrics and benchmarking for parallel job scheduling," Job Scheduling Strategies for Parallel Processing, LNCS, vol. 1459, pp. 1-24, 1998. (Pubitemid 128119028)
30. S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken, "The nature of data center traffic: Measurements & analysis," in Proc. ACM Conf. Internet Measurement (IMC), 2009.
31. P. Mahadevan, S. Banerjee, and P. Sharma, "Energy proportionality of an enterprise network," in Proc. ACM Workshop on Green Netw., 2010, pp. 53-60.
32. R. Bolla, R. Bruschi, F. Davoli, and F. Cucchietti, "Energy efficiency in the future Internet: A survey of existing approaches and trends in energy-aware fixed network infrastructures," IEEE Commun. Surveys Tuts., vol. 13, no. 2, pp. 223-244, 2011.
33. T. Loukopoulos and I. Ahmad, "Static and adaptive data replication algorithms for fast information access in large distributed systems,"in Proc. Int'l Conf. Distrib. Comput. Syst. (ICDCS), 2000.