Towards high-performance and cost-effective distributed storage systems with information dispersal algorithms

Proceedings - IEEE International Conference on Cluster Computing, ICCC

Volumn , Issue , 2013, Pages

  Zhao, Dongfang ^a   Burlingame, Kent ^a,b   Debains, Corentin ^a   Alvarez Tabio, Pedro ^a   Raicu, Ioan ^a,c  

^a ILLINOIS INSTITUTE OF TECHNOLOGY   (United States)

^b MICROSOFT   (United States)

^c ARGONNE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL CAPACITY; COMPUTING COMMUNITY; DISTRIBUTED FILE SYSTEMS; DISTRIBUTED KEY-VALUE STORES; DISTRIBUTED STORAGE SYSTEM; HIGH PERFORMANCE COMPUTING (HPC); INFORMATION DISPERSAL ALGORITHMS; PARAMETER COMBINATION;

ALGORITHMS; CLOUD COMPUTING; CLUSTER COMPUTING; MULTIPROCESSING SYSTEMS;

DIGITAL STORAGE;

EID: 84893520117     PISSN: 15525244     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLUSTER.2013.6702655     Document Type: Conference Paper

References (39)

1. A. S. Tanenbaum et al., Distributed Systems: Principles and Paradigms, pp. 531-532. Prentice Hall; 2nd edition, 2006.
2. I. Raicu, et al., "The quest for scalable support of data-intensive workloads in distributed systems", in Proceedings of the 18th ACM international symposium on High performance distributed computing, HPDC'09, 2009.
3. Y. Zhao, et al., "Swift: Fast, reliable, loosely coupled parallel computation", in Services, 2007 IEEE Congress on, 2007.
4. M. Wilde, et al., "Extreme-scale scripting: Opportunities for large task parallel applications on petascale computers", in SCIDAC, Journal of Physics: Conference Series 180. DOI, pp. 10-1088, 2009.
5. D. Zhao, et al., "Exploring reliability of exascale systems through simulations", in Proceedings of the High Performance Computing Symposium, HPC ' 13, 2013.
6. Y. Zhao, et al., "Opportunities and challenges in running scientific workflows on the cloud", in Proceedings of the 2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, CYBERC ' 11, 2011.
7. I. Foster, et al., "Cloud computing and grid computing 360-degree compared", in Grid Computing Environments Workshop, 2008. GCE'08, 2008.
8. S. Ghemawat, et al., "The Google file system", in Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP'03, 2003.
9. K. Shvachko, et al., "The Hadoop distributed filesystem: Balancing portability and performance", in Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), 2010.
10. R. Rodrigues et al., "High availability in DHTs: erasure coding vs. replication", in Proceedings of the 4th international conference on Peer-to-Peer Systems, IPTPS'05, 2005.
11. H. Xia et al., "RobuSTore: a distributed storage architecture with robust and high performance", in Proceedings of the 2007 ACMIIEEE conference on Supercomputing, SC'07, 2007.
12. A. Khasymski, et al., "On the use of GPUs in realizing cost-effective distributed RAID", in Proceedings of the 2012 IEEE 20th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS'12, 2012.
13. P. Schwan, "Lustre: Building a file system for 1, 000-node clusters", in Proceedings of the linux symposium, p. 9, 2003.
14. J. Dean et al., "MapReduce: Simplified Data Processing on Large Clusters", in Sixth Symposium on Operating System Design and Implementation (OSDI), 2004.
15. I. Raicu, et al., "Middleware support for many-task computing", Cluster Computing, vol. 13, Sept. 2010.
16. I. Raicu, "Many-task computing: Bridging the gap between high-throughput computing and high-performance computing", Doctoral dissertation, The University of Chicago, 2009.
17. I. Raicu, et al., "Many-task computing for grids and supercomputers", in Many-Task Computing on Grids and Supercomputers, 2008. MTAGS 2008. Workshop on, 2008.
18. P. Bhatotia, et al., "Shredder: GPU-accelerated incremental storage and computation", in Proceedings of the 10th USENIX conference on File and Storage Technologies, FAST' 12, 2012.
19. D. Nagle, et al., "The Panasas ActiveScale storage cluster: Delivering scalable high bandwidth storage", in Proceedings of the 2004 ACMIIEEE conference on Supercomputing, SC'04, 2004.
20. A. Fikes, "Storage architecture and challenges", in Google Faculty Summit, 2010.
21. M. Bjorkqvist, et al., "Dynamic replication in service-oriented systems", in Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012), CCGRID'12, 2012.
22. C. McConnell, et al., "Detouring and replication for fast and reliable internet-scale stream processing", in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC ' 10, 2010.
23. C. L. Abad, et al., "Dare: Adaptive data replication for efficient cluster scheduling", in Proceedings of the 2011 IEEE International Conference on Cluster Computing, CLUSTER ' 11, 2011.
24. Q. Wei, et al., "CORM: A cost-effective dynamic replication management scheme for cloud storage cluster", in Cluster Computing (CLUSTER), 2010 IEEE International Conference on, 2010.
25. T. Li, et al., "ZHT: A light-weight reliable persistent dynamic scalable zero-hop distributed hash table", in IEEE International Parallel & Distributed Processing Symposium, IEEE IPDPS, 13, 2013.
26. D. Zhao et al., "Distributed File System for Exascale Computing", Doctoral Research, Supercomputing (SC ' 12), 2012.
27. D. Zhao, et al., "Distributed data provenance for large-scale data-intensive computing", in IEEE International Conference on Cluster Computing, IEEE CLUSTER ' 13, 2013.
28. D. Zhao et al., "HyCache: a user-level caching middleware for distributed file systems", International Workshop on High Performance Data Intsensive Computing, IEEE IPDPS ' 13, 2013.
29. "Intrepid", https://www.alcf.anl.gov/resource- guideslintrepidfile-systems.
30. Amazon Elastic Compute Cloud (Amazon EC2)", http://aws.amazon. com/ec2/.' '
31. Y. Gu et al., "Supporting configurable congestion control in data transport services", in Proceedings of the 2005 ACM/IEEE conference on Supercomputing, SC'05, 2005.
32. J. S. Plank, et al., "A performance evaluation and examination of open-source erasure coding libraries for storage", in Proccedings of the 7th conference on File and storage technologies, FAST'09, 2009.
33. J. S. Plank, "Jerasure: A library in CIC++ facilitating erasure coding for storage applications", tech. rep., University of Tennessee, 2007.
34. M. L. Curry, et al., "Gibraltar: A Reed-Solomon coding library for storage applications on programmable graphics processors", Concurr. Comput.: Pract. Exper., vol. 23, pp. 2477-2495, Dec. 2011.
35. FUSE Project, "http://fuse.sourceforge.net.' '
36. F. Schmuck et al., "GPFS: A shared-disk file system for large computing clusters", in Proceedings of the 1st USENIX Conference on File and Storage Technologies, FAST'02, 2002.
37. I. F. Haddad, "PVFS: A parallel virtual file system for linux clusters," Linux J., vol. 2000, Nov. 2000.
38. "Intel xeon phi coprocessor 5110p", http://www.intel.com/ contentlwww/us/en/processors/xeonlxeonphi-detail.html.
39. I. Raicu, et al, "Making a case for distributed file systems at exascale", in Proceedings of the third international workshop on Large-scale system and application performance, LSAP, 11, 2011.