Distributed data provenance for large-scale data-intensive computing

Proceedings - IEEE International Conference on Cluster Computing, ICCC

Volumn , Issue , 2013, Pages

  Zhao, Dongfang ^a   Shou, Chen ^a   Maliky, Tanu ^b,c   Raicu, Ioan ^a,c  

^a Illinois Institute of Technology   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

^c ARGONNE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA-INTENSIVE COMPUTING; DESIGN OPTIMALITY; DISTRIBUTED DATA; DISTRIBUTED HASH TABLE; DISTRIBUTED MODULES; METADATA MANAGEMENT; METADATA STORAGES; PARALLEL FILE SYSTEM;

CLUSTER COMPUTING; DIGITAL STORAGE; DISTRIBUTED DATABASE SYSTEMS; FILE ORGANIZATION; INFORMATION MANAGEMENT; QUERY PROCESSING; SUPERCOMPUTERS;

METADATA;

EID: 84893605665     PISSN: 15525244     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLUSTER.2013.6702685     Document Type: Conference Paper

References (42)

1. P. A. Freeman, et al., "Cyberinfrastructure for science and engineering: Promises and challenges", Proceedings of the IEEE, vol. 93, no. 3, pp. 682-691, 2005.
2. K.-L. Ma, et al., "In-situ processing and visualization for ultrascale simulations", in Journal of Physics: Conference Series, vol. 78, p. 012043, IOP Publishing, 2007.
3. K.-L. Ma, "In situ visualization at extreme scale: Challenges and opportunities", Computer Graphics and Applications, IEEE, vol. 29, no. 6, pp. 14-19, 2009.
4. S. Ghemawat, et al., "The google file system", in Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP'03, 2003.
5. N. Ali, et al, "Scalable i/o forwarding framework for highperformance computing systems", in Cluster Computing and Workshops, 2009. CLUSTER'09. IEEE International Conference on, pp. 1-10, IEEE, 2009.
6. I. Raicu, et al., "Middleware support for many-task computing", Cluster Computing, vol. 13, Sept. 2010.
7. I. Raicu, "Many-task computing: Bridging the gap between high-throughput computing and high-performance computing", Doctoral dissertation, The University of Chicago, 2009.
8. K. Shvachko, et al., "The hadoop distributed file system", in Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), MSST ' 10, 2010.
9. I. Stoica, et al, "Chord: Scalable Peer-to-peer Lookup Service for Internet Applications", in Proceedings of the 2001 conference on applications, technologies, architectures, and protocols for computer communications (ACM SIGCOMM), 2001.
10. I. Raicu, et al, "Making a case for distributed file systems at exascale", in Proceedings of the third international workshop on Large-scale system and application performance, LSAP ' 11, 2011.
11. D. Zhao, et al., "Exploring reliability of exascale systems through simulations", in Proceedings of the High Performance Computing Symposium, HPC ' 13, 2013.
12. I. Raicu, et al., "Toward loosely coupled programming on petascale systems", in Proceedings of the 2008 ACM/IEEE conference on Supercomputing, SC'08, 2008.
13. Z. Zhang, et al., "Design and evaluation of a collective i/o model for loosely-coupled petascale programming", IEEE MTAGS, 2008.
14. T. Malik, et al., "Sketching distributed data provenance", in Data Provenance and Data Management in eScience, pp. 85-107, 2013.
15. D. Zhao et al., "Distributed File Systems for Exascale Computing", ACM/IEEE Supercomputing (Doctoral Showcase), 2012.
16. A. Gehani et al., "SPADE: Support for Provenance Auditing in Distributed Environments", ACM/USENIX Middleware, pp. 101-120, 2012.
17. C. Shou, et al., "Towards a provenance-aware distributed flesystem", in 5th Workshop on the Theory and Practice of Provenance, 2013.
18. T. Li, et al., "ZHT: A light-weight reliable persistent dynamic scalable zero-hop distributed hash table", in IEEE International Parallel & Distributed Processing Symposium, IEEE IPDPS, 13, 2013.
19. K.-K. Muniswamy-Reddy, "Foundations for provenance-aware systems", Doctoral dissertation, Harvard University, 2010.
20. I. T. Foster, et al, "The virtual data grid: A new model and architecture for data-intensive collaboration", in CIDR'03, pp. 1-1, 2003.
21. Provenance aware service oriented architecture, "http://twiki.pasoa. ecs.soton. ac.uk/bin/view/pasoa/webhome"
22. A. Parker-Wood, et al., "Making sense of file systems through provenance and rich metadata", Tech. Rep. UCSC-SSRC-12-01, University of California, Santa Cruz, Mar. 2012.
23. K.-K. Muniswamy-Reddy, et al, "Provenance-aware storage systems", in Proceedings of the annual conference on USENIX'06 Annual Technical Conference, ATEC'06, 2006.
24. K.-K. Muniswamy-Reddy, et al., "Making a cloud provenanceaware" , in 1st Workshop on the Theory and Practice of Provenance, 2009.
25. K.-K. Muniswamy-Reddy, et al., "Layering in provenance systems", in Proceedings of the 2009 USENIX Annual Technical Conference, 2009.
26. W. Zhou, et al., "Efficient querying and maintenance of network provenance at internet-scale", in Proceedings of the 2010 international conference on Management of data, pp. 615-626, ACM, 2010.
27. J. Abraham, et al., "Distributed storage and querying techniques for a semantic web of scientific workflow provenance", in Services Computing (SCC), 2010 IEEE International Conference on, pp. 178-185, IEEE, 2010.
28. T Heinis et al., "Efficient lineage tracking for scientific workfows", in Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1007-1018, ACM, 2008.
29. FUSE Project, "http://fuse.sourceforge.net."
30. I. Raicu, et al., "The quest for scalable support of data-intensive workloads in distributed systems", in Proceedings of the 18th ACM international symposium on High performance distributed computing, HPDC'09, 2009.
31. K. Wang, et al., "Using simulation to explore distributed keyvalue stores for exascale systems services", in Proceedings of the 2013 ACMIIEEE conference on Supercomputing, SC ' 13, 2013.
32. D. Zhao, et al., "Towards high-performance and cost-effective distributed storage systems with information dispersal algorithms", in IEEE International Conference on Cluster Computing, IEEE CLUSTER ' 13, 2013.
33. D. Zhao et al., "HyCache: a user-level caching middleware for distributed file systems", International Workshop on High Performance Data Intsensive Computing, IEEE IPDPS ' 13, 2013.
34. Intrepid https://www.alcf.anl.gov/resource-guideslintrepid-filesystems.
35. Amazon Elastic Compute Cloud (Amazon EC2)", http://aws.amazon. com/ec2/"
36. B. Fitzpatrick, "Distributed caching with memcached", Lima J., vol. 2004, pp. 5-, Aug. 2004.
37. G. De Candia, et al., "Dynamo: amazon's highly available key-value store", in Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, SOSP'07, 2007.
38. Y. Gu et al, "Supporting configurable congestion control in data transport services", in Proceedings of the 2005 ACMIIEEE conference on Supercomputing, SC 'O5, 2005.
39. D. Capps, "IOzone Filesystem Benchmark", www.iozone.org, 2008.
40. Y. Zhao, et al., "Swift: Fast, reliable, loosely coupled parallel computation", in IEEE SCW, pp. 199-206, 2007.
41. M. Wilde, et al., "Extreme-scale scripting: Opportunities for large task parallel applications on petascale computers", in SCIDAC, Journal of Physics: Conference Series 180. DOI, pp. 10-1055, 2009.
42. Y Zhao, et al., "Opportunities and challenges in running scientific workflows on the cloud", in Proceedings of the 2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, CYBERC ' 11, 2011.