File grouping for scientific data management: Lessons from experimenting with real traces

Proceedings of the 17th International Symposium on High Performance Distributed Computing 2008, HPDC'08

Volumn , Issue , 2008, Pages 153-163

  Doraimani, Shyamala ^a   Lamnitchi, Adriana ^a  

^a UNIVERSITY OF SOUTH FLORIDA   (United States)

Author keywords

Caching; Data management; File grouping; Job scheduling; Science grids; Trace analysis

Indexed keywords

DIE CASTING; DISTRIBUTED COMPUTER SYSTEMS; HIGH ENERGY PHYSICS; HIGH PERFORMANCE LIQUID CHROMATOGRAPHY; MANAGEMENT INFORMATION SYSTEMS; TRACE ANALYSIS;

CACHING; DATA MANAGEMENT; FILE GROUPING; JOB SCHEDULING; SCIENCE GRIDS;

SCHEDULING;

EID: 57349105027     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1383422.1383429     Document Type: Conference Paper

References (51)

1. The DZero Experiment, http://www-dO.fnal.gov.
2. S. Acharya, B. Smith, and P. Parnes. Characterizing user access to videos on the world wide web. In Proceedings of Multimedia Computing and Networking, 2000.
3. M. Allen and R. Wolski. The Livny and Plank-Beck problems: Studies in data movement on the computational grid. In Supercomputing, 2003.
4. V. Almeida, A. Bestavros, M. Crovella, and A. deOliveira. Characterizing reference locality in the www. Technical report, Boston, MA, USA, 1996.
5. A. Amer, D. D. E. Long, and R. C. Burns. Group-based management of distributed file caches. In ICDCS '02: Proceedings of the 22 nd International Conference on Distributed Computing Systems (ICDCS'02), page 525, Washington, DC, USA, 2002. IEEE Computer Society.
6. A. Amer, D. D. E. Long, and R. C. Burns. Group-based management of distributed file caches. In ICDCS, 2002.
7. M. Arlitt, R. Friedrich, and T. Jin. Workload characterization of a web proxy in a cable modern environment. SIGMETRICS Perform. Eval. Rev., 27(2):25-36, 1999.
8. M. Arlitt and T. Jin. A workload characterization study of the 1998 world cup web site. Technical report, 1999.
9. P. Barford, A. Bestavros, A. Bradley, and M. Crovella. Changes in web client access patterns: Characteristics and caching implications. Technical report, Boston, MA, USA, 1998.
10. A. Bassi, M. Beck, J.-P. Gelas, L. Lefèvre, T. Moore, J. Plank, and P. V.-B. Primet. Active and logistical networking for grid computing: the e-toile architecture. Future Gener. Comput. Syst, 21(l):199-208, 2005.
11. M. Beck, T. Moore, and J. S. Plank. An end-to-end approach to globally scalable network storage. In SIGCOMM '02: Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications, pages 339-346, New York, NY, USA, 2002. ACM Press.
12. A. Bestavros. Demand-based document dissemination to reduce traffic and balance load in distributed information systems. In SPDP '95: Proceedings of the 7th IEEE Symposium on Parallel and Distributeed Processing, page 338, Washington, DC, USA, 1995. IEEE Computer Society.
13. U. Catalyurek, T. Kurc, P. Sadayappan, and J. Saltz. Scheduling file transfers for data-intensive jobs on heterogeneous clusters. In Proceedings of the 13th European Conference on Parallel and Distributed Computing (Europar), 2007.
14. C. Cunha, A. Bestavros, and M. Crovella. Characteristics of www client-based traces. Technical report, Boston, MA, USA, 1995.
15. X. Ding, S. Jiang, F. Chen, K. Davis, and X. Zhang. Diskseen: Exploiting disk layout and access history to enhance I/O prefetch. In USENIX Annual Technical Conference, 2007.
16. S. Doraimani. Filecules: A new granularity for resource management in grids. Master's thesis, University of South Florida, 2007.
17. J. R. Douceur and W. J. Bolosky. A large-scale study of file-system contents. In SIGMETRICS '99: Proceedings of the 1999 ACM SIGMETRICS International conference on Measurement and Modeling of Computer Systems, pages 59-70, New York, NY, USA, 1999. ACM Press.
18. G. R. Ganger and M. F. Kaashoek. Embedded inodes and explicit grouping: Exploiting disk bandwidth for small files. In USENIX Annual Technical Conference, pages 1-17, 1997.
19. S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google file system. In SOSP '03: Proceedings of the nineteenth ACM symposium on Operating systems principles, pages 29-43, New York, NY, USA, 2003. ACM Press.
20. A. S. Gish, Y. Shavitt, and T. Tankel. Geographical statistics and characteristics of p2p query strings. In 1PTPS2007 -Proceedings of the 6th International Workshop on Peer-to-Peer Systems, February 2007.
21. C. Gkantsidis, T. Karagiannis. and M. Vojnovic. Planet scale software updates. In SIGCOMM, pages 423-434, 2006.
22. A. Haeberlen, A. Mislove, A. Post, and P. Druschel. Fallacies in evaluating decentralized systems. In The 5th International Workshop on Peer-to-Peer Systems (IPTPS'06), 2006.
23. A. Iamnitchi, S. Doraimani, and G. Garzoglio. Filecules in high-energy physics: Characteristics and impact on resource management. In HPDC-15: Proceedings of the 15th IEEE International Symposium on High Performance Distributed Computing, pages 69-79, June 2006.
24. A. Iamnitchi and I. Foster. Interest-aware information dissemination in small-world communities. In High Performance Distributed Computing, 2005.
25. A. Iamnitchi and M. Ripeanu. Myth and reality: Usage behavior in a large data-intensive physics project, 2003.
26. A. Iamnitchi, M. Ripeanu, and I. Foster. Small-world file-sharing communities. In Infocom, Hong Kong, China, 2004.
27. A. Iosup, D.H.J.Epema, P. Couvares, A. Karp, and M. Livny. Build-and-test workloads for grid middleware: Problem, analysis, and applications, 2007.
28. A. Iosup and D. Epema. Grenchmark: A framework for analyzing, testing, and comparing grids. In CCGRID '06: Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid. (CCGRID'06), pages 313-320, Washington, DC, USA, 2006. IEEE Computer Society.
29. A. Iosup, M. Jan, O. Sonmez, and D. Epema. The characteristics and performance of groups of jobs in grids. In EuroPar, 2007.
30. A. Iosup, M. Jan, O. Sonmez, and D. Epema. On the dynamic resource availability in grids. In Grid, 2007.
31. M. Karisson and C. Karamanolis. Choosing replica placement heuristics for wide-area systems. In The 24th International Conference on Distributed. Computing Systems (ICDCS), March 2004.
32. G. Khanna, N. Vydyanathan, U. Catalyurek, T. Kurc, S. Krishnamoorthy, P. Sadayappan, and J. Saltz. Task scheduling and file replication for data-intensive jobs with batch-shared I/O. In Proceedings of the 15th IEEE International Symposium on High Performance Distributed. Computing (HPDC), 2006.
33. G. H. Kuenning and G. J. Popek. Automated hoarding for mobile computers. In SOSP '97: Proceedings of the sixteenth A CM symposium on Operating systems principles, pages 264-275, New York, NY, USA, 1997. ACM Press.
34. L. Loebel-Carpenter, L. Lueking, C. Moore, R. Pordes, J. Trumbo, S. Veseli, I. Terekhov, M. Vranicar, S. White, and V. White. Sam and the particle physics data grid. In In Computing in High-Energy and. Nuclear Physics, Beijing, China, 2001.
35. P. Maniatis, M. Roussopoulos, T. J. Giuli, D. S. H. Rosenthal, and M. Baker. The LOCKSS peer-to-peer digital preservation system. ACM Trans. Comput. Syst., 23(l):2-50, 2005.
36. E. Otoo, D. Rotem, and A. Romosan. Optimal file-bundle caching algorithms for data-grids. In SC '04: Proceedings of the 2004 ACM/IEEE conference on Supercomputing, page 6, Washington, DC, USA, 2004. IEEE Computer Society.
37. E. Otoo, D. Rotem, and A. Romosan. Optimal file-bundle caching algorithms for data-grids. In SC '04: Proceedings of the 2004 ACM/IEEE conference on Supercomputing, page 6, Washington, DC, USA, 2004. IEEE Computer Society.
38. E. Otoo, D. Rotem, A. Romosan, and S. Seshadri. File caching in data intensive scientific applications. In Data Management in Grids - Lecture Notes in Computer Science, volume Volume 3836/2006, pages 85-99. Springer Berlin / Heidelberg, 2006.
39. E. Otoo, D. Rotem, and S. Seshadri. Efficient algorithms for multi-file caching. In Database and Expert Systems Applications - Lecture Notes in Computer Science, pages 707-719. Springer Berlin/Heidelberg, 2004.
40. A. Rajasekar, M. Wan, R. Moore, G. Kremenek, and T. Guptil. Data grids, collections, and grid bricks. In MSS '03: Proceedings of the 20 th IEEE/El th NASA Goddard Conference on Mass Storage Systems and Technologies (MSS'03), page 2, Washington, DC, USA, 2003. IEEE Computer Society.
41. A. Rajasekar, M. Wan, R. Moore, and W. Schroeder. A prototype rale-based distributed data management system. In Workshop on Next Generation Distributed Data Management, 2006.
42. K. Ranganathan and I. Foster. Design and evaluation of dynamic replication strategies for a high performance data Grid. In International Conference on Computing in High Energy and Nuclear Physics, 2001.
43. K. Ranganathan and I. Foster. Identifying dynamic replication strategies for a high performance data Grid. In International Workshop on Grid Computing, 2001.
44. K. Ranganathan and I. Foster. Decoupling computation and data scheduling in distributed data-intensive applications. In 11th IEEE International Symposium on High Performance Distributed Computing (HPDC-11), Edinburgh, Scotland, 2002.
45. Y. Saito, C. Karamanolis, M. Karlsson, and M. Mahalingam. Taming aggressive replication in the Pangaea wide-area file system. In OSDI, 2002.
46. E. A. M. Shriver, E. Gabber, L. Huang, and C. A. Stein. Storage management for web proxies. In Proceedings of the General Track: 2002 USENIX Annual Technical Conference, pages 203-216, Berkeley, CA, USA, 2001. USENIX Association.
47. K. Sripanidkulchai. The popularity of gnutella queries and its implications on scalability, 2001.
48. C. Staelin and H. Garcia-Molina. Clustering active disk data to improve disk performance. Technical Report CS-TR-298-90, Princeton, NJ, USA, 1990.
49. C. Staelin and H. Garcia-Molina. Smart filesystems. In USENIX Winter, pages 45-52, 1991.
50. C. D. Tait and D. Duchamp. Detection and exploitation of file working sets. In Proceedings of the 11th International Conference on Distributed Computing Systems (ICDCS), pages 2-9, Washington, DC, 1991. IEEE Computer Society.
51. I. Terekhov. Meta-computing at dO. In In Nuclear Instruments and Methods in Physics Research, Section A, NIMA14225, volume 502/2-3, pages 402-106, 2002.