Using data accessibility for resource selection in large-scale distributed systems

IEEE Transactions on Parallel and Distributed Systems

Volumn 20, Issue 6, 2009, Pages 788-801

  Kim, Jinoh ^a   Chandra, Abhishek ^a   Weissman, Jon B ^a  

^a University of Minnesota   (United States)

Author keywords

Data Accessibility; Data intensive computing; Large scale distributed systems; Passive network performance estimation; Resource selection

Indexed keywords

COMPUTATION POWER; CONVENTIONAL TECHNIQUES; DATA ACCESS; DATA ACCESSIBILITY; DATA-INTENSIVE COMPUTING; JOB ALLOCATION; LARGE-SCALE DISTRIBUTED SYSTEM; LARGE-SCALE DISTRIBUTED SYSTEMS; LOCAL DATA; NETWORK APPLICATIONS; NUMBER OF DATUM; PLANETLAB; RANDOM ALLOCATION; REMOTE DATA SOURCES; RESOURCE SELECTION; SCALABLE INFRASTRUCTURE; SYNTHETIC EXPERIMENTS;

PASSIVE NETWORKS; SYSTEMS ENGINEERING;

NETWORK PERFORMANCE;

EID: 67349225046     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2009.13     Document Type: Article

References (51)

1. D.P. Anderson and G. Fedak, "The Computational and Storage Potential of Volunteer Computing," Proc. IEEE Int'l Symp. Cluster Computing and the Grid (CCGRID '06), pp. 73-80, 2006.
2. A. Haeberlen, A. Mislove, and P. Druschel, "Glacier: Highly Durable, Decentralized Storage Despite Massive Correlated Failures," Proc. Symp. Networked Systems Design and Implementation (NSDI '05), May 2005.
3. J. Kubiatowicz, D. Bindel, Y. Chen, P. Eaton, D. Geels, R. Gummadi, S. Rhea, H. Weatherspoon, W. Weimer, C. Wells, and B. Zhao, "Oceanstore: An Architecture for Global-Scale Persistent Storage," Proc. ACM Int'l Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS '07), Nov. 2000.
4. R. Bhagwan, K. Tati, Y.-C. Cheng, S. Savage, and G.M. Voelker, "Total Recall: System Support for Automated Availability Management," Proc. Symp. Networked Systems Design and Implementation (NSDI '04), p. 25, 2004.
5. A. Chien, B. Calder, S. Elbert, and K. Bhatia, "Entropia: Architecture and Performance of an Enterprise Desktop Grid System," J. Parallel and Distributed Computing, vol. 63, no. 5 pp. 597-610, 2003.
6. D. Kondo, A.A. Chien, and H. Casanova, "Resource Management for Rapid Application Turnaround on Enterprise Desktop Grids," Proc. ACM/IEEE Conf. Supercomputing (SC '04), p. 17, 2004.
7. J.-S. Kim, B. Nam, P. Keleher, M. Marsh, B. Bhattacharjee, and A. Sussman, "Resource Discovery Techniques in Distributed Desktop Grid Environments," Proc. IEEE/ACM Int'l Conf. Grid Computing (GRID '06) Sept. 2006.
8. D. Zhou and V. Lo, "Cluster Computing on the Fly: Resource Discovery in a Cycle Sharing Peer-to-Peer System," Proc. IEEE Int'l Symp. Cluster Computing and the Grid (CCGRID '04), pp. 66-73, 2004.
9. D.P. Anderson, J. Cobb, E. Korpela, M. Lebofsky, and D. Werthimer, "Setiηome: An Experiment in Public-Resource Computing," Comm. ACM, vol. 45, no. 11, pp. 56-61, 2002.
10. "Search for Extraterrestrial Intelligence (SETI) Project," http://setiathome.berkeley.edu, 2009.
11. "BOINC: Berkeley Open Infrastructure for Network Computing," http://boinc.berkeley.edu/, 2009.
12. "PPDG: Particle Physics Data Grid," http://www.ppdg.net, 2009.
13. N. Massey, T. Aina, M. Allen, C. Christensen, D. Frame, D. Goodman, J. Kettleborough, A. Martin, S. Pascoe, and D. Stainforth, "Data Access and Analysis with Distributed Federated Data Servers in climateprediction.net," Advances in Geosciences, vol. 8, pp. 49-56, June 2006.
14. G.B. Berriman, A.C. Laity, J.C. Good, J.C. Jacob, D.S. Katz, E. Deelman, G. Singh, M.-H. Su, and T.A. Prince, "Montage: The Architecture and Scientific Applications of a National Virtual Observatory Service for Computing Astronomical Image Mosaics," Proc. Earth Sciences Technology Conf., 2006.
15. "BLAST: The Basic Local Alignment Search Tool," http:// www.ncbi.nlm.nih.gov/blast, 2009.
16. W. Hoschek, F.J. Jaén-Martínez, A. Samar, H. Stockinger, and K. Stockinger, "Data Management in an International Data Grid Project," Proc. IEEE/ACM Int'l Conf. Grid Computing (GRID '00 , pp. 77-90, 2000.
17. Y.-M. Teo, X. Wang, and Y.-K. Ng, "Glad: A System for Developing and Deploying Large-Scale Bioinformatics Grid," Bioinformatics, vol. 21, no. 6, pp. 794-802, 2005.
18. S. Hotz, "Routing Information Organization to Support Scalable Interdomain Routing with Heterogeneous Path Requirements," PhD dissertation, 1994.
19. J.D. Guyton and M.F. Schwartz, "Locating Nearby Copies of Replicated Internet Servers," SIGCOMM Computer Comm. Rev., vol. 25, no. 4, pp. 288-298, 1995.
20. E. Ng and H. Zhang, "Predicting Internet Network Distance with Coordiantes-Based Approaches," Proc. IEEE INFOCOM '02, pp. 170-179, 2002.
21. E. Cohen and S. Shenker, "Replication Strategies in Unstructured Peer-to-Peer Networks," Proc. ACM SIGCOMM '02, pp. 177-190, 2002.
22. Q. Lv, P. Cao, E. Cohen, K. Li, and S. Shenker, "Search and Replication in Unstructured Peer-to-Peer Networks," Proc. ACM SIGMETRICS '02, pp. 258-259, 2002.
23. I. Stoica, R. Morris, D. Karger, M.F. Kaashoek, and H. Balakrishnan, "Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications," Proc. ACM SIGCOMM '01, pp. 149-160, 2001.
24. S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Schenker, "A Scalable Content-Addressable Network," Proc. ACM SIGCOMM '01, pp. 161-172, 2001.
25. A. Rowstron and P. Druschel, "Pastry: Scalable, Distributed Object Location and Routing for Large-Scale Peer-to-Peer Systems," Proc. IFIP/ACM Int'l Conf. Distributed Systems Platforms (Middleware '01), pp. 329-350, Nov. 2001.
26. B. Zhao, L. Huang, J. Stribling, S. Rhea, A. Joseph, and J. Kubiatowicz, "Tapestry: A Resilient Global-Scale Overlay for Service Deployment," IEEE J. Selected Areas in Comm., 2003.
27. D. Oppenheimer, J. Albrecht, D. Patterson, and A. Vahdat, "Design and Implementation Tradeoffs for Wide Area Resource Discovery," Proc. Int'l Symp. High Performance Distributed Computing (HPDC), 2005.
28. "PlanetLab," http://www.planet-lab.org, 2009.
29. S.G. Dykes, K.A. Robbins, and C.L. Jeffery, "An Empirical Evaluation of Client-Side Server Selection Algorithms," Proc. IEEE INFOCOM '00, pp. 1361-1370, 2000.
30. R. Wolski, "Experiences with Predicting Resource Performance On-Line in Computational Grid Settings," SIGMETRICS Performance Evaluation Rev. vol. 30, no. 4, pp. 41-49, 2003.
31. K. Lai and M. Baker, "Measuring Bandwidth," Proc. IEEE INFOCOM '99 Mar. 1999.
32. J. Padhye, V. Firoiu, D.F. Towsley, and J.F. Kurose, "Modeling tcp Reno Performance: A Simple Model and Its Empirical Validation," IEEE/ACM Trans. Networking, vol. 8, no. 2, pp. 133-145, 2000.
33. Ö ". B. Akan, "On the Throughput Analysis of Rate-Based and Window-Based Congestion Control Schemes," Computer Networks, vol. 44, no. 5, pp. 701-711, 2004.
34. Y. Chawathe, S. Ratnasamy, L. Breslau, N. Lanham, and S. Shenker, "Making Gnutella Like p2p Systems Scalable," Proc. ACM SIGCOMM '03 pp. 407-418, 2003.
35. S. Seshan, M. Stemm, and R.H. Katz, "SPAND: Shared Passive Network Performance Discovery," Proc. USENIX Symp. Internet Technologies and Systems, pp. 135-146, Dec. 1997.
36. M. Andrews, B. Shepherd, A. Srinivasan, P. Winkler, and F. Zane, "Clustering and Server Selection Using Passive Monitoring," Proc. IEEE INFOCOM '02, pp. 1717-1725, 2002.
37. F. Dabek, R. Cox, F. Kaashoek, and R. Morris, "Vivaldi: A Decentralized Network Coordinate System," Proc. ACM SIGCOMM '04, pp. 15-26, 2004.
38. R. Zhang, C. Tang, Y.C. Hu, S. Fahmy, and X. Lin, "Impact of the Inaccuracy of Distance Prediction Algorithms on Internet Applications - An Analytical and Comparative Study," Proc. IEEE INFOCOM, 2006.
39. L. Tang and M. Crovella, "Virtual Landmarks for the Internet," Proc. Third ACM SIGCOMM Conf. Internet Measurement (IMC '03), pp. 143-152, 2003.
40. "FreePastry," http://freepastry.org, 2009.
41. H. Yu, P.B. Gibbons, and S. Nath, "Availability of Multi-Object Operations," Proc. Symp. Networked Systems Design and Implementation (NSDI), 2006.
42. R. Raman, M. Livny, and M. Solomon, "Matchmaking: Distributed Resource Management for High Throughput Computing," Proc. Int'l Symp. High Performance Distributed Computing (HPDC '98), p. 140, 1998.
43. A.B. Downey, "Using Pathchar to Estimate Internet Link Characteristics," Proc. ACM SIGCOMM '99, pp. 241-250, 1999.
44. S. Keshav, "Packet-Pair Flow Control," IEEE/ACM Trans. Networking 1995.
45. R.L. Carter and M. Crovella, "Server Selection Using Dynamic Path Characterization in Wide Area Networks," Proc. IEEE INFOCOM '97, pp. 1014-1021, 1997.
46. P. Francis, S. Jamin, C. Jin, Y. Jin, D. Raz, Y. Shavitt, and L. Zhang, "Idmaps: A Global Internet Host Distance Estimation Service," IEEE/ACM Trans. Networking, vol. 9, no. 5, pp. 525-540, 2001.
47. B. Wong, A. Slivkins, and E.G. Sirer, "Meridian: A Lightweight Network Location Service without Virtual Coordinates," SIGCOMM Computer Comm. Rev., vol. 35, no. 4, pp. 85-96, 2005.
48. M. Costa, M. Castro, A. Rowstron, and P. Key, "Pic: Practical Internet Coordinates for Distance Estimation," Proc. Int'l Conf. Distributed Systems, 2004.
49. R. Wolski, "Dynamically Forecasting Network Performance Using the Network Weather Service," Cluster Computing, vol. 1, no. 1, pp. 119-132, 1998.
50. Q. He, C. Dovrolis, and M. Ammar, "On the Predictability of Large Transfer tcp Throughput," Proc. ACM SIGCOMM '05, pp. 145-156, 2005.
51. "PlanetLab Iperf," http://jabber.services.planet-lab.org/ php/iperf, 2008.