A taxonomy of market-based resource management systems for utility-driven cluster computing

Software - Practice and Experience

Volumn 36, Issue 13, 2006, Pages 1381-1419

  Yeo, Chee Shin ^a   Buyy, Rajkumar ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Cluster computing; Market based resource management systems; Taxonomy; Utility computing

Indexed keywords

COMPUTATION THEORY; DISTRIBUTED COMPUTER SYSTEMS; MARKETING; RESOURCE ALLOCATION;

CLUSTER COMPUTING; MARKET-BASED RESOURCE MANAGEMENT SYSTEMS; TAXONOMY; UTILITY COMPUTING;

INFORMATION MANAGEMENT;

EID: 33750365790     PISSN: 00380644     EISSN: 1097024X     Source Type: Journal    
DOI: 10.1002/spe.725     Document Type: Article

References (79)

1. Gropp W, Lusk E, Sterling T (eds.). Beowulf Cluster Computing with Linux (2nd edn). MIT Press: Cambridge, MA, 2003.
2. Pfister GF. In Search of Clusters (2nd edn). Prentice-Hall PTR: Upper Saddle River, NJ, 1998.
3. Buyya R (ed.). High Performance Cluster Computing: Architectures and Systems. Prentice-Hall PTR: Upper Saddle River, NJ, 1999.
4. Buyya R, Cortes T, Jin H. Single system image. The International Journal of High Performance Computing Applications 2001; 15(2): 124-135.
5. University of Wisconsin-Madison. Condor version 6.7.1 manual. http://www.cs.wisc.edu/condor/manual/v6.? [6 December 2004].
6. IBM. LoadLeveler for AIX 5L version 3.2 using and administering, October 2003. http://www-1.ibm.com/servers/eserver/pseries/library/sp_books/loadleveler. html [6 December 2004].
7. Platform Computing. LSF version 4.1 administrator's guide, 2001. http://www.platform.com/services/support [6 December 2004].
8. Altair Grid Technologies. OpenPBS release 2.3 administrator guide, August 2000. http://www.openpbs.org/docs.html [6 December 2004].
9. Sun Microsystems. Sun ONE Grid Engine, administration and user's guide, October 2002. http://gridengine.sunsource.net/project/gridengine/documentation. html [6 December 2004].
10. Foster I, Kesselman C (eds.). The Grid 2: Blueprint for a New Computing Infrastructure. Morgan Kaufmann: San Francisco, CA, 2003.
11. Buyya R, Abramson D, Giddy J. A case for economy Grid architecture for service oriented Grid computing. Proceedings of the 10th International Heterogeneous Computing Workshop (HCW 2001), San Francisco, CA, April 2001. IEEE Computer Society Press: Los Alamitos, CA, 2001.
12. Buyya R, Abramson D, Giddy J. Nimrod/G: An architecture for a resource management and scheduling system in a global Computational Grid. Proceedings of the 4th International Conference on High Performance Computing in Asia-Pacific Region (HPC Asia 2000), Beijing, China, May 2000. IEEE Computer Society Press: Los Alamitos, CA, 2000.
13. Venugopal S, Buyya R, Winton L. A Grid service broker for scheduling distributed data-oriented applications on global Grids. Proceedings of the 2nd International Workshop on Middleware for Grid Computing (MGC 2004), Toronto, Canada, October 2004. ACM Press: New York, NY, 2004; 75-80.
14. Yu J, Buyya R. A novel architecture for realizing Grid workflow using Tuple Spaces. Proceedings of the 5th International Workshop on Grid Computing (GRID 2004), Pittsburgh, PA, November 2004. IEEE Computer Society Press: Los Alamitos, CA, 2004; 119-128.
15. The TeraGrid project. http://www.teragrid.org [6 December 2004].
16. The LHC computing Grid project. http://lcg.web.cern.ch/LCG [6 December 2004].
17. The NAREGI project. http://www.naregi.org [6 December 2004].
18. The APAC Grid project. http://www.apac.edu.au [6 December 2004].
19. IBM Grid computing. http://www.ibm.com/grid [6 December 2004].
20. HP Grid computing. http://www.hp.com/techservers/grid [6 December 2004].
21. Sun Microsystems utility computing. http://www.sun.com/service/utility [6 December 2004].
22. Buyya R, Abramson D, Venugopal S. The Grid economy. Proceedings of the IEEE 2005; 93(3):698-714.
23. Chun BN, Culler DE. Market-based proportional resource sharing for clusters. Technical Report CSD-1092, Computer Science Division, University of California at Berkeley, January 2000.
24. Sherwani J, Ali N, Lotia N, Hayat Z. Buyya R. Libra: A computational economy-based job scheduling system for clusters. Software: Practice and Experience 2004; 34(6):573-590.
25. Irwin DE, Grit LE, Chase JS. Balancing risk and reward in a market-based task service. Proceedings of the 13th International Symposium on High Performance Distributed Computing (HPDC13), Honolulu, HI, June 2004. IEEE Computer Society Press: Los Alamitos, CA, 2004; 160-169.
26. Anastasiadi A, Kapidakis S, Nikolaou C, Sairamesh J. A computational economy for dynamic load balancing and data replication. Proceedings of the 1st International Conference on Information and Computation Economies (ICE '98), Charleston, SC, October 1998. ACM Press: New York, NY, 1998; 166-180.
27. Stonebraker M, Devine R, Kornacker M, Litwin W, Pfeffer A, Sah A, Staelin C. An economic paradigm for query processing and data migration in Mariposa. Proceedings of the 3rd International Conference on Parallel and Distributed Information Systems (PDIS '94), Austin, TX. September 1994. IEEE Computer Society Press: Los Alamitos, CA, 1994; 58-68.
28. Abramson D, Buyya R. Giddy J. A computational economy for Grid computing and its implementation in the Nimrod-G resource broker. Future Generation Computer Systems 2002; 18(8): 1061-1074.
29. Kale LV, Kumar S, Potnuru M, DeSouza J, Bandhakavi S. Faucets: Efficient resource allocation on the Computational Grid. Proceedings of the International Conference on Parallel Processing (ICPP 2004), Montreal, Canada, August 2004. IEEE Computer Society Press: Los Alamitos, CA, 2004; 396-405.
30. Lai K, Huberman BA, Fine L. Tycoon: A distributed market-based resource allocation system. Technical Report cs.DC/0404013, HP Lab, Palo Alto, April 2004.
31. Malone TW, Fikes RE, Grant KR, Howard MT. Enterprise: A market-like task scheduler for distributed computing environments. The Ecology of Computation, Huberman BA (ed.). Elsevier Science Publisher: New York, NY, 1988; 177-205.
32. Waldspurger CA, Hogg T, Huberman BA, Kephart JO, Stornetta WS. Spawn: A distributed computational economy. IEEE Transactions on Software Engineering 1992; 18(2):103-117.
33. Bredin J, Kotz D, Rus D. Utility driven mobile-agent scheduling. Technical Report PCS-TR98-331, Department of Computer Science, Dartmouth College, October 1998.
34. Cooper BF, Garcia-Molina H. Bidding for storage space in a peer-to-peer data preservation system. Proceedings of the 22nd International Conference on Distributed Computing Systems (ICDCS 2002), Vienna, Austria, July 2002. IEEE Computer Society Press: Los Alamitos, CA, 2002; 372-381.
35. Alexandrov AD, Ibel M, Schauser KE, Scheiman CJ. SuperWeb: Research issues in Java-based global computing. Concurrency: Practice and Experience 1997; 9(6):535-553.
36. Regev O, Nisan N. The POPCORN market - an online market for computational resources. Proceedings of the 1st International Conference on Information and Computation Economies (ICE '98), Charleston, SC, October 1998. ACM Press: New York, NY, 1998; 148-157.
37. Lalis S, Karipidis A. JaWS: An open market-based framework for distributed computing over the Internet. Proceedings of the 1st International Workshop on Grid Computing (GRID 2000), Bangalore, India, December 2000 (Lecture Notes in Computer Science, vol. 1971). Springer: Heidelberg, Germany, 2000; 36-46.
38. Casavant TL, Kuhl JG. A taxonomy of scheduling in general-purpose distributed computing systems. IEEE Transactions on Software Engineering 1988; 14(2): 141-154.
39. Rotithor HG. Taxonomy of dynamic task scheduling schemes in distributed computing systems. IEEE Proceedings of Computers and Digital Techniques 1994; 141(1): 1-10.
40. 3: A taxonomy of heterogeneous computing systems. IEEE Computer 1995; 28(12):68-70.
41. Ekmecić I, Tartalja I, Milutinović V. A survey of heterogeneous computing: Concepts and systems. Proceedings of the IEEE 1996; 84(8): 1127-1144.
42. Braun TD, Siegel HJ, Beck N, Bölöni L, Maheswaran M, Reuther AI, Robertson JP, Theys MD, Yao B. A taxonomy for describing matching and scheduling heuristics for mixed-machine heterogeneous computing systems. Proceedings of the 17th Symposium on Reliable Distributed Systems, West Lafayette, IN, October 1998. IEEE Computer Society Press: Los Alamitos, CA, 1998; 330-335.
43. Krauter K, Buyya R, Maheswaran M. A taxonomy and survey of Grid resource management systems for distributed computing. Software: Practice and Experience 2002; 32(2): 135-164.
44. Buyya R, Abramson D, Giddy J, Stockinger H. Economic models for resource management and scheduling in Grid computing. Concurrency and Computation: Practice and Experience 2002; 14(13-15): 1507-1542.
45. Yeo CS, Buyya R. Service level agreement based allocation of cluster resources: Handling penalty to enhance utility. Proceedings of the 7th IEEE International Conference on Cluster Computing (CLUSTER 2005), Boston, MA, September 2005. IEEE Computer Society Press: Los Alamitos, CA, 2005 (CDROM).
46. Amir Y, Awerbuch B, Barak A, Borgstrom RS, Keren A. An opportunity cost approach for job assignment in a scalable computing cluster. IEEE Transactions on Parallel and Distributed Systems 2000; 11(7):760-768.
47. Ranjan R, Buyya R. A case for cooperative and incentive-based coupling of distributed clusters. Proceedings of the 7th IEEE International Conference on Cluster Computing (CLUSTER 2005), Boston, MA, September 2005. IEEE Computer Society Press: Los Alamitos, CA, 2005 (CDROM).
48. Barmouta A, Buyya R. GridBank: A Grid accounting services architecture (GASA) for distributed systems sharing and integration. Proceedings of the 3rd Workshop on Internet Computing and E-Commerce (ICEC 2003), International Parallel and Distributed Processing Symposium (IPDPS 2003), Nice, France, April 2003. IEEE Computer Society Press: Los Alamitos, CA, 2003.
49. Jackson SM. Allocation management with QBank. http://www.emsl.pnl.gov/ docs/mscf/qbank [6 December 2004].
50. IBM. Business Process Execution Language for Web Services, version 1.1. http://www.ibm.com/developerworks/library/specification/ws-bpel [25 July 2005].
51. Global Grid Forum. Web Services Agreement Specification (WS-Agreement), version 1.1, draft 18. http://www.gridforum.org/Meetings/GGF11/Documents/draft- ggf-graap-agreement.pdf [25 July 2005].
52. Keller A, Ludwig H. The WSLA framework: Specifying and monitoring service level agreements for Web services. Journal of Network and Systems Management 2003; 11(1):57-81.
53. Yeo CS, Buyya R. Pricing for utility-driven resource management and allocation in clusters. Proceedings of the 12th International Conference on Advanced Computing and Communication (ADCOM 2004), Ahmedabad, India, December 2004. Allied Publisher: New Delhi, India, 2004; 32-41.
54. Byde A, Sallé M, Bartolini C. Market-based resource allocation for utility data centers. Technical Report HPL-2003-188, HP Lab, Bristol, September 2003.
55. AuYoung A, Chun BN, Snoeren AC, Vahdat A. Resource allocation in federated distributed computing infrastructures. Proceedings of the 1st Workshop on Operating System and Architectural Support far the on demand IT Infrastructure (OASIS 2004), Boston, MA, October 2004 (CDROM).
56. Eymann T, Reinicke M, Ardaiz O, Artigas P, Freitag F, Navarro L. Decentralized resource allocation in application layer networks. Proceedings of the 3rd International Symposium on Cluster Computing and the Grid (CCGrid 2003), Tokyo, Japan, May 2003. IEEE Computer Society Press: Los Alamitos, CA, 2003; 645-650.
57. Wolski R, Plank JS, Brevik J, Bryan T. Analyzing market-based resource allocation strategies for the Computational Grid. International Journal of High Performance Computing Applications 2001; 15(3):258-281.
58. Chen M, Yang G, Liu X. Gridmarket: A practical, efficient market balancing resource for Grid and P2P computing. Proceedings of the 2nd International Workshop on Grid and Cooperative Computing (GCC 2003), Shanghai, China, December 2003 (Lecture Notes in Computer Science, vol. 3033). Springer: Heidelberg, Germany, 2003; 612-619.
59. Grosu D, Das A. Auction-based resource allocation protocols in Grids. Proceedings of the 16th International Conference on Parallel and Distributed Computing and Systems (PDCS 2004), Cambridge, MA. November 2004. ACTA Press: Calgary, Canada, 2004; 20-27.
60. Chen C, Maheswaran M, Toulouse M. Supporting co-allocation in an auctioning-based resource allocator for Grid systems. Proceedings of the 11th International Heterogeneous Computing Workshop (HCW 2002), Fort Lauderdale, FL, April 2002. IEEE Computer Society Press: Los Alamitos, CA, 2002.
61. Padala P, Harrison C, Pelfort N, Jansen E, Frank MP, Chokkareddy C. OCEAN: The open computation exchange and arbitration network, a market approach to meta computing. Proceedings of the 2nd International Symposium on Parallel and Distributed Computing (ISPDC 2003), Ljubljana. Slovenia, October 2003. IEEE Computer Society Press: Los Alamitos, CA, 2003; 185-192.
62. Miller MS, Drexler KE. Incentive engineering for computational resource management. The Ecology of Computation, Huberman BA (ed.). Elsevier Science Publisher: New York, NY, 1988; 231-266.
63. Backschat M, Pfaffinger A, Zenger C. Economic-based dynamic load distribution in large workstation networks. Proceedings of the 2nd International Euro-Par Conference (Euro-Par 1996), Lyon, France, August 1996 (Lecture Notes in Computer Science, vol. 1124). Springer: Heidelberg, Germany, 1996; 631-634.
64. Ferguson DF, Yemini Y. Nikolaou C. Microeconomic algorithms for load balancing in distributed computer systems. Proceedings of the 8th International Conference on Distributed Computing Systems (ICDCS'88), San Jose, CA, June 1988. IEEE Computer Society Press: Los Alamitos, CA, 1988; 491-499.
65. Kurose JF, Simha R. A microeconomic approach to optimal resource allocation in distributed computer systems. IEEE Transactions on Computers 1989; 38(5):705-717.
66. Yemini Y, Dailianas A, Florissi D, Huberman G. MarketNet: Protecting access to information systems through financial market controls. Decision Support Systems 2000; 25(1-2):205-216.
67. Preist C, Byde A, Bartolini C. Economic dynamics of agents in multiple auctions. Proceedings of the 5th International Conference on Autonomous Agents (AGENTS 2001), Montreal, Canada, May-June 2001. ACM Press: New York, NY, 2001:545-551.
68. Stoica I, Abdel-Wahab H, Pothen A. A microeconomic scheduler for parallel computers. Proceedings of the 1st Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP '95), Santa Barbara, CA, April 1995 (Lecture Notes in Computer Science, vol. 949). Springer. Heidelberg, Germany, 1995; 200-218.
69. Wellman MP. A market-oriented programming environment and its application to distributed multicommodity flow problems. Journal of Artificial Intelligence Research 1993; 1:1-23.
70. Amir Y, Awerbuch B, Borgstrom RS. A cost-benefit framework for online management of a metacomputing system. Proceedings of the 1st International Conference on Information and Computation Economies (ICE '98), Charleston, SC, October 1998. ACM Press: New York, NY, 1998; 140-147.
71. Reed D, Pratt I, Menage P, Early S, Stratford N. Xenoservers: Accountable execution of untrusted programs. Proceedings of the 7th Workshop on Hot Topics in Operating Systems (HotOS-VII), Rio Rico, AZ, March 1999. IEEE Computer Society Press: Los Alamitos, CA. 1999; 136-141.
72. Chase JS, Irwin DE, Grit LE, Moore JD, Sprenkle SE. Dynamic virtual clusters in a Grid site manager. Proceedings of the 12th International Symposium on High Performance Distributed Computing (HPDC12), Seattle, WA, June 2003. IEEE Computer Society Press: Los Alamitos, CA, 2003; 90-100.
73. Barak A, La'adan O. The MOSIX multicomputer operating system for high performance cluster computing. Future Generation Computer Systems 1998; 13(4-5):361-372.
74. Chun BN, Culler DE. User-centric performance analysis of market-based cluster batch schedulers. Proceedings of the 2nd International Symposium on Cluster Computing and the Grid (CCGrid 2002), Berlin, Germany, May 2002. IEEE Computer Society Press: Los Alamitos. CA, 2002; 22-30.
75. Kale LV, Krishnan S. Charm++: Parallel programming with message-driven objects. Parallel Programming Using C++, Wilson GV, Lu P (eds.). MIT Press: Cambridge, MA, 1996; 175-213.
76. Bhandarkar M, Kalé LV, de Sturler E, Hoeflinger J. Adaptive load balancing for MPI programs. Proceedings of the International Conference on Computational Science (ICCS 2001), San Francisco, CA, May 2001 (Lecture Notes in Computer Science, vol. 2074). Springer: Heidelberg, Germany, 2001; 108-117.
77. Kalé LV, Kumar S, DeSouza J. A malleable-job system for timeshared parallel machines. Proceedings of the 2nd International Symposium on Cluster Computing and the Grid (CCGrid 2002), Berlin, Germany, May 2002. IEEE Computer Society Press: Los Alamitos, CA, 2002; 215-222.
78. Abramson D, Sosic R, Giddy J, Hall B. Nimrod: A tool for performing parametrised simulations using distributed workstations. Proceedings of the 4th International Symposium on High Performance Distributed Computing (HPDC4), Pentagon City, VA, August 1995. IEEE Computer Society Press: Los Alamitos, CA, 1995; 112-121.
79. Buyya R, Giddy J, Abramson D. An evaluation of economy-based resource trading and scheduling on computational power Grids for parameter sweep applications. Proceedings of the 2nd Annual Workshop on Active Middleware Services (AMS 2000), Pittsburgh, PA, August 2000. Kluwer Academic Publishers: Dordrecht, Netherlands, 2000.