Dynamic load balancing for I/O-intensive applications on clusters

ACM Transactions on Storage

Volumn 5, Issue 3, 2009, Pages

  Qin, Xiao ^a   Jiang, Hong ^b   Manzanares, Adam ^a   Ruan, Xiaojun ^a   Yin, Shu ^a  

^a AUBURN UNIVERSITY   (United States)

^b UNIVERSITY OF NEBRASKA LINCOLN   (United States)

Author keywords

Clusters; Heterogeneity; I O intensive applications; Load balancing; Storage systems

Indexed keywords

ANALYTIC MODELS; BATCH SCHEDULERS; CHOLESKY; COMPUTING POWER; DISK SPEED; DYNAMIC LOAD BALANCING; HETEROGENEOUS CLUSTERS; HOMOGENEOUS CLUSTER; I/O-INTENSIVE APPLICATIONS; LOAD-BALANCING; LOAD-BALANCING SCHEMES; LU DECOMPOSITION; MEMORY CAPACITY; MEMORY LOAD; MEMORY RESOURCES; PARALLEL APPLICATION; PARALLEL JOBS; PARALLEL TEXT; PERFORMANCE CHARACTERISTICS; PERFORMANCE IMPROVEMENTS; SEQUENTIAL JOBS; STORAGE SYSTEMS; SYSTEM LEVELS; TRANSFER PROCESS; WIDE SPECTRUM;

SCHEDULING ALGORITHMS;

EID: 72449176435     PISSN: 15533077     EISSN: 15533093     Source Type: Journal    
DOI: 10.1145/1629075.1629078     Document Type: Article

References (46)

1. ACHARYA, A. AND SETIA, S. 1999. Availability and utility of idle memory in workstation clusters. In Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS'99). ACM, New York, NY, 35-46.
2. AGARWALA, S., AGARWALA, I., POELLABAUER, C., KONG, J., SCHWAN, K., AND WOLF, M. 2003. Resourceaware stream management with the customizable dproc distributed monitoring mechanisms. In Proceedings of the 12th IEEE International Symposium on High Performance Distributed Computing (HPDC-12). 250-259.
3. BASNEY, J. AND LIVNY, M. 2000. Managing network resources in condor. In Proceedings of the 9th IEEE International Symposium on High Performance Distributed Computing (HPDC'00). IEEE Computer Society, Los Alamitos, CA, 298.
4. BODE,B.,HALSTEAD,D. M.,KENDALL, R., LEI, Z., AND JACKSON,D. 2000. The portable batch scheduler and the maui scheduler on linux clusters. In Proceedings of the 4th Annual Linux Showcase & Conference (ALS'00). USENIX Association, Berkeley, CA, 27-27
5. BRAUN, T. D., SIEGAL, H. J., AND BECK, ET AL. 1999. A comparison study of static mapping heuristics for a class of meta-tasks on heterogeneous computing systems. In Proceedings of the 8th Heterogeneous Computing Workshop (HCW'99). IEEE Computer Society, Los Alamitos, CA, 15.
6. BROWN, T. 1979. M/g/1 round robin discipline. Computing 22, 3, 225-241.
7. CAO, J., LI, Y., AND GUO, M. 2005. Process migration for mpi applications based on coordinated checkpoint. In Proceedings of the 11th International Conference on Parallel and Distributed Systems (ICPADS'05). IEEE Computer Society, Los Alamitos, CA, 306-312.
8. CARNS, P. H., LIGON, W. B., III, ROSS, R. B., AND THAKUR, R. 2000. Pvfs: A parallel file system for linux clusters. In Proceedings of the 4th Annual Linux Showcase and Conference. USENIX Association, 317-327.
9. CARR, R. W. AND HENNESSY, J. L. 1981. Wsclock-a simple and effective algorithm for virtual memory management. SIGOPS Oper. Syst. Rev. 15, 5, 87-95.
10. CHANG, C., MOON, B., ACHARYA, A., SHOCK, C., SUSSMAN, A., AND SALTZ, J. 1997. Titan: a highperformance remote-sensing database. In Proceedings of the International Conference on Data Engineering. IEEE Computer Society Press, 375-384.
11. CHOW, K.-P. ANDKWOK, Y.-K. 2002. On load balancing for distributed multiagent computing. IEEE Trans. Parall. Distrib. Syst. 13, 8, 787-801.
12. CRUZ, J. AND PARK, K. 2001. Towards communication-sensitive load balancing. In Proceedings of the 21st International Conference on Distributed Computing Systems, 731-734.
13. DEAN, J. AND GHEMAWAT, S. 2008. Mapreduce: Simplified data processing on large clusters. Comm. ACM 51, 1, 107-113.
14. DENG, Q., WANG, X., AND ZANG, D. 2005. Monitoring MPI running nodes status for load balance. In Proceedings of the 4th International Conference on Grid and Cooperative Computing. Lecture Notes in Computer Science, vol.3795, Springer, Berlin, 467-473.
15. DUSSEAU, A. C., ARPACI, R. H., AND CULLER, D. E. 1996. Effective distributed scheduling of parallel workloads. SIGMETRICS Perform. Eval. Rev. 24, 1, 25-36.
16. EAGER, D. L., LAZOWSKA, E. D., AND ZAHORJAN, J. 1986. Adaptive load sharing in homogeneous distributed systems. IEEE Trans. Softw. Engin. 12, 5, 662-675.
17. FORNEY, B. C., ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEAU, R. H. 2002. Storage-aware caching: Revisiting caching for heterogeneous storage systems. In Proceedings of the 1st USENIX Conference on File and Storage Technologies (FAST'02). USENIX Association, Berkeley, CA, 5.
18. GEOFFRAY, P. 2002. Opiom: off-processor i/o with myrinet. Future Gener. Comput. Syst. 18, 4, 491-499.
19. GHEMAWAT, S., GOBIOFF, H., AND LEUNG, S.-T. 2003. The google file system. SIGOPS Oper. Syst. Rev. 37, 5, 29-43.
20. HARCHOL-BALTER, M. AND DOWNEY, A. B. 1996. Exploiting process lifetime distributions for dynamic load balancing. In Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS'96). ACM, New York, NY, 13-24.
21. HUI, C.-C. AND CHANSON, S. T. 1999. Improved strategies for dynamic load balancing. IEEE Concurrency 7, 3, 58-67.
22. KANNAN, S., ROBERTS, M.,MAYES, P., BRELSFORD, D., AND SKOVIRA, J. F. 2001. Workload Management with LoadLeveler. IBM.
23. KEREN, A. AND BARAK, A. 2003. Opportunity cost algorithms for reduction of i/o and interprocess communication overhead in a computing cluster. IEEE Trans. Parall. Distrib. Syst. 14, 1, 39-50.
24. KIM, M. Y. 1986. Synchronized disk interleaving. IEEE Trans. Comput. 35, 11, 978-988.
25. KOTZ, D. AND NIEUWEJAAR, N. 1994. Dynamic file-access characteristics of a production parallel scientific workload. In Proceedings of the Conference on Supercomputing (Supercomputing'94). IEEE Computer Society Press, Los Alamitos, CA, 640-649.
26. LAVI, R. AND BARAK, A. 2001. The home model and competitive algorithms for load balancing in a computing cluster. In Proceedings of the The 21st International Conference on Distributed Computing Systems (ICDCS'01). IEEE Computer Society, Los Alamitos, CA, 127.
27. LEE, L.-W., SCHEUERMANN, P., AND VINGRALEK, R. 2000. File assignment in parallel i/o systems with minimal variance of service time. IEEE Trans. Comput. 49, 2, 127-140.
28. MA, X., WINSLETT, M., LEE, J., AND YU, S. 2002. Faster collective output through active buffering. In Proceedings of the 16th International Parallel and Distributed Processing Symposium (IPDPS'02). IEEE Computer Society, Los Alamitos, CA, 151.
29. PASQUALE, B. K. AND POLYZOS, G. C. 1994. Dynamic i/o characterization of i/o intensive scientific applications. In Proceedings of the Conference on Supercomputing (Supercomputing'94). IEEE Computer Society Press, Los Alamitos, CA, 660-669.
30. QIN, X. 2008. Performance comparisons of load balancing algorithms for i/o-intensive workloads on clusters. J. Netw. Comput. Appl. 31, 1, 32-46.
31. QIN, X., JIANG, H., ZHU, Y., AND SWANSON, D. R. 2003a. Dynamic load balancing for i/o- and memory-intensive workload in clusters using a feedback control mechanism. In Proceedings of the 9th International Euro-Par Conference on Parallel Processing (Euro-Par'03). 224-229.
32. QIN, X., JIANG, H., ZHU, Y., AND SWANSON, D. R. 2003b. Dynamic load balancing for i/o-intensive tasks on heterogeneous clusters. In Proceedings of the 10th International Conference on High Performance Computing (HiPC'03). 300-309.
33. QIN, X., JIANG, H., ZHU, Y., AND SWANSON, D. R. 2003c. A dynamic load balancing scheme for i/ointensive applications in distributed systems. Proceedings of the International Conference on Parallel Processing Workshops. 79.
34. ROADS, J. AND ET AL. 1992. Apreliminary description of the western u.s. climatology. In Proceedings of the Annual Pacific Climate Workshop.
35. SINGH, R. AND GRAHAM, P. 2008. Performance driven partial checkpoint/migrate for lam-mpi. In Proceedings of the 22nd International Symposium on High Performance Computing Systems and Applications (HPCS'08). IEEE Computer Society, Los Alamitos, 110-116.
36. SURDEANU, M., MOLDOVAN, D. I., AND HARABAGIU, S. M. 2002. Performance analysis of a distributed question/answering system. IEEE Trans. Parall. Distrib. Syst. 13, 6, 579-596. (Pubitemid 34835457)
37. TANAKA, T. 1993. Configurations of the solar wind flow and magnetic field around the planets with no magnetic field: Calculation by a new mhd. Geophys. Res., 17251-17262.
38. UYSAL, M., ACHARYA, A., AND SALTZ, J. 1997. Requirements of i/o systems for parallel machines: an application-driven study. Tech. rep., College Park, MD.
39. VARMAN, P. J. AND VERMA, R. M. 1999. Tight bounds for prefetching and buffer management algorithms for parallel i/o systems. IEEE Trans. Parall. Distrib. Syst. 10, 12, 1262-1275.
40. VAZHKUDAI, S. AND SCHOPF, J.M. 2002. Using disk throughput data in predictions of end-to-end grid data transfers. In Proceedings of the 3rd International Workshop on Grid Computing (GRID'02). Springer-Verlag, Berlin, 291-304.
41. VOELKER, G. M., JAMROZIK, H. A., VERNON,M. K., LEVY, H. M., AND LAZOWSKA, E. D. 1997. Managing server load in global memory systems. In Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS'97). ACM, New York, NY, 127-138.
42. VYDYANATHAN, N., KHANNA, G., KURC, T., CATALYUREK, U., WYCKOFF, P., SALTZ, J., AND SADAYAPPAN, P. 2004. Use of pvfs for efficient execution of jobs with pipeline-shared i/o. In Proceedings of the 5th IEEE/ACM InternationalWorkshop on Grid Computing (GRID'04). IEEE Computer Society, Los Alamitos, CA, 235-242.
43. XIAO, L., ZHANG, X., ANDQU, Y. 2000. Effective load sharing on heterogeneous networks of workstations. In Proceedings of the of International Symposium on Parallel and Distributed Processing. IEEE Computer Society Press, 431-438.
44. ZHANG, X., XIAO, L., ANDQU, Y. 2000. Improving distributed workload performance by sharing both cpu and memory resources. In Proceedings of the 20th International Conference on Distributed Computing Systems (ICDCS'00). IEEE Computer Society, Los Alamitos, CA, 233.
45. ZHANG, Y., YANG, A., SIVASUBRAMANIAM, A., AND MOREIRA, J. 1993. Gang scheduling extensions for i/o intensive workloads. In Proceedings of the Job Scheduling Strategies for Parallel Processing Workshop.
46. ZHU, Y., JIANG, H., QIN, X., AND SWANSON, D. 2004. A case study of parallel i/o for biological sequence search on linux clusters. Int. J. High Perform. Comput. Netw. 1, 4, 214-222.