Shared memory computing on clusters with symmetric multiprocessors and system area networks

ACM Transactions on Computer Systems

Volumn 23, Issue 3, 2005, Pages 301-335

  Kontothanassis, Leonidas ^a,g   Stets, Robert ^b   Hunt, Galen ^c   Rencuzogllari, Umit ^d   Altekar, Gautam ^e   Dwarkadas, Sandhya ^f,h   Scott, Michael L ^f,h  

^a HEWLETT PACKARD LABORATORIES   (United States)

^b GOOGLE INC   (United States)

^c MICROSOFT RESEARCH   (United States)

^d VMWARE INC   (United States)

^e UNIVERSITY OF CALIFORNIA   (United States)

^f UNIVERSITY OF ROCHESTER   (United States)

^g HEWLETT PACKARD COMPANY   (United States)

^h University of Rochester   (United States)

Author keywords

Distributed shared memory; Relaxed consistency; Software coherence

Indexed keywords

DISTRIBUTED SHARED MEMORY; MESSAGE ORDERING; RELAXED CONSISTENCY; SOFTWARE COHERENCE;

COMPUTER NETWORKS; COMPUTER SOFTWARE; DATA STORAGE EQUIPMENT; DATABASE SYSTEMS; MICROPROCESSOR CHIPS; NETWORK PROTOCOLS;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 33745212645     PISSN: 07342071     EISSN: 07342071     Source Type: Journal    
DOI: 10.1145/1082469.1082472     Document Type: Article

References (70)

1. ADVE, S. V. AND HILL, M. D. 1993. A unified formulation of four shared-memory models. IEEE Trans. Para. Distrib. Syst. 4, 6 (June), 613-624.
2. AGAHWAL, A., BIANCHINI, R., CHAIKEN, D., JOHNSON, K., KRANZ, D., KUBIATOWICZ, J., LIM, B.-H., MACKENZIE, K., AND YEUNG, D. 1995. The MIT Alewife machine: Architecture and performance. In Proceedings of the 22nd International Symposium on Computer Architecture, Santa Margherita Ligure, Italy, June.
3. AMERICAN NATIONAL STANDARDS INSTITUTE. 1996. Information Systems - High-Performance Parallel Interface-Mechanical, Electrical, and Signalling Protocol Specification (HIPPI-PH). ANSI X3.183-1991 (R1996), New York, NY.
4. AMZA, C., COX, A. L., DWARKADAS, S., KELEHER, P., LU, H., RAJAMONY, R., YU, W., AND ZWAENEPOEL, W. 1996. TreadMarks: Shared memory computing on networks of Workstations. Computer 29, 2 (Feb.), 18-28.
5. AMZA, C., COX, A., DWARKADAS, S., AND ZWAENEPOEL, W. 1997. Software DSM protocols that adapt between single writer and multiple writer. In Proceedings of the 3rd International Symposium on High Performance Computer Architecture, San Antonio, TX, Feb.
6. BENNETT, J. K., CARTER, J. B., AND ZWAENEPOEL, W. 1990. Adaptive software cache management for distributed shared memory architectures. In Proceedings of the 17th International Symposium on Computer Architecture, Seattle, WA, May.
7. BILAS, A., IPTODE, L., MARTIN, D., AND SINGH, J. P. 1996. Shared Virtual Memory Across SMP Nodes Using Automatic Update: Protocols and Performance. Tech. Rep. TR-517-96, Dept. of Computer Science, Princeton Univ., Oct.
8. BILAS, A., LIAO, C., AND SINGH, J. P. 1999. Using network interface support to avoid asynchronous protocol processing in shared virtual memory systems. In Proceedings of the 26th International Symposium on Computer Architecture, Atlanta, GA, May.
9. BLUMRICH, M., LI, K., ALPERT, R., DUBNICKI, C., FELTEN, E., AND SANDBERG, J. 1994. Virtual memory mapped network interface for the SHRIMP multicomputer. In Proceedings of the 21st International Symposium on Computer Architecture, Chicago, IL, Apr.
10. BOLOSKY, W. J., FITZGERALD, R. P., AND SCOTT, M. L. 1989. Simple but effective techniques for NUMA memory management. In Proceedings of the 12th ACM Symposium on Operating Systems Principles, Litchfield Park, AZ, Dec.
11. BOLOSKY, W. J., SCOTT, M. L., FITZGERALD, R. P., FOWLER, R. J., AND COX, A. L. NUMA 1991. Policies and their relation to memory architecture. In Proceedings of the 4th International Conference on Architectural Support for Programming Languages and Operating Systems, Santa Clara, CA, Apr.
12. BOLOSKY, W. J. AND SCOTT, M. L. 1992. Evaluation of multiprocessor memory systems using off-line optimal behavior. J. Para. Distrib. Comput. 15, 4 (Aug.), 382-398.
13. BUZZARD, G., JACOBSON, D., MACKEY, M., MAROVICH, S., AND WILKES, J. 1996. An implementation of the Hamlyn sender-managed interface architecture. In Proceedings of the 2nd Symposium on Operating Systems Design and Implementation, Seattle, WA, Oct.
14. CARTER, J. B., BENNETT, J. K., AND ZWAENEPOEL, W. 1991. Implementation and performance of Munin. In Proceedings of the 13th ACM Symposium on Operating Systems Principles, Pacific Grove, CA, Oct.
15. CHASE, J. S., AMADOR, F. G., LAZOWSKA, E. D., LEVY, H. M., AND LITTLEFIELD, R. J. 1989. The amber system: Parallel programming on a network of multiprocessors. In Proceedings of the 12th ACM Symposium on Operating Systems Principles, Litchfield Park, AZ, Dec.
16. COMPAQ, INTEL, AND MICROSOFT. 1997. Virtual Interface Architecture Specification. Draft Revision 1.0, Dec. Available at ftp://download.intel.com/ design/servers/vi/san_10.pdf.
17. COX, A. L. AND FOWLER, R. J. 1989. The implementation of a coherent memory abstraction on a NUMA multiprocessor: Experiences with PLATINUM. In Proceedings of the 12th ACM Symposium on Operating Systems Principles, Litchfield Park, AZ, Dec.
18. COX, A., DWARKADAS, S., KELEHER, P., LU, H., RAJAMONY, R., AND ZWAENEPOEL, W. 1994. Software versus hardware shared-memory implementation: A Case Study. In Proceedings of the 21st International Symposium on Computer Architecture, Chicago, IL, Apr.
19. CULLER, D., DUSSEAU, A., GOLDSTEIN, S., KRISHNAMURTHY, A., LUMETTA, S., VON EICKEN, T., AND YELICK, K. 1993. Parallel programming in split-C. In Proceedings of Supercomputing '93, Portland, OR, Nov.
20. DUNNING, D., REGNIER, G., MCALPINE, G., CAMEHON, D., SHUBERT, B., BERRY, F., MERRITT, A. M., GHONKE, E., AND DODD, C. 1998. The virtual interface architecture. IEEE Micro. 18, 2 (Mar.), 66-76.
21. DWARKADAS, S., SCHÄFFER, A. A., COTTINGHAM JR., R. W., COX, A. L., KELEHER, P., AND ZWAENEPOEL, W. 1994. Parallelization of General Linkage Analysis Problems. Human Heredity 44, 127-141.
22. DWARKADAS, S., COX, A. L., AND ZWAENEPOEL, W. 1996. An integrated compile-time/Run-time Software Distributed Shared Memory System. In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, Cambridge, MA, Oct.
23. DWARKADAS, S., HARDAVELLAS, N., KONTOTHANASSIS, L. I., NIKHIL, R., AND STETS, R. 1999a. Cashmere-VLM: Remote memory paging for software distributed shared memory. In Proceedings of the 13th International Parallel Processing Symposium, San Juan, Puerto Rico, Apr.
24. DWARKADAS, S., GHARACHORLOO, K., KONTOTHANASSIS, L. I., SCALES, D. J., SCOTT, M. L., AND STETS, R. 1999b. Comparative evaluation of fine- and coarse-grain approaches for software distributed shared memory. In Proceedings of the 5th International Symposium on High Performance Computer Architecture, Orlando, FL, Jan.
25. ERLICHSON, A., NUCKOLLS, N., CHESSON, G., AND HENNESSY, J. 1996. SoftFLASH: Analyzing the performance of clustered distributed virtual shared memory. In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, Cambridge, MA, Oct.
26. FEELEY, M. J., CHASE, J. S., NARASAYYA, V. R., AND LEVY, H. M. 1994. Integrating coherency and recovery in distributed systems. In Proceedings of the 1st Symposium on Operating Systems Design and Implementation, Monterey, CA, Nov.
27. FILLO, M. AND GILLETT, R. B. 1997. Architecture and implementation of memory channel 2. Digital Technical Journal 9, 1, 27-41.
28. GILLETT, R. 1996. Memory channel: An optimized cluster interconnect. IEEE Micro 16, 2 (Feb.), 12-18.
29. GOODMAN, J. R. 1987. Coherency for multiprocessor virtual address caches. In Proceedings of the 2nd International Conference on Architectural Support for Programming Languages and Operating Systems, Palo Alto, CA, Oct.
30. HILL, M. D., LARUS, J. R., REINHARDT, S. K., AND WOOD, D. A. 1993. Cooperative shared memory: Software and hardware for scalable multiprocessors. ACM Trans. Comput. Syst. 11, 4, 300-318, Nov.
31. IFTODE, L., DUBNICKI, C., FELTEN, E. W., AND LI, K. 1996. Improving release-consistent shared virtual memory using automatic update. In Proceedings of the 2nd International Symposium on High Performance Computer Architecture, San Jose, CA, Feb.
32. INFINIBAND TRADE ASSOCIATION. 2002. InfiniBand Architecture Specification. Release 1.1, Nov. Available at www.infinibandta.org/specs.
33. JOHNSON, K. L., KAASHOEK, M. F., AND WALLACH, D. A. 1995. CRL: High-performance all-software distributed shared memory. In Proceedings of the 15th ACM Symposium on Operating Systems Principles, Copper Mountain, CO, Dec.
34. KARLSSON, M. and STENSTROM, M. P. 1996. Performance evaluation of a cluster-based multiprocessor built from ATM switches and bus-based multiprocessor servers. In Proceedings of the 2nd International Symposium on High Performance Computer Architecture, San Jose, CA, Feb.
35. KELEHER, P., COX, A. L., AND ZWAENEPOEL, W. 1992. Lazy release consistency for software distributed shared memory. In Proceedings of the 19th International Symposium on Computer Architecture, Gold Coast, Australia, May.
36. KONTOTHANASSIS, L. I. AND SCOTT, M. L. 1995a. High performance software coherence for current and future architectures. J. Para. Distrib. Comput. 29, 2 (Nov.), 179-195.
37. KONTOTHANASSIS, L. I. AND SCOTT, M. L. 1995b. Software cache coherence for large scale multi-processors. In Proceedings of the 1st International Symposium on High Performance Computer Architecture, Raleigh, NC, Jan.
38. KONTOTHANASSIS, L. I. AND SCOTT, M. L. 1996. Using memory-mapped network interfaces to improve the performance of distributed shared memory. In Proceedings of the 2nd International Symposium on High Performance Computer Architecture, San Jose, CA, Feb.
39. KONTOTHANASSIS, L. I., HUNT, G. C., STETS, R., HARDAVELLAS, N., CIERNIAK, M., PARTHASARATHY, S., MEIRA, W., DWARKADAS, S., AND SCOTT, M. L. 1997. VM-based shared memory on low-latency, remote-memory-access networks. In Proceedings of the 24th International Symposium on Computer Architecture, Denver, CO, June.
40. LAROWE JR., R. P. AND ELLIS, C. S. 1991. Experimental comparison of memory management policies for NUMA multiprocessors. ACM Trans. Comput. Syst. 9, 4 (Nov.), 319-363.
41. LAUDON, J. AND LENOSKI, D. 1997. The SGI origin: A ccNUMA highly scalable server. In Proceedings of the 24th International Symposium on Computer Architecture, Denver, CO, June.
42. LI, K. AND SCHAEFER, R. 1989. A hypercube shared virtual memory system. In Proceedings of the 1989 International Conference on Parallel Processing, St. Charles, IL, Aug. Penn. State Univ. Press.
43. LI, K. AND HUDAK, P. 1989. Memory coherence in shared virtual memory systems. ACM Trans. Comput. Syst. 7, 4 (Nov.), 321-359.
44. MARCHETTI, M., KONTOTHANASSIS, L. I., BIANCHINI, R., AND SCOTT, M. L. 1995. Using simple page placement policies to reduce the cost of cache fills in coherent shared-memory systems. In Proceedings of the 9th International Parallel Processing Symposium, Santa Barbara, CA, Apr.
45. MONNERAT, L. R. AND BIANCHINI, R. 1998. Efficiently adapting to sharing patterns in software DSMs. In Proceedings of the 4th International Symposium on High Performance Computer Architecture, Las Vegas, NV, Feb.
46. NIKHIL, R. S. 1994. Cid: A parallel, "Shared-memory" C for Distributed-Memory Machines. In Proceedings of the 7th Annual Workshop on Languages and Compilers for Parallel Computing, Aug.
47. NITZBERG, B. AND Lo, V. 1991. Distributed shared memory: A Survey of issues and algorithms. Comput. 24, 8 (Aug.), 52-60.
48. PETERSEN, K. AND LI, K. 1993. Cache coherence for shared memory multiprocessors based on virtual memory support. In Proceedings of the 7th International Parallel Processing Symposium, Newport Beach, CA, Apr.
49. PHILBIN, J. F., DUBNICKI, C., BILAS, A., AND LI, K. 1997. Design and implementation of virtual memory-mapped communication on myrinet. In Proceedings of the 11th International Parallel Processing Symposium, Geneva, Switzerland, Apr.
50. REINHARDT, S. K., LARUS, J. R., and WOOD, D. A. 1994. Tempest and Typhoon: User-level shared-memory. In Proceedings of the 21st International Symposium on Computer Architecture, Chicago, IL, Apr.
51. SAMANTA, R., BILAS, A., IFTODE, L., AND SINGH, J. P. 1998. Home-based SVM protocols for SMP clusters: Design and performance. In Proceedings of the 4th International Symposium on High Performance Computer Architecture, Las Vegas, NV, Feb.
52. SANDHU, H. S., GAMSA, B., AND ZHOU, S. 1993. The shared regions approach to software cache coherence on multiprocessors. In Proceedings of the 4th ACM Symposium on Principles and Practice of Parallel Programming, San Diego, CA, May.
53. SCALES, D. J., GHARACHORLOO, K., AND THEKKATH, C. A. 1996. Shasta: A low overhead, software-only approach for supporting fine-grain shared memory. In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, Cambridge, MA, Oct.
54. SCALES, D. J. AND GHARACHORLOO, K. 1997. Towards transparent and efficient software distributed shared memory. In Proceedings of the 16th ACM Symposium on Operating Systems Principles, St. Malo, France, Oct.
55. SCALES, D. J., GHARACHORLOO, K., AND AGGARWAL, A. 1998. Fine-grain software distributed shared memory on SMP clusters. In Proceedings of the 4th International Symposium on High Performance Computer Architecture, Las Vegas, NV, Feb.
56. SCHOINAS, I., FALSAFI, B., HILL, M. D., LARUS, J. R. AND WOOD, D. A. 1998. Sirocco: Cost-effective fine-grain distributed shared memory. In Proceedings of the International Conference on Parallel Architectures and Compilation Techniques (PACT), Paris, France, Oct.
57. SCHOINAS, I., FALSAFI, B., LEBECK, A. R., REINHARDT, S. K., LARUS, J. R., AND WOOD, D. A. 1994. Fine-grain access control for distributed shared memory. In Proceedings of the 6th International Conference on Architectural Support for Programming Languages and Operating Systems, San Jose, CA, Oct.
58. SINGH, J. P., WEBER, W.-D., AND GUPTA, A. 1992. SPLASH: Stanford parallel applications for shared-memory. ACM SIGARCH Computer Architecture News 20, 1 (Mar.), 5-44.
59. SPEIGHT, E. AND BENNETT, J. K. 1998. Using multicast and multithreading to reduce communication in software DSM systems. In Proceedings of the 4th International Symposium on High Performance Computer Architecture, Las Vegas, NV, Feb.
60. STETS, R., DWARKADAS, S., KONTOTHANASSIS, L. I., RENCUZOGULLARI, U., AND SCOTT, M. L. 2000. The Effect of Network Total Order, Broadcast, and Remote-Write Capability on Network-Based Shared Memory Computing. In Proceedings of the 6th International Symposium on High Performance Computer Architecture, Toulouse, France, Jan.
61. STETS, R., DWARKADAS, S., HARDAVELLAS, N., HUNT, G. C., KONTOTHANASSIS, L. I., PARTHASARATHY, S., AND SCOTT, M. L. 1997. Cashmere-2L: Software coherent shared memory on a clustered remote-write network. In Proceedings of the 16th ACM Symposium on Operating Systems Principles, St. Malo, France, Oct.
62. TANENBAUM, A. S., KAASHOEK, M. F., AND BAL, H. E. 1992. Parallel programming using shared objects and broadcasting. Comput. 25, 8 (Aug.), 10-19.
63. Top 500 Supercomputer Sites. 2003. Univ. of Manheim, Univ. of Tennessee, and NERSC/LBNL, June, http://www.top500.org/lists/2003/06/.
64. VERGHESE, B., DEVINE, S., GUPTA, A., AND ROSENBLUM, M. 1996. Operating system support for improving data locality on CC-NUMA compute servers. In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, Cambridge, MA, Oct.
65. VON EICKEN, T., BASU, A., BUCH, V., AND VOGELS, W. 1995. U-Net: A user-level network interface for parallel and distributed computing. In Proceedings of the 15th ACM Symposium on Operating Systems Principles, Copper Mountain, CO, Dec.
66. WELSH, M., BASU, A., and VON EICKEN, T. 1997. Incorporating memory management into user-level network interfaces. Tech. Rep. TR97-1620, Cornell Univ., Aug.
67. Woo, S. C., OHARA, M., TORRIE, E., SINGH, J. P., AND GUPTA, A. 1995. Methodological Considerations and Characterization of the SPLASH-2 Parallel Application Suite. In Proceedings of the 22nd International Symposium on Computer Architecture, Santa Margherita Ligure, Italy, June.
68. YEUNG, D., KUBIATOWITCZ, J., AND AGARWAL, A. 1996. MGS: A multigrain shared memory system. In Proceedings of the 23rd International Symposium on Computer Architecture, Philadelphia, PA, May.
69. ZEKAUSKAS, M. J., SAWDON, W. A., AND BERSHAD, B. N. 1994. Software write detection for distributed shared memory. In Proceedings of the 1st Symposium on Operating Systems Design and Implementation, Monterey, CA, Nov.
70. ZHOU, Y., IFTODE, L., SINGH, J. P., LI, K., TOONEN, B. R., SCHOINAS, I., HILL, M. D., AND WOOD, D. A. 1997. Relaxed consistency and coherence granularity in DSM systems: A performance evaluation. In Proceedings of the 6th ACM Symposium on Principles and Practice of Parallel Programming, Las Vegas, NV, June.