Exploiting domain-specific properties: Compiling parallel dynamic neural network algorithms into efficient code

IEEE Transactions on Parallel and Distributed Systems

Volumn 10, Issue 11, 1999, Pages 1105-1117

  Prechelt, Lutz ^a,b  

^a UNIVERSITY OF KARLSRUHE   (Germany)

^b IEEE   (Germany)

Author keywords

Communication optimization; Compiler optimizations; Dynamic data structures; High level parallel language; Irregular problems

Indexed keywords

CODES (SYMBOLS); CONSTRAINT THEORY; DATA COMMUNICATION SYSTEMS; DATA REDUCTION; DATA STRUCTURES; HIGH LEVEL LANGUAGES; LEARNING ALGORITHMS; OPTIMIZATION; PARALLEL ALGORITHMS; PARALLEL PROCESSING SYSTEMS; PROGRAM COMPILERS;

DOMAIN SPECIFIC CONSTRAINTS;

NEURAL NETWORKS;

EID: 0033220971     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/71.809571     Document Type: Article

References (31)

1. Neurocomputing: Foundations of Research, J.A. Anderson and E. Rosenfeld, eds. Cambridge, Mass.: MIT Press, 1988.
2. G.E. Blelloch, S. Chatterjee, J.C. Hardwick, J. Sipelstein, and M. Zagha, "Implementation of a Nested Data-Parallel Language," Proc. Fourth ACM SIGPLAN Symp. Principles and Practice of Parallel Programming (PPoPP), ACM SIGPLAN Notices, vol. 28, no. 7, pp. 102-111, July 1993.
3. S. Chatterjee, J.R. Gilbert, R. Schreiber, and S.-H. Teng, "Automatic Array Alignment in Data Parallel Programs," Proc. 20th ACM SIGPLAN-SIGACT Symp. Principles of Programming Languages (POPL), pp. 16-28, Charleston, S.C., Jan. 1993.
4. W. Finnoff, F. Hergert, and H.G. Zimmermann, "Improving Model Selection by Nonconvergent Methods," Neural Networks, vol. 6, pp. 771-783, 1993.
5. S.F. Hummel, E. Schonberg, and L.E. Flynn, "Factoring - A Method for Scheduling Parallel Loops," Comm. ACM, vol. 35, no. 8, pp. 90-101, Aug. 1992.
6. B. Gomes, "A Framework for Mapping Connectionist Networks onto Parallel Machines," PhD thesis, Electrical Engineering and Computer Science Dept., Univ. of California, Berkeley, May 1997.
7. K.A. Grajski, "Neurocomputing Using the MasPar MP-1," Technical Report 90-010, MasPar Computers, Sunnyvale, Calif., 1990.
8. R.W. Gray, V.P. Heuring, S.P. Levi, A.M. Sloane, and W.M. Waite, "Eli: A Complete, Flexible Compiler Construction System," Comm. ACM, vol. 35, no. 2, pp. 121-131, Feb. 1992.
9. D. Hammerstrom, The CNAPS Architecture, Adaptive Solutions, Beaverton, Ore., Jan. 1993.
10. B. Hendrickson and R. Leland, The Chaco User's Guide, version 1.0. UC-405 SAND93-2339, Sandia National Laboratories, Albuquerque, N.M., Oct. 1993.
11. H. Hopp and L. Prechelt, "CUPIT-2: A Portable Parallel Programming Language for Artificial Neural Networks," Proc. 15th IMACS World Congress Scientific Computation, Modelling, and Applied Math., A. Sydow, ed., vol. 6, pp. 493-498, Berlin: Wissenschaft and Technik Verlag, Aug. 1997.
12. C. Jacob and P. Wilke, "A Distributed Network Simulation Environment for Multi-Processing Systems," Proc. Int'l Joint Conf. Neural Networks (IJCNN), pp. 1,178-1,183, Singapore, 1991.
13. G. Kock and T. Becher, "Mind: An Environment for the Development, Integration, and Acceleration of Connectionist Systems," Proc. 15th IMACS World Congress Scientific Computation, Modelling, and Applied Math., pp. 499-504, 1997.
14. G. Kock and N.B. Serbedzija, "Artificial Neural Networks: From Compact Descriptions to C++," Proc. Int'l Conf. Artificial Neural Networks, 1994.
15. D. Koll, M. Riedmiller, and H. Braun, "Massively Parallel Training of Multi Layer Perceptrons with Irregular Topologies," Proc. Int'l Conf. Artificial Neural Networks and Genetic Algorithms (ICANNGA), Ales, France, Springer Verlag, 1995.
16. X. Liu and G.L. Wilcox, "Benchmarking of the CM-5 and the Cray Machines with a Very Large Backpropagation Neural Network, Technical Report 93/38, Univ. of Minnesota Supercomputer Inst., Apr. 1993.
17. MPL Language Reference Manual. Sunnyvale, Calif.: MasPar Computers, 1990.
18. W. McCulloch and W. Pitts, "A Logical Calculus of Ideas Immanent in Nervous Activity," Bull. Math. Biophysics, vol. 5, pp. 115-133, 1943.
19. M. Misra, "Parallel Environments for Implementing Neural Networks," Neural Computing Surveys vol. 1, pp. 48-60, 1997.
20. S. Müller and B. Gomes, "A Performance Analysis of CNS-1 on Sparse Connectionist Networks," Technical Report TR-94-009, Int'l Computer Science Inst., Berkeley, Calif., Feb. 1994.
21. M. Philippsen, "Automatic Alignment of Array Data and Processes to Reduce Communication Time on DMPPs." Proc. Fifth ACM SIGPLAN Symp. Principles and Practice of Parallel Programming (PPoPP), pp. 156-165, Santa Barbara, Calif., July 1995.
22. M. Philippsen, E.A. Heinz, and P. Lukowicz, "Compiling Machine-Independent Parallel Programs," ACM SIGPLAN Notices, vol. 28, no. 8 pp. 99-108, Aug. 1993.
23. L. Prechelt, "CUPIT - A Parallel Language for Neural Algorithms: Language Reference and Tutorial," Technical Report 4/94, Fakultät für Informatik, Univ., Karlsruhe, Germany, Jan. 1994, ftp://ftp.ira.uka.de/pub/papers/techreports/1994/ 1994-04.ps.gz.
24. L. Prechelt, "PROBEN1 - A Set of Benchmarks and Benchmarking Rules for Neural Network Training Algorithms," Technical Report 21/94, Fakultät für Informatik, Univ. Karlsruhe, Germany, Sept. 1994, ftp://ftp.ira.uka.de/pub/papers/ techreports/1994/1994-21.ps.gz.
25. L. Prechelt, "The CUPIT Compiler for the MasPar - A Literate Programming Document," Technical Report 1/95, Fakultät für Informatik, Univ. Karlsruhe, Germany, Jan. 1995, ftp:// ftp.ira.uka.de/pub/papers/techreports/1995/1995-01.ps.gz.
26. L. Prechelt, "A Parallel Programming Model for Irregular Dynamic Neural Networks," W.K. Giloi, S. Jähnichen, and B.D. Shriver, eds., Proc. Programming Models for Massively Parallel Computers, Berlin, Oct. 1995. GMD First, IEEE CS Press. By accident, the article was not printed in the proceedings volume, but see http://wwwipd.ira.uka.de/ ∼prechelt/Biblio/.
27. U. Ramacher, W. Raab, J. Anlauf, U. Hachmann, J. Beichter, N. Brüls, M. Weßeling, E. Sicheneder, J. Gläß, A. Würz, and R. Männer, "Synapse-1: A High-Speed General Purpose Parallel Neurocomputer System," Proc. Ninth Int'l Symp. Parallel Processing (IPPS '95), pp. 774-781, IEEE/CS Press, Los Alamitos, Calif., Apr. 1995.
28. M. Riedmiller and H. Braun, "A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm," Proc. IEEE Int'l Conf. Neural Networks, pp. 586-591, San Francisco, Apr. 1993.
29. "Application in Modelling and Simulation," Proc. 15th IMACS World Congress on Scientific Computation, Modelling, and Applied Math., vol. 6. A. Sydow, ed., Berlin: Wissenschaft und Technik Verlag, Aug. 1997.
30. T. Tollenaere and G.A. Orban, "Decomposition and Mapping of Locally Connected Layered Neural Networks on Message-Passing Multiprocessors," Parallel Algorithms and Applications, vol. 1, pp. 43-56, 1993.
31. X. Zhang, M. McKenna, J.P. Mesirov, and D. Waltz, "An Efficient Implementation of the Backpropagation Algorithm on the Connection Machine CM-2," Advances in Neural Information Processing Systems 2, D. Touretzky, ed., pp. 801-809. San Mateo, Calif.: Morgan Kaufmann, 1989.