Fast Training Algorithms for Multilayer Neural Nets

IEEE Transactions on Neural Networks

Volumn 2, Issue 3, 1991, Pages 346-354

  Brent, Richard P ^a  

^a AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

DECISION THEORY AND ANALYSIS; LEARNING SYSTEMS; OPTIMIZATION; PATTERN RECOGNITION; SPEECH - RECOGNITION;

CANBERRA SPEECH DATA; CONTINUOUS OPTIMIZATION PROBLEM; DISCRETE OPTIMIZATION PROBLEM; FAST TRAINING ALGORITHMS; MULTILAYER NEURAL NETS; PARITY PROBLEM;

NEURAL NETWORKS;

EID: 0026154952     PISSN: 10459227     EISSN: 19410093     Source Type: Journal    
DOI: 10.1109/72.97911     Document Type: Article

References (58)

1. F. Rosenblatt, Perceptrons and the Theory of Brain Mechanisms. New York: Spartan Books, 1962.
2. R. P. Lippmann, “An introduction to computing with neural nets,” IEEE ASSP Magazine, pp. 4–22, Apr. 1987.
3. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processings vol. 1, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, 1986, pp. 318–362.
4. R. P. Lippmann, “Pattern classification using neural networks,” IEEE Communications Magazine, 47–50, 59–64, Nov. 1989.
5. L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification and Regression Trees. Belmont, CA: Wadsworth International. 1984.
6. J. R. Quinlan, “Induction of decision trees,” Machine Learning, vol. 1, pp. 81–106, 1986.
7. A. K. Jain, “Advances in statistical pattern recognition,” in Pattern Recognition Theory and Applications (P. A. Devijver and J. Kittler, Eds. New York: Springer Verlag, 1989, pp. 1–19.
8. I. K. Sethi, “Entropy nets: From decision trees to neural networks,” Proc. IEEE, vol. 78, pp. 1605–1613, 1990.
9. P. E. Utgoff. “Perceptron trees: A case study in hybrid concept representations.” Connection Sci., vol. 1, no. 4, pp. 377–391, 1989.
10. G. Cybenko, “Continuous valued neural networks: Approximation theoretic results,” in Proc. 20th Symp. Interface (Alexandria, VA), 1988, pp. 174–183.
11. K. Funahashi, “On the approximate realization of continuous mappings by neural networks,” Neural Networks, vol. 2, pp. 183–192, 1989.
12. A. N. Kolmogorov, “On the representation of continuous functions of several variables by superpositions of continuous functions of one continuous variable and addition,” Dokl. Akad. Nauk SSSR, vol. 114, pp. 679–681, 1957 (AMS Translation, no. 28, 55–59, 1963).
13. G. G. Lorentz, “The 13-th problem of Hilbert,” in Proc. Symp. Pure Math., 1976, pp. 419–430.
14. G. J. Gibson and C. F. N. Cowan, “On the decision regions of multilayer perceptrons,” Proc. IEEE, vol. 78, pp. 1590–1594, 1990.
15. R. L. Harvey, “An introduction to multi-layer perceptrons,” Project Report NN-1. MIT Lincoln Lab Nov 1988
16. C. Koutsougeras and C. A. Papachristou, “Training of a neural network for pattern classification based on an entropy measure,” in Proc. 1988 IEEE Conf. Neural Networks, July 1988, vol. 1, pp. 247–254.
17. M. Bichsel and P. Seitz, “Minimum class entropy: A maximum information approach to layered networks,” Neural Networks, vol. 2. pp. 133–141, 1989.
18. R. E. Blahut, Theory and Practice of Error Control Codes. Reading, MA: Addison-Wesley, 1983.
19. R. P. Brent, Algorithms for Minimization without Derivatives. Englewood Cliffs, NJ: Prentice-Hall, 1973.
20. J. Dennis and R. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Englewood Cliffs, NJ: Prentice-Hall, 1983.
21. J. Kowalik and M. R. Osborne, Methods for Unconstrained Optimization Problems. New York: American Elsevier, 1968.
22. G. E. Forsythe, “On the asymptotic directions of the s-dimensional optimum gradient method,” Numer. Math., vol. 11, pp. 57–76, 1968.
23. R. Fletcher and C. M. Reeves, “Function minimization by conjugate gradients,” Computer J., vol. 7, pp. 149–154, 1964.
24. E. Barnard and R. A. Cole, “A neural-net training program based on conjugate-gradient optimization,” Tech. Report CSE 89–014, Oregon Graduate Center, Beaverton, OR, July 1989.
25. R. L. Watrous, “Learning algorithms for connectionist networks: Applied gradient methods of nonlinear optimization,” in Proc. 1987 IEEE Conf. Neural Networks, vol. 2, pp. 619–627.
26. P. J. Werbos, “Backpropagation: Past and future,” in Proc. 1988 IEEE Conf. Neural Networks, July 1988, vol. 1, pp. 343–353.
27. S. Becker and Y. le Cun, “Improving the convergence of back-propagation learning with second order methods,” Department of Computer Science, University of Toronto, 1987.
28. S. M. Omohundro, “Efficient algorithms with neural network behaviour,” Complex Systems, vol. 1, pp. 273–347, 1987.
29. J. L. Bentley, “Multidimensional divide and conquer,” Commun. Ass. Comput. Mach., vol. 23, pp. 214–229, 1980.
30. J. R. Quinlan, “Simplifying decision trees,” Int. J. Man-Machine Studies, vol. 27, 221–234, 1987.
31. R. E. Bellman, Adaptive Control Processes. Princeton, NJ: Princeton Univ. Press, 1961.
32. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973.
33. A. D. Gordon, Classification—Methods for the Exploratory Analysis of Multivariate Data. London and New York: Chapman and Hall, 1981.
34. G. H. Golub and C. Van Loan, Matrix Computations. Baltimore, MD: Johns Hopkins Press, 1983.
35. L. Atlas et al., “A performance comparison of trained multilayer perceptrons and trained classification trees,” Proc. IEEE, vol. 78, pp. 1614–1619, 1990.
36. R. Fletcher, “Conjugate direction methods,” in Numerical Methods for Unconstrained Optimization. W. Murray, Ed. London and New York: Academic Press, 1972, pp. 73–86.
37. J. Daniel, The Approximate Minimization of Functionals. Englewood Cliffs, NJ: Prentice-Hall, 1971.
38. D. Shanno, “Conjugate gradient methods with inexact line searches,” Math. Operations Res., vol. 3. pp. 249–256, 1978.
39. H. Crowder and P. Wolfe, “Linear convergence of the conjugate gradient method,” IBM J. Res. Develop., vol. 16, pp. 431–433, 1972.
40. D. H. Deterding, “Speaker normalisation for automatic speech recognition,” Ph.D. thesis, Cambridge University, Cambridge, UK, 1989.
41. J. B. Millar, M. O’Kane, and P. Bryant, “Design, collection, and description of a database of spoken Australian English,” Aust. J. Linguistics, vol. 9, pp. 165–189, 1989.
42. A. J. Robinson, “Dynamic error propagation networks,” Ph.D. thesis, Department of Engineering, Cambridge University. Cambridge, UK, 1989.
43. S. E. Fahlman, NN-Bench. Test data available via electronic mail, Aug. 1989.
44. J. Schwartz, data distributed by electronic mail, Dec. 1988.
45. M. Minsky and S. Papert, Perceptrons: An Introduction to Computational Geometry. Cambridge, MA: MIT Press, 1969.
46. N. J. Nilsson, Learning Machines. New York: McGraw-Hill, 1965.
47. S. E. Fahlman, “An empirical study of learning speed in back-propagation networks,” Report CMU-CS-88-162, Dept. Computer Science, Carnegie-Mellon Univ., Pittsburgh, PA, June 1988.
48. G. E. Hinton, “Connectionist learning procedures,” Tech. Report CMU-CS-87-115 (version 2), Dept. Computer Science, Carnegie-Mellon University, Pittsburgh, PA, Dec. 1987.
49. J. J. Hopfield, “Learning algorithms and probability distributions in feed-forward and feed-back networks,” Proc. Acad. Sci. U.S., vol. 84, pp. 8429–8433, 1987.
50. A. R. Barron and R. L. Barron, “Statistical learning networks: A unifying view,” in Proc. 20th Symp. Interface (Alexandria, VA), 1988, p. 192–203.
51. Y. le Cun, “A theoretical framework for back-propagation,” Department of Computer Science, University of Toronto, 1987.
52. W. Y. Huang and R. P. Lippmann, “Neural net and traditional classifiers,” in Neural Information Processing Systems, D. Z. Anderson, Ed. New York: American Institute of Physics, 1988, 387–396.
53. K. M. Passino, M. A. Sartori and P. J. Antsaklis, “Neural computing for numeric-to-symbolic conversion in control systems,” IEEE Control Systems Magazine, vol. 9, no. 3, pp. 44–51, Apr. 1989.
54. D. E. Rumelhart et al., Eds., Parallel Distributed Processing, vol. 1. Cambridge, MA: MIT Press, 1986.
55. D. Z. Anderson, Ed., Neural Information Processing Systems. New York: American Institute of Physics, 1988.
56. F. Crick, “The recent excitement about neural networks,” Nature, vol. 337, pp. 129–132, Jan. 1989.
57. R. P. Brent, “Fast training algorithms for multi-layer neural nets,” Technical Report NA-90-03, Department of Computer Science, Stanford University, Mar. 1990. Extended abstract in Proc. First Australian Conf. Neural Networks (University of Sydney Australia), Jan. 1990, 97–98.
58. E. J. Wegman, D. T. Gantz, and J. J. Miller, Eds., Computing Science and Statistics, Proceedings of the 20th Symposium on the Interface, Amer. Statist. Assn., Alexandria, VA, 1988.