A Smoothing Regularizer for Feedforward and Recurrent Neural Networks

Neural Computation

Volumn 8, Issue 3, 1996, Pages 461-489

  Wu, Lizhong ^a   Moody, John ^a  

^a OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 2342475914     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/neco.1996.8.3.461     Document Type: Article

References (43)

1. Abu-Mostafa, Y. 1995. Hints. Neural Comp. 7(4), 639-671.
2. Amari, S. 1972. Characteristics of random nets of analog neural-like elements. IEEE Trans. Systems, Man Cybern. SMC-2, 643-653.
3. Bishop, C. 1993. Curvature-driven smoothing: A learning algorithm for feed-forward networks. IEEE Trans. Neural Networks 4(5), 882-884.
4. Bishop, C. 1995. Training with noise is equivalent to Tikhonov regularization. Neural Comp. 7(1), 108-116.
5. Chauvin, Y. 1990. Dynamic behavior of constrained back-propagation networks. In Advances in Neural Information Processing Systems 2, D. Touretzky, ed. pp. 642-649. Morgan Kaufmann, San Mateo, CA.
6. Elman, J. 1990. Finding structure in time. Cog. Sci. 14, 179-211.
7. Eubank, R. L. 1988. Spline Smoothing and Nonparametric Regression. Marcel Dekker, New York.
8. Geman, S., Bienenstock, E., and Doursat, R. 1992. Neural networks and the bias/variance dilemma. Neural Comp. 4(1), 1-58.
9. Girosi, F., Jones, M., and Poggio, T. 1995. Regularization theory and neural networks architectures. Neural Comp. 7, 219-269.
10. Grossberg, S. 1969. On learning and energy-entropy dependence in recurrent and nonrecurrent signed networks. J. Stat. Phys. 1, 319-350.
11. Hastie, T. J., and Tibshirani, R. J. 1990. Generalized Additive Models, Vol. 43 of Monographs on Statistics and Applied Probability. Chapman and Hall, London.
12. Hirsch, M. 1989. Convergent activation dynamics in continuous time networks. Neural Networks 2(5), 331-349.
13. Hoerl, A., and Kennard, R. 1970a. Ridge regression: Applications to nonorthogonal problems. Technometrics 12, 69-82.
14. Hoerl, A., and Kennard, R. 1970b. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12, 55-67.
15. Hopfield, J. 1984. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. U.S.A. 81, 3088-3092.
16. Korn, G., and Korn, T., eds. 1968. Mathematical Handbook for Scientists and Engineers. McGraw-Hill, New York.
17. Kuan, C., Hornik, K., and White, H. 1994. A convergence result for learning in recurrent neural networks. Neural Comp. 6(3), 420-440.
18. Leen, T. 1995. From data distributions to regularization in invariant learning, Neural Comp. 7(5), 974-981.
19. Levin, A. U., Leen, T. K., and Moody, J. E. 1994. Fast pruning using principal components. In Advances in Neural Information Processing Systems 6, J. Cowan, G. Tesauro, and J. Alspector, eds., pp. 35-42. Morgan Kaufmann, San Mateo, CA.
20. Marcus, C., and Westervelt, R. 1989. Dynamics of analog neural networks with time delay. In Advances in Neural Information Processing Systems 1, D. Touretzky, ed., Morgan Kaufmann, San Mateo, CA.
21. Moody, J., and Rögnvaldsson, T. 1995. Smoothing regularizers for feed-forward neural networks. Tech. Rep., Oregon Graduate Institute.
22. Moody, J. E., and Utans, J. 1992. Principled architecture selection for neural networks: Application to corporate bond rating prediction. In Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, eds., pp. 683-690. Morgan Kaufmann, San Mateo, CA.
23. Moody, J. E., and Yarvin, N. 1992. Networks with learned unit response functions. In Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, eds., pp. 1048-1055, Morgan Kaufmann, San Mateo, CA.
24. Moody, J., Levin, U., and Rehfuss, S. 1993. Predicting the U.S. index of industrial production. In Proceedings of the 1993 Parallel Applications in Statistics and Economics Conference, Zeist, the Netherlands. Special issue of Neural Network World 3(6), 791-794.
25. Nowlan, S., and Hinton, G. 1992. Simplifying neural networks by soft weight-sharing. Neural Comp. 4(4), 473-493.
26. Pearlmutter, B. 1989. Learning state space trajectories in recurrent neural networks. Neural Comp. 1(2), 261-269.
27. Pineda, F. 1988. Dynamics and architecture for neural computation. J. Complexity 4, 216-245.
28. Pineda, F. 1989. Recurrent backpropagation and the dynamical approach to adaptive neural computation. Neural Comp. 1(2), 161-172.
29. Flaut, D., Nowlan, S., and Hinton, G. 1986. Experiments on Learning by Back Propagation. Tech. Rep. CMU-CS-86-126, Carnegie Mellon University, Pittsburgh, PA.
30. Poggio, T., and Girosi, F. 1990. Networks for approximation and learning. IEEE Proc. 78(9), 1481-1497.
31. Powell, M. 1987. Radial basis functions for multivariable interpolation: A review. In Algorithms for Approximation, J. Mason and M. Cox, eds. Clarendon Press, Oxford.
32. Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Exploration in the Microstructure of Cognition, D. Rumelhart and J. McClelland, eds., Chap. 8, pp. 319-362. MIT Press, Cambridge, MA.
33. Scalettar, R., and Zee, A. 1988. Emergence of grandmother memory in feed forward networks: Learning with noise and forgetfulness. In Connectionist Models and Their Implications: Readings from Cognitive Science, D. Waltz and J. Feldman, eds. Ablex.
34. Sejnowski, T. 1977. Storing covariance with nonlinearly interacting neurons. J. Math. Biol. 4, 303-321.
35. Shao, M., and Nikias, C. 1993. Signal processing with fractional lower order moments: Stable processes and their aplications. IEEE Proc. 81(7), 986-1010.
36. Sjöberg, J., and Ljung, L. 1992. Overtraining, regularization and searching for minimum in neural nets. In Preprint 4th IFAC Symposium on Adaptive Systems in Control and Signal Processing, pp. 669-674.
37. Sjöberg, J., and Ljung, L. 1995. Overtraining, regularization and searching for minimum with application to neural nets. International J. Control (submitted).
38. Tikhonov, A. N., and Arsenin, V. I. 1977. Solutions of Ill-Posed Problems. Winston, New York. Distributed solely by Halsted Press. Scripta series in mathematics. Translation editor, Fritz John.
39. Tresp, V., Hollatz, J., and Ahmad, S. 1993. Network structuring and training using rule-based knowledge. In Advances in Neural Information Processing Systems 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, eds., pp. 871-878. Morgan Kaufmann, San Mateo, CA.
40. Wahba, G. 1990. Spline Models for Observational Data. CBMS-NSF Regional Conference Series in Applied Mathematics.
41. Weigend, A., Rumelhart, D., and Huberman, B. 1990. Back-propagation, weight-elimination and time series prediction. In Proceedings of the Connectionist Models Summer School, T. Sejnowski, G. Hinton, and D. Touretzky, eds., pp. 105-116. Morgan Kaufmann, San Mateo, CA.
42. Werbos, P. 1992. Neurocontrol and supervised learning: An overview and evaluation. In Handbook of Intelligent Control, D. White and D. Sofge, eds., Van Nostrand Reinhold, New York.
43. Werbos, P., and Titus, J. 1978. An empirical test of new forecasting methods derived from a theory of intelligence: The prediction of conflict in Latin America. IEEE Trans. Syst. Man Cybern. SMC-8(9), 657-666.