Use of bias term in projection pursuit learning improves approximation and convergence properties

IEEE Transactions on Neural Networks

Volumn 7, Issue 5, 1996, Pages 1168-1183

  Kwok, Tin Yau ^b   Yeung, Dit Yan ^a,b  

^a IEEE

^b HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION THEORY; CONVERGENCE OF NUMERICAL METHODS; LEARNING ALGORITHMS; OPTIMIZATION; REGRESSION ANALYSIS; TIME SERIES ANALYSIS; TRANSFER FUNCTIONS;

BIAS TERMS; CASCADE CORRELATION ARCHITECTURE; HERMITE FUNCTIONS; PROJECTION PURSUIT LEARNING; TWO LAYER FEEDFORWARD NEURAL NETWORKS;

NEURAL NETWORKS;

EID: 0030244218     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.536312     Document Type: Article

References (49)

1. B. Hassibi and D. G. Stork, "Second-order derivatives for network pruning: Optimal brain surgeon," in Advances in Neural Information Processing Systems 5. San Mateo, CA: Morgan Kaufmann, 1993, pp. 164-171.
2. E. D. Karnin, "A simple procedure for pruning backpropagation trained neural networks," IEEE Trans. Neural Networks, vol. 1, pp. 239-242, June 1990.
3. Y. Le Cun, J. S. Denker, and S. A. Solla, "Optimal brain damage," in Advances in Neural Information Proccessing Systems 2, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 598-605.
4. M. C. Mozer and P. Smolensky, "Skeletonization: A technique, for trimming the fat from a network via relevance assessment," in Advances in Neural Information Processing Systems 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 107-115.
5. R. Reed, "Pruning algorithms - A survey," IEEE Trans. Neural Networks, vol. 4, pp. 740-747, Sept. 1993.
6. T. Ash, "Dynamic node creation in backpropagation networks," Connection Sci., vol. 1, no. 4, pp. 365-375, 1989.
7. S. E. Fahlman and C. Lebiere, "The cascade-correlation learning architecture," in Advances in Neural Information Processing Systems 2, D. S. Touretzky, Ed. Los Altos, CA: Morgan Kaufmann, 1990, pp. 524-532.
8. Y. Hirose, K. Yamashita, and S. Hijiya, "Backpropagation algorithm which varies the number of hidden units," Neural Networks, vol. 4, pp. 61-66, 1991.
9. J. N. Hwang, S. R. Lay, M. Maechler, D. Martin, and J. Schimert, "Regression modeling in backpropagation and projection pursuit learning," IEEE Trans. Neural Networks, vol. 5, pp. 342-353, 1994.
10. S. Sjøgaard, "A conceptual approach to generalization in dynamic neural networks," Ph.D. dissertation, Comput. Sci. Dep., Aarhus Univ., Aarhus C, Denmark, 1991.
11. D. Y. Yeung, "Constructive neural networks as estimators of Bayesian discriminant functions," Pattern Recognition, vol. 26, no. 1, pp. 189-204, 1993.
12. Y. Chauvin, "A backpropagation algorithm with optimal use of hidden units," in Advances in Neural Information Processing Systems 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 519-526.
13. S. J. Hanson and L. Y. Pratt, "Comparing biases for minimal network construction with backpropagation," in Advances in Neural Information Processing Systems 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 177-185.
14. A. S. Weigend, D. E. Rumelhart, and B. A. Huberman, "Generalization by weight-elimination with application to forecasting," in Advances in Neural Information Processing Systems 3, R. Lippmann, J. Moody, and D. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, pp. 875-882.
15. E. Hartman and J. D. Keeler, "Predicting the future: Advantages of semilocal units," Neural Computa., vol. 3, pp. 566-578, 1991.
16. G. W. Flake, "Nonmonotonic activation functions in multilayer perceptrons," Ph.D. dissertation, Inst. Advanced Comput. Studies, Dep. Cornput. Sci., Univ. Maryland, College Park, MD, 1993.
17. J. Moody and N. Yarvin, "Networks with learned unit response functions," in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 1048-1055.
18. Y. Zhao and C. G. Atkeson, "Projection pursuit learning," in Proc. Int. Joint Conf. Neural Networks, Seattle, WA, July 1991, vol. 1, pp. 869-874.
19. J. N. Hwang, H. Li, M. Maechler, R. D. Martin, and J. Schimert, "A comparison of projection pursuit and neural network regression modeling," in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 1159-1166.
20. J. N. Hwang, H. Li, D. Martin, and J. Schimert, "The learning parsimony of projection pursuit and backpropagation networks," in Conf. Record 25th Asilomar Conf. Signals, Syst., Comput., vol. 1, Pacific Grove, CA, Nov. 1991, pp. 491-495.
21. J. N. Hwang, S. S. You, S. R. Lay, and I. C. Jou, "What's wrong with a cascaded correlation learning network? A projection pursuit learning perspective," submitted to IEEE Trans. Neural Networks.
22. M. Maechler, D. Martin, J. Schimert, M. Csoppenszky, and J. N. Hwang, "Projection pursuit learning networks for regression," in Proc. 2nd Int. IEEE Conf. Tools Artificial Intell., Herndon, VA, Nov. 1990, pp. 350-358.
23. J. H. Friedman and W. Stuetzle, "Projection pursuit regression," J. Amer. Stat. Assoc., vol. 76, no. 376, pp. 817-823, 1981.
24. R. E. Bellman, Adaptive Control Processes. Princeton, NJ: Princeton Univ. Press, 1961.
25. T. E. Flick, L. K. Jones, R. G. Priest, and C. Herman, "Pattern classification using projection pursuit," Pattern Recognition, vol. 23, no. 12, pp. 1367-1376, 1990.
26. C. B. Roosen and T. J. Hastie, "Logistic response projection pursuit," AT&T Bell Lab., Murray Hill, NJ, Tech. Rep. BL011214-930806-09TM, Aug. 1993.
27. J. H. Friedman, "A variable span smoother," Dep. Statistics, Stanford Univ., Stanford, CA, Tech. Rep. 5, 1984.
28. C. B. Roosen and T. J. Hastie, "Automatic smoothing spline projection pursuit," J. Computa. Graphical Statist., vol. 3, no. 3, pp. 235-248, 1994.
29. T. J. Hastie and R. J. Tibshirani, "Generalized additive models," in Monographs on Statistics and Applied Probability 43. New York: Chapman and Hall, 1990.
30. G. Sansone, Orthogonal Functions. New York: Dover, 1991.
31. J. H. Friedman, "Exploratory projection pursuit," J. Amer. Statist. Assoc., vol. 82, no. 397, pp. 249-266, Mar. 1987.
32. J. H. Friedman, E. Grosse, and W. Stuetzle, "Multidimensional additive spline approximation," SIAM J. Sci. Statist. Comput., vol. 4, no. 2, pp. 291-301, June 1983.
33. A. Saha, C. L. Wu, and D. S. Tang, "Approximation, dimension reduction, and nonconvex optimization using linear superpositions of Gaussians," IEEE Trans. Comput., vol. 42, no. 10, pp. 1222-1233, Oct. 1993.
34. S. Geman, E. Bienenstock, and R. Doursat, "Neural networks and the bias/variance dilemma," Neural Computa., vol. 4, pp. 1-58, 1992.
35. A. R. Barron, "Approximation and estimation bounds for artificial neural networks," in Proc. 4th Annu. Wkshp. Computa. Learning Theory, Santa Cruz, CA, Aug. 1991, pp. 243-249.
36. F. Riesz and B. S.-Nagy, Functional Analysis. New York: Dover, 1990.
37. L. K. Jones, "On a conjecture of Huber concerning the convergence of projection pursuit regression," Ann. Statist., vol. 15, no. 2, pp. 880-882, 1987.
38. P. J. Huber, "Projection pursuit (with discussion)," Ann. Statist., vol. 13, no. 2, pp. 435-525, 1985.
39. S. E. Fahlman, "Faster learning variations on backpropagation: An empirical study," in Proc. 1988 Connectionist Models Summer School, D. S. Touretzky, G. E. Hinton, and T. J. Sejnowski, Eds. San Mateo, CA: Morgan Kaufmann, 1988, pp. 38-51.
40. H. Akaike, "A new look at the statistical model identification," IEEE Trans. Automat. Contr., vol. AC-19, pp. 716-723, Dec. 1974.
41. K. Hornik, "Some new results on neural network approximation," Neural Networks, vol. 6, pp. 1069-1072, 1993.
42. _, "Approximation capabilities of multilayer feedforward networks," Neural Networks, vol. 4, pp. 251-257, 1991.
43. L. K. Jones, "A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training," Ann. Statist., vol. 20, no. 1, pp. 608-613, 1992.
44. R. S. Crowder, "Predicting the Mackey-Glass timeseries with cascade-correlation learning," in Proc. Cunnectionist Models Summer School, 1990, pp. 117-123.
45. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in C, 2nd ed. Cambridge, U.K.: Cambridge Univ. Press, 1992.
46. V. K. Rohatgi, Statistical Inference. New York: Wiley, 1984.
47. A. Lapedes and F. Farber, "Nonlinear signal processing using neural networks: Prediction and system modeling," Los Alamos Nat. Lab., LA-UR 87-2662, 1987.
48. J. Moody and C. Darken, "Learning with localized receptive fields," in Proc. 1988 Connectionlist Models Summer School, 1988, pp. 133-143.
49. J. Park, and I. Sandberg, "Universal approximation using radial basis function networks," Neural Computa., vol. 3, pp. 246-257, 1991.