Error surface of recurrent neural networks

IEEE Transactions on Neural Networks and Learning Systems

Volumn 24, Issue 11, 2013, Pages 1709-1721

  Phan, Manh Cong ^a   Hagan, Martin T ^a  

^a Oklahoma State University   (United States)

Author keywords

Error surface; Fibonacci polynomials; recurrent neural networks; spurious valleys

Indexed keywords

DYNAMIC NETWORK; ERROR SURFACE; FIBONACCI POLYNOMIALS; INPUT SEQUENCE; RECURRENT NETWORKS; SINGLE LAYER; SPURIOUS VALLEYS;

LANDFORMS; RECURRENT NEURAL NETWORKS;

NETWORK LAYERS;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; FEEDBACK SYSTEM; PROCEDURES; STATISTICAL MODEL;

ALGORITHMS; COMPUTER SIMULATION; FEEDBACK; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED;

EID: 84886953036     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2013.2258470     Document Type: Article

References (29)

1. A. F. Atiya and A. G. Parlos, "New results on recurrent network training: Unifying the algorithms and accelerating convergence," IEEE Trans. Neural Netw., vol. 11, no. 3, pp. 697-709, May 2000
2. M. Gori, B. Hammer, P. Hitzler, and G. Palm, "Perspectives and challenges for recurrent neural network training," Logic J. IGPL, vol. 18, no. 5, pp. 617-619, 2010
3. K. Doya, "Bifurcations in the learning of recurrent neural networks," in Proc. IEEE Int. Symp. Circuits Syst., vol. 6. May 1992, pp. 2777-2780
4. F. Pasemann, "Dynamics of a single model neuron," Int. J. Bifurcat. Chaos, vol. 3, no. 2, pp. 271-278, 1993
5. R. Haschke and J. Steil, "Input space bifurcation manifolds of recurrent neural networks," Neurocomputing, vol. 64, pp. 25-38, Mar. 2005 (Pubitemid 40425310)
6. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157-166, Mar. 1994
7. P. Frasconi, M. Gori, and G. Soda, "Local feedback multilayer networks," Neural Comput., vol. 4, no. 1, pp. 120-130, 1991
8. P. Frasconi, M. Gori, M. Maggini, and G. Soda, "Unified integration of explicit knowledge and learning by example in recurrent networks," IEEE Trans. Knowl. Data Eng., vol. 7, no. 2, pp. 340-346, Apr. 1995
9. T. Lin, B. Horne, P. Tino, and C. Giles, "Learning long-term dependencies in NARX recurrent networks," IEEE Trans. Neural Netw., vol. 7, no. 6, pp. 1329-1338, Nov. 1996 (Pubitemid 126780474)
10. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, Nov. 1997 (Pubitemid 127462305)
11. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber, "Gradient flow in recurrent nets: The difficulty of learning long-term dependencies," in A Field Guide to Dynamical Recurrent Neural Networks, S. C. Kremer and J. F. Kolen, Eds. Piscataway, NJ, USA: IEEE Press, 2001
12. A. Schaefer, S. Udluft, and H. Zimmermann, "Learning long-term dependencies with recurrent neural networks," Neurocomputing, vol. 71, nos. 13-15, pp. 2481-2488, 2008
13. D. P. Mandic and J. A. Chambers, Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability. New York, NY, USA: Wiley, 2001
14. O. D. Jesus, J. Horn, and M. T. Hagan, "Analysis of recurrent network training and suggestions for improvements," in Proc. Int. Joint Conf. Neural Netw., Jul. 2001, pp. 2632-2637 (Pubitemid 32794725)
15. J. Horn, O. D. Jesus, and M. T. Hagan, "Spurious valleys in the error surface of recurrent networks-Analysis and avoidance," IEEE Trans. Neural Netw., vol. 20, no. 4, pp. 686-700, Apr. 2009
16. A. I. Hanna, I. R. Krcmar, and D. P. Mandic, "Perlustration of error surfaces for nonlinear stochastic gradient descent algorithms," in Proc. 6th Seminar Neural Netw. Appl. Electr. Eng., Sep. 2002, pp. 11-16
17. K. Gouhara, T. Watanabe, and Y. Uchikawa, "Learning process of recurrent neural networks," in Proc. Int. Joint Conf. Neural Netw., Nov. 1991, pp. 746-751
18. K. Gouhara, K. Yokoi, and Y. Uchikawa, "Valley searching method for recurrent neural networks," in Proc. Int. Joint Conf. Neural Netw., Jun. 1992, pp. 972-979
19. K. Yokoi, K. Gouhara, and Y. Uchikawa, "Comparative study of searching method in recurrent neural networks," Syst. Comput. Jpn., vol. 27, no. 12, pp. 22-32, 1996
20. C.-T. Chen, Linear System Theory and Design. Oxford, U.K.: Oxford Univ. Press, 1998
21. A. V. Oppenheim, A. S. Willsky, and S. Hamid, Signals and Systems. Englewood Cliffs, NJ, USA: Prentice-Hall, 1997
22. B. A. Brousseau, Linear Recursion and Fibonacci Sequences. San Jose, CA, USA: Fibonacci Association, 1971
23. K. Ogata, Discrete-Time Control Systems, 2nd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 1995
24. M. T. Hagan, H. B. Demuth, and O. D. Jesus, "An introduction to the use of neural networks in control systems," Int. J. Robust Nonlinear Control, vol. 12, no. 11, pp. 959-985, 2002
25. V. E. Hoggatt, "Divisibility properties of generalized Fibonacci polynomials," Fibonacci Quart., vol. 12, no. 2, pp. 113-120, Apr. 1974
26. O. D. Jesus and M. T. Hagan, "Backpropagation algorithms for a broad class of dynamic networks," IEEE Trans. Neural Netw., vol. 18, no. 1, pp. 14-27, Jan. 2007 (Pubitemid 46062913)
27. N. H. Nguyen and M. T. Hagan, "Stability analysis of layered digital dynamic networks using dissipativity theory," in Proc. Int. Joint Conf. Neural Netw., Aug. 2011, pp. 1692-1699
28. R. Jafari and M. T. Hagan, "Global stability analysis using the method of reduction of dissipativity domain," in Proc. Int. Joint Conf. Neural Netw., Aug. 2011, pp. 2550-2556
29. P. Guillaume, J. Schoukens, and R. Pintelon, "Sensitivity of roots to errors in the coefficient of polynomials obtained by frequency-domain estimation methods," IEEE Trans. Instrum. Meas., vol. 38, no. 6, pp. 1050-1056, Dec. 1989. (Pubitemid 20642760)