A fast and scalable recurrent neural network based on stochastic meta descent

IEEE Transactions on Neural Networks

Volumn 19, Issue 9, 2008, Pages 1652-1658

  Liu, Zhenzhen ^a   Elhanany, Itamar ^a  

^a University of Tennessee Knoxville   (United States)

Author keywords

Constrained optimization; Real time recurrent learning RTRL); Recurrent neural networks (RNNs)

Indexed keywords

CONSTRAINED OPTIMIZATION; GRADIENT METHODS; LEARNING ALGORITHMS; STOCHASTIC SYSTEMS;

CURVATURE INFORMATION; ONLINE LEARNING ALGORITHMS; REAL TIME RECURRENT LEARNING ALGORITHMS; REAL-TIME RECURRENT LEARNING; RECURRENT NEURAL NETWORK (RNNS); RESOURCE REQUIREMENTS; STOCHASTIC GRADIENT DESCENT; STOCHASTIC META DESCENTS;

RECURRENT NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; METHODOLOGY; STATISTICAL MODEL; STATISTICS;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED; STOCHASTIC PROCESSES;

EID: 54549105599     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2008.2000838     Document Type: Article

References (21)

1. D. Nguyen and B. Widrow, "Neural networks for self-learning control system," IEEE Control Syst. Mag., vol. 10, no. 3, pp. 18-23, Apr. 1990.
2. L. Jain, Recurrent Neural Networks. Boca Raton, FL: CRC Press, 2000.
3. D. V. Prokhorov, "Training recurrent neurocontrollers for real-time applications," IEEE Trans. Neural Netw., vol. 18, no. 4, pp. 1003-1015, Jul. 2007.
4. R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks," Neural Comput., no. 1, pp. 270-280, 1989.
5. D. Zipser, "A subgrouping strategy that reduces complexity and speeds up learning in recurrent networks," Neural Comput., no. 1, pp. 552-558, 1989.
6. 2) algorithm," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Istanbul, 2000, vol. 6, pp. 3418-3421.
7. O. D. Jesus and M. T. Hagan, "Backpropagation algorithms for abroad class of dynamic networks," IEEE Trans. Neural Netw., vol. 18, no. 1, pp. 14-27, Jan. 2007.
8. 3) time complexity learning algorithm for fully recurrent continually running networks," Neural Comput., vol. 4, pp. 243-248, 1992.
9. G.-Z. Sun, H.-H. Chen, and Y.-C. Lee, "Green's function method for fast on-line learning algorithm of recurrent neural networks," in Proc. Neural Inf. Process. Syst., 1991, pp. 333-340.
10. V. S. Asirvadam, S. F. McLoone, and G. W. Irwin, "Memory efficient BFGS neural-network learning algorithms using MLP-network: A survey," in Proc. IEEE Int. Conf. Control Appl., Sep. 2004, vol. 1, pp. 586-591.
11. D. Budik and I. Elhanany, "TRTRL: A localized resource-efficient learning algorithm for recurrent neural networks," in Proc. IEEE Midwest Symp. Circuits Syst., Puerto Rico, Aug. 2006, pp. 371-374.
12. Z. Liu and I. Elhanany, "A scalable model-free recurrent neural network framework for solving POMDPs," in Proc. IEEE Int. Symp. Approx. Dyn. Programm. Reinforcement Learn., Apr. 2007, pp. 119-126.
13. S. E. Fahlman, "An empirical study of learning speed in back-propagation networks," Computer Sci. Tech. Rep., 1988.
14. N. N. Schraudolph, "Local gain adaptation in stochastic gradient descent," in Proc. 9th Int. Conf. Neural Netw., Sep. 1999, pp. 569-574.
15. N. N. Schraudolph, J. Yu, and D. Aberdeen, "Fast online policy gradient learning with SMD gain vector adaptation," in Proc. 19th Annu. Conf. Neural Inf. Process. Syst., Vancouver, BC, Canada, Dec. 2005, pp. 1185-1192.
16. J. Kivinen and M. Warmuth, "Exponentiated gradient versus gradient descent for linear predictors," Inf. Comput., no. 1, pp. 1-64, 1997.
17. B. A. Pearlmutter, "Fast exact multiplication by the Hessian," Neural Comput., vol. 6, no. 1, pp. 147-160, 1994.
18. G. D. Magoulas, M. N. Vrahatis, and G. S. Androulakis, "Improving the convergence of the backpropagation algorithm using learning rate adaptation methods," Neural Comput., vol. 11, no. 7, pp. 1769-1796, 1999.
19. T. Tollenære, "Supersab: Fast adaptive backpropagation with good scaling properties," Neural Netw., vol. 3, no. 5, pp. 561-573, 1990.
20. M. Mackey and L. Glass, "Oscillation and chaos in physiological control systems," Science, vol. 197, pp. 287-289, Jul. 1977.
21. R. Croder, III, "Predicting the Mackey-Glass time-series with cascade-correlation learning," in Connectionist Models Summer School Proceedings, D. Touretzky, G. Hinton, and T. Sejnowski, Eds. Pittsburgh, PA: Carnegie Mellon Univ., 1990, pp. 117-123.