Simultaneous perturbation learning rule for recurrent neural networks and its FPGA implementation

IEEE Transactions on Neural Networks

Volumn 16, Issue 6, 2005, Pages 1664-1672

  Maeda, Yutaka ^a   Wakamura, Masatoshi ^a,b  

^a KANSAI UNIVERSITY   (Japan)

^b PANASONIC CORPORATION   (Japan)

Author keywords

Field programmable gate array (FPGA) implementation; Hopfield neural networks (HNNs); Recurrent neural networks (RNNs); Recursive learning; Simultaneous perturbation

Indexed keywords

CORRELATION METHODS; ERROR ANALYSIS; FIELD PROGRAMMABLE GATE ARRAYS; LEARNING SYSTEMS; PERTURBATION TECHNIQUES; VECTORS;

FIELD PROGRAMMABLE GATE ARRAY (FPGA) IMPLMENTATION; HOPFIELD NEURAL NETWORKS; RECURSIVE LEARNING; SIMULTANEOUS PERTURBATION;

RECURRENT NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; EQUIPMENT; EQUIPMENT DESIGN; FEEDBACK SYSTEM; INSTRUMENTATION; METHODOLOGY; SEMICONDUCTOR; SIGNAL PROCESSING;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; EQUIPMENT DESIGN; EQUIPMENT FAILURE ANALYSIS; FEEDBACK; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED; SIGNAL PROCESSING, COMPUTER-ASSISTED; TRANSISTORS;

EID: 28344446251     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2005.852237     Document Type: Article

References (28)

1. J. J. Hopfield, "Neurons with graded response have collective computational properties like those of two-state neuron," in Proc. Nat. Acad. Sci., vol. 81, 1984, pp. 3088-3092.
2. B. Kosko, Neural Networks and Fuzzy Systems. Englewood Cliffs, NJ: Prentice-Hall, 1992.
3. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing. Cambridge, MA: MIT Press, 1986, vol. 1.
4. B. J. Sheu and J. Choi, Neural Information Processing and VLSI. Norwell, MA: Kluwer, 1995.
5. S. M. Fakhraie, "Scalable closed-boundary analog neural networks," IEEE Trans. Neural Netw., vol. 15, pp. 492-504, Mar. 2004.
6. L. M. Reyneri, "Implementation issues of neuro-fuzzy hardware: going toward HW/SW codesign," IEEE Trans. Neural Netw., vol. 14, pp. 176-194, Jan. 2003.
7. R. Genov and G. Cauwenberghs, "Kemeltron: support vector "machine" in silicon," IEEE Trans. Neural Netw., vol. 14, pp. 1426-1434, Sep. 2003.
8. T. Morie and Y. Amemiya, "An all-analog expandable neural network LSI with on-chip backpropagation learning," IEEE J. Solid-State Circuits, vol. 29, pp. 1086-1093, Sep. 1994.
9. T. Morie, O. Fujita, and K. Uchimura, "Self-learning analog neural network LSI with high-resolution nonvolatile analog memory and a partially-serial weight-update architecture," IEICE Trans. Electron., vol. E80-C, no. 7, pp. 990-995, 1997.
10. J. C. Spall, "A stochastic approximation technique for generating maximum likelihood parameter estimates," in Proc. Amer. Control Conf., 1987, pp. 1161-1167.
11. J. C. Spall, "Multivariable stochastic approximation using a simultaneous perturbation gradient approximation," IEEE Trans. Autom. Control, vol. 37, pp. 332-341, Mar. 1992.
12. J. Alespector, R. Meir, B. Yuhas, A. Jayakumar, and D. Lippe, "A parallel gradient descent method for learning in analog VLSI neural networks," in Advances in Neural Information Processing Systems 5, S. J. Hanson, J. D. Cowan, and C. Lee, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 836-844.
13. G. Cauwenberghs, "A fast stochastic error-descent algorithm for supervised learning and optimization," in Advances in Neural Information Processing Systems 5, S. J. Hanson, J. D. Cowan, and C. Lee, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 244-251.
14. Y. Maeda and Y. Kanata, "Learning rules for recurrent neural networks using perturbation and their application to neuro-control," Trans. Inst. Elect. Eng. Jpn., vol. 113-C, no. 6, pp. 402-408, 1993.
15. Y. Maeda and Y. Kanata, "A learning rule of neural networks for neuro-controller," in Proc. World Congr. Neural Networks, vol. 2, 1995, pp. II-402-II-405.
16. Y. Maeda and R. J. P. de Figueiredo, "Learning rules for neuro-controller via simultaneous perturbation," IEEE Trans. Neural Netw., vol. 8, pp. 1119-1130, Sep. 1997.
17. Y. Maeda, H. Hirano, and Y. Kanata, "A learning rule of neural networks via simultaneous perturbation and its hardware implementation," Neural Netw., vol. 8, no. 2, pp. 251-259, 1995.
18. G. Cauwenberghs, "An analog VLSI recurrent neural network learning a continuous-time trajectory," IEEE Trans. Neural Netw., vol. 7, pp. 346-361, Mar. 1996.
19. Y. Maeda, A. Nakazawa, and Y. Kanata, "Hardware implementation of a pulse density neural network using simultaneous perturbation learning rule," Analog Integr. Circuits Signal Process., vol. 18, no. 2, pp. 1-10, 1999.
20. Y. Maeda and T. Tada, "FPGA implementation of a pulse density neural network with learning ability using simultaneous perturbation," IEEE Trans. Neural Netw., vol. 14, pp. 688-695, May 2003.
21. S. Hauck "The role of FPGA's in reprogrammable systems," Proc. IEEE, vol. 86, no. 4, pp. 615-638, Apr. 1998.
22. H. Hikawa, "Frequency-based multilayer neural networks with on-chip learning and enhanced neuron characteristics," IEEE Trans. Neural Netw., vol. 10, pp. 545-553, May 1999.
23. H. Hikawa, "A new digital pulse-mode neuron with adjustable activation function," IEEE Trans. Neural Netw., vol. 14, pp. 236-242, Jan. 2003.
24. D. Anguita, A. Boni, and S. Ridella, "A digital architecture for support vector machines: Theory, algorithm and FPGA implementation," IEEE Trans. Neural Netw., vol. 14, pp. 993-1009, Sep. 2003.
25. F. Yang and M. Paindavoine, "Implementation of an RBF neural network on embedded systems: Real-time face tracking and identity verification," IEEE Trans. Neural Netw., vol. 14, pp. 1162-1175, Sep. 2003.
26. C. T. Yen, W. Weng, and Y. T. Lin, "FPGA realization of a neural-network-based nonlinear channel equalizer," IEEE Trans. Ind. Electron., vol. 51, no. 2, pp. 472-479, 2004.
27. H. S. Ng and K. P. Lam, "Analog and digital FPGA implementation of BRAIN for optimization problem," IEEE Trans. Neural Netw., vol. 14, pp. 1413-1425, Sep. 2003.
28. C. Lehmann, M. Viredaz, and F. Blayo, "A generic systolic array building block for neural networks with on-chip learning," IEEE Trans. Neural Netw., vol. 4, pp. 400-407, May 1993.