Reservoir optimization in recurrent neural networks using properties of Kronecker product

Logic Journal of the IGPL

Volumn 18, Issue 5, 2009, Pages 670-685

  Rad, Ali Ajdari ^a   Hasler, Martin ^a   Jalili, Mahdi ^b  

^a EPFL   (Switzerland)

^b SHARIF UNIVERSITY OF TECHNOLOGY   (Iran)

Author keywords

Echo state networks; Kronecker product; Optimization; Recurrent neural networks; Reservoir computing

Indexed keywords

MATRIX ALGEBRA; TOPOLOGY;

ECHO STATE NETWORKS; INTERNAL CONNECTIONS; KRONECKER PRODUCT; MATRIX; NETWORK-BASED; OPTIMISATIONS; PROPERTY; RESERVOIR COMPUTING; RESERVOIR OPTIMIZATIONS; TOPOLOGICAL STRUCTURE;

RECURRENT NEURAL NETWORKS;

EID: 77957281093     PISSN: 13670751     EISSN: 13689894     Source Type: Journal    
DOI: 10.1093/jigpal/jzp044     Document Type: Article

References (39)

1. Ajdari Rad, A., Jalili, M., and Hasler, M., Reservoir optimization in recurrent neural networks using Kronecker kernels, In Proceedings of the IEEE International Symposium on Circuits and Systems, pp. 868-871, 2008.
2. Atiya, A.F. and Parlos, A.G., New results on recurrent network training: unifying the algorithms and accelerating convergence, IEEE Transaction on Neural Networks, vol. 11, no. 3, pp. 697-709, 2000.
3. Boyd, S. and Chua, L.O., Fading memory and the problem of approximating nonlinear operators with Volterra series, IEEE Transactions on Circuits and Systems, vol. 32, pp. 1150-1161, 1985.
4. Buehner, M., and Young, P., A tighter bound for the echo state property, IEEE Transactions on Neural Networks, vol. 17, no. 3, pp. 820-824, 2006.
5. Bush, K. and Tsendjav, B., Improving the richness of echo state features using next ascent local search, In Proceedings of the Artificial Neural Networks in Engineering Conference, pp. 227-232, 2005.
6. Dutoit, X., Van Brussel, H., and Nuttin, M., Reservoir size reduction in echo state networks for classification problems, In Proceedings of the Neural Information Processing Systems: Workshop on Echo State Networks and Liquid State Machines, 2006.
7. Dutoit, X., Schrauwen, B., Campenhout, J, Van Stroobandt, D., Van Brussel, H., and Nuttin, M., Pruning and regularization in reservoir computing, Neurocomputing, Vol. 72, No. 7-9, pp. 1534-1546, 2009.
8. Feldkamp, L.A., Prokhorov, D.V., Eagen, C.F., and Yuan, F., Enhanced multi-stream Kalman filter training for recurrent neural networks, In Suykens J. and Vandewalle, J., (eds), Nonlinear Modeling: Advanced Black-Box Techniques, pp. 29-54, Kluwer, 1998.
9. Hammer, B., Schrauwen, B., and Steil, J., Recent advances in efficient learning of recurrent networks, In Proceedings of European Symposium on Artificial Neural Networks (ESANN 2009), Brugge: d-facto, pp. 213-226, 2009.
10. Hopfield, J., Neural networks and physical systems with emergent collective computational abilities, Proceedings of the National Academy of Sciences of the USA, vol. 79, no. 8, pp. 2554-2558, 1982.
11. Ishii, K., van der Zant, T., Vlatko Becanovic, V., and Ploger, P., Identification of motion with echo state network, In Proceedings of the IEEE OCEANS, pages 1205-1210, 2004.
12. Jaeger, H., Short term memory in echo state networks, GMD report 152, German National Research Center for Information Technology, 2001.
13. Jaeger, H., Lukoševičius, M., Popovici, D., and Siewert, U., Optimization and applications of echo state networks with leaky integrator neurons, Neural Networks, vol. 20, no. 3, pp. 335-352, 2007.
14. Jaeger, H., The echo state approach to analysing and training recurrent neural networks, GMD report 148, German National Research Center for Information Technology, 2001.
15. Jaeger, H., and Haas, H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless telecommunication, Science, vol. 308, no. 5667, pp. 78-80, 2004.
16. Jiang, F., Berry, H., and Schoenauer, M., Supervised and evolutionary learning of echo state networks, Proceedings of 10th International Conference on Parallel Problem Solving From Nature, pp. 215-224, 2008.
17. Joshi, P. and Maass, W., Movement generation and control with generic neural microcircuits, In Ijspeert, A., Murata, M., and Wakamiya, N., (eds.) Biologically Inspired Approaches to Advanced Information Technology, Lecture Notes in Computer Science, vol. 3141, pp. 258-273, 2004.
18. Kilian J., and Siegelmann, H., The dynamic universality of sigmoidal neural networks, Information and Computation, vol. 128, pp. 48-56, 1996.
19. Laub, A., Matrix Analysis for Scientists and Engineers, SIAM Publications, Philadelphia, PA, 2005.
20. Leskovec, J. and Faloutsos, C., Scalable modeling of real graphs using kronecker multiplication, In Proceedings of the International Conference on Machine Learning, pp. 497-504, 2007.
21. Lukoševičius, M. and Jaeger H., Overview of Reservoir Recipes, Jacobs University Technical Report no. 11, 2007.
22. Lukoševičius, M. and Jaeger H., Reservoir Computing Approaches to Recurrent Neural Network Training, to appear Computer Science Review, 2009.
23. Maass, W., Natschlager, T., and Markram, H., Fading memory and kernel properties of generic cortical microcircuit models, Journal of Physiology, vol. 98, no. 4-6, pp. 315-330, 2004.
24. Maass, W., Natschlager, T., and Markram, H. Realtime computing without stable states: a new framework for neural computation based on perturbations, Neural Computing, vol. 14, no.11, pp. 2531-2560, 2002.
25. Ozturk, M., Xu, D., and Principe, J., Analysis and design of echo state networks, Neural Computation, vol. 19, no. 1, pp. 111-138, 2007.
26. Reinhart, R.F. and Steil, J.J., Recurrent Neural Associative Learning of Forward and Inverse Kinematics for Movement Generation of the Redundant PA-10 Robot, ECSIS Symposium on Learning and Adaptive Behaviors for Robotic Systems, pp. 35-40, 2008.
27. Schiller, U., and Steil, J., Analyzing the weight dynamics of recurrent learning algorithms, Neurocomputing, vol. 63, pp. 5-23, 2005.
28. Schrauwen, B., Verstraeten D., and Van Campenhout, J., An overview of reservoir computing: theory, applications and implementations, In Proceedings of the 15th European Symposium on Artificial Neural Networks, pp. 471-482, 2007.
29. Schrauwen, B.,Warderman, M., Verstraeten, D., Steil, J., and Stroobandt, D., Improving reservoirs using intrinsic plasticity, Neurocomputing, vol. 71, no. 7-9, pp. 1159-1171, 2008.
30. Steil, J., Backpropagation-decorrelation: online recurrent learning with O(N) complexity, In Proceedings of the International Joint Conference on Neural Networks, pp. 843-848, 2004.
31. Steil, J., Online stability of backpropagation-decorrelation recurrent learning, Neurocomputing, vol. 69, no. 7-9, pp. 642-650, 2006.
32. Steil, J., Several ways to solve the MSO problem, In Proceedings of the European Symposium on Artificial Neural Networks (ESANN), pp. 489-494, 2007.
33. Steil, J. Online reservoir adaptation by intrinsic plasticity for backpropagationdecorrelation and echo state learning, Neural Networks, vol. 20, no. 3, pp. 353-364, 2007.
34. Verstraeten, D., Schrauwen, B., D'Haene, M., and Stroobandt, D., An experimental unification of reservoir computing methods, Neural Networks, vol. 20, no. 3, pp. 391-403, 2007.
35. Werbos, P.J., Backpropagation through time: what it does and how to do it, Proceedings of the IEEE, vol. 78, no. 10, pp. 1550-1560, 1990.
36. Williams, R.J. and Zipper, D., A learning algorithm for continuosly running fully recurrent neural networks, Neural Computation, vol. 1, pp. 270-280, 1989.
37. Wyffels, F., Schrauwen, B., Verstraeten, D., and Dirk, S., Band-pass reservoir computing, In Z. Hou and N. Zhang, editors, Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 2008), pp. 3204-3209, 2008.
38. Wyffels, F., Schrauwen, B., and Dirk, S., Stable Output Feedback in Reservoir Computing Using Ridge Regression, Lecture Notes In Computer Science, vol. 5163, Proceedings of the 18th International Conference on Artificial Neural Networks (ICANN 2008), pp. 807-818, 2008.
39. Xue, Y., Yang, L., Haykin, S., Decoupled echo state networks with lateral inhibition, Neural Networks, vol. 20, no. 3, pp. 365-376, 2007.