A model for learning to segment temporal sequences, utilizing a mixture of RNN experts together with adaptive variance

Neural Networks

Volumn 21, Issue 10, 2008, Pages 1466-1475

  Namikawa, Jun ^a   Tani, Jun ^a  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

Author keywords

Maximum likelihood estimation; Mixture of experts; Recurrent neural network; Segmentation of temporal sequences; Self organization

Indexed keywords

BLOCK CODES; EDUCATION; FACE RECOGNITION; IMAGE CLASSIFICATION; MARKOV PROCESSES; MAXIMUM LIKELIHOOD; MIXTURES; NEURAL NETWORKS; RECURRENT NEURAL NETWORKS; REINFORCEMENT LEARNING; SECURITY OF DATA; TIME SERIES ANALYSIS; TURBO CODES;

CONVENTIONAL METHODS; GATING NETWORKS; GENERALIZATION CAPABILITIES; GRADIENT DESCENT ALGORITHMS; HUMANOID ROBOTS; LEARNING METHODS; LEARNING PERFORMANCES; LIKELIHOOD FUNCTIONS; LISSAJOUS CURVES; MARKOV CHAINS; MIXTURE OF EXPERTS; REALISTIC PROBLEMS; SEGMENTATION OF TEMPORAL SEQUENCES; SELF-ORGANIZATION; TEMPORAL SEQUENCES; TIME SERIES PREDICTIONS;

MAXIMUM LIKELIHOOD ESTIMATION;

ALGORITHM; ARTICLE; DYNAMICS; FEEDBACK SYSTEM; LEARNING; MATHEMATICAL COMPUTING; NERVE CELL NETWORK; PREDICTION; PRIORITY JOURNAL; PROBABILITY; QUANTITATIVE STUDY; ROBOTICS; SENSORIMOTOR FUNCTION; SIMULATION;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; FEEDBACK; LIKELIHOOD FUNCTIONS; MARKOV CHAINS; NEURAL NETWORKS (COMPUTER); ROBOTICS; TIME FACTORS;

EID: 56949101180     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2008.09.005     Document Type: Article

References (28)

1. Bengio Y., Simard P., and Frasconi P. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5 2 (1994) 157-166
2. Elman J.L. Finding structure in time. Cognitive Science 14 (1990) 179-211
3. Evans G. Semantic theory and tacit knowledge. In: Holzman S., and Leich C. (Eds). Wittgenstein: To follow a rule (1981), Routledge and Kegan Paul, London 118-137
4. Hammer B., and Tin ̌o P. Recurrent neural networks with small weights implement definite memory machines. Neural Computation 15 (2003) 1897-1929
5. Hichreiter S., and Schmidhuber J. Long short-term memory. Neural Computation 9 8 (1997) 1735-1780
6. Igari, I., & Tani, J. (2008). Incremental learning of sequence patterns with a modular network model. Neurocomputing (in press)
7. Ikeda K., Otsuka K., and Matsumoto K. Maxwell Bloch turbulence. Progress of Theoretical Physics (Suppl.) 99 (1989) 295-324
8. Jacobs R.A., Jordan M.I., Nowlan S.J., and Hinton G.E. Adaptive mixtures of local experts. Neural Computation 3 1 (1991) 79-87
9. Jaeger, H. (2001). Short term memory in echo state networks. GMD report, 152, pp. 1-60
10. Jaeger H., and Haas H. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science (2004) 78-80
11. Jordan M.I. Attractor dynamics and parallelism in a connectionist sequential machine. Proc. of eighth annual conference of cognitive science society (1986) 531-546
12. Jordan M.I. Indeterminate motor skill learning problems. In: Jeannerod M. (Ed). Attention and performances, XIII (1988), MIT Press, Cambridge, MA
13. Jordan M.I., and Jacobs R.A. Hierarchical mixtures of experts and the EM algorithm. Neural Computation 6 2 (1994) 181-214
14. Kaneko K., and Tsuda I. Chaotic itinerancy. Chaos: Focus Issue on Chaotic Itinerancy 13 (2003) 926-936
15. Kataoka N., and Kaneko K. Functional dynamics I: Articulation process. Physica D 138 (2000) 225-250
16. Kataoka N., and Kaneko K. Functional dynamics II: Syntactic structure. Physica D 149 (2001) 174-196
17. Kataoka N., and Kaneko K. Dynamical networks in function dynamics. Physica D 181 (2003) 235-251
18. Maass, W., Natschläger, T., & Markram, H. (2002). A fresh look at real-time computation in generic recurrent neural circuits. Technical report. Institute for Theoretical Computer Science, TU Graz
19. Namikawa J., and Hashimoto T. Dynamics and computation in functional shifts. Nonlinearity 17 (2004) 1317-1336
20. Namikawa, J., & Tani, J. Learning to imitate stochastic time series in a compositional way by chaos. Neural Networks (submitted for publication)
21. Pollack J. The induction of dynamical recognizers. Machine Learning 7 (1991) 227-252
22. Rumelhart D.E., Hinton G.E., and Williams R.J. Learning internal representations by error propagation. In: Rumelhart D.E., and McClelland J.L. (Eds). Parallel distributed processing (1986), MIT Press, Cambridge, MA 318-362
23. Sato Y., and Ikegami T. Nonlinear computation with switching map systems. Journal of Universal Computer Science 6 (2000) 881-905
24. Schmidhuber J., Gers F., and Eck D. Learning nonregular languages: A comparison of simple recurrent networks and LSTM. Neural Computation 14 (2002) 2039-2041
25. Tani J., Nishimoto R., Namikawa J., and Ito M. Co-developmental learning between human and humanoid robot using a dynamic neural network model. IEEE Transactions on System, Man and Cybernetics 38 1 (2008) 43-59
26. Tani J., and Nolfi S. Learning to perceive the world as articulated: An approach for hierarchical learning in sensory-motor systems. Neural Networks 12 (1999) 1131-1141
27. Williams R.J., and Zipser D. A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1 (1989) 270-280
28. Wolpert D., and Kawato M. Multiple paired forward and inverse models for motor control. Neural Networks 11 (1998) 1317-1329