Recurrent neural networks with small weights implement definite memory machines

Neural Computation

Volumn 15, Issue 8, 2003, Pages 1897-1929

  Hammer, Barbara ^a   Tiňo, Peter ^b  

^a UNIVERSITY OF OSNABRÜCK   (Germany)

^b UNIVERSITY OF BIRMINGHAM   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0042326343     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/08997660360675080     Document Type: Article

References (55)

1. Anthony, M., & Bartlett, P. L. (1999). Neural network learning: Theoretical foundations. Cambridge: Cambridge University Press.
2. Baldi, P., Brunak, S., Frasconi, P., Pollastri, G., & Soda, G. (2001). Bidirectional dynamics for protein secondary structure prediction. In R. Sun & C. L. Giles (Eds.), Sequence learning: Paradigms, algorithms, and applications (pp. 80-104). Berlin: Springer-Verlag.
3. Bartlett, P. L. (1997). For valid generalization, the size of the weights is more important than the size of the network. In M. C. Mozer, M. I. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems, 9 (pp. 134-141). Cambridge, MA: MIT Press.
4. Bartlett, P. L., Long P., & Williamson, R. (1994). Fat-shattering and the learn-ability of real valued functions. In Proceedings of the 7th ACM Conference on Computational Learning Theory (pp. 299-310).
5. Baum, E. B., & Haussler, D. (1989). What size net gives valid generalization? Neural Computation, 1(1), 151-165.
6. Bengio, Y., & Frasconi, P. (1996). Input/output HMMs for sequence processing. IEEE Transactions on Neural Networks, 7(5), 1231-1249.
7. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157-166.
8. Bühlmann, P., & Wyner, A. J. (1999). Variable length Markov chains. Annals of Statistics, 27, 480-513.
9. Carrasco, R. C., & Forcada, M. L. (2001). Simple strategies to encode tree automata in sigmoid recursive neural networks. IEEE Transactions on Knowledge and Data Engineering, 13(2), 148-156.
10. Christiansen, M. H., & Chater, N. (1999). Towards a connectionist model of recursion in human linguistic performance. Cognitive Science, 23, 157-205.
11. Clouse, D. S., Giles, C. L., Home, B. G., & Cottrell, G. W. (1997). Time-delay neural networks: Representation and induction of finite state machines. IEEE Transactions on Neural Networks, 8(5), 1065.
12. Elman, J., Bates, E., Johnson, M., Karmiloff-Smith, A., Parisi, D., & Plunkett, K. (1996). Rethinking innateness: A connectionist perspective on development. Cambridge, MA: MIT Press.
13. Frasconi, P., Gori, M., Maggini, M., & Soda, G. (1995). Unified integration of explicit rules and learning by example in recurrent networks. IEEE Transactions on Knowledge and Data Engineering, 8(6), 313-332.
14. Funahashi, K., & Nakamura, Y. (1993). Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks, 12, 831-864.
15. Giles, C. L., Lawrence, S., & Lin, T. (1995). Learning a class of large finite state machines with a recurrent neural network. Neural Networks, 8, 1359-1365.
16. Giles, C. L., Lawrence, S., & Tsoi, A. C. (1997). Rule inference for financial prediction using recurrent neural networks. Proceedings of the Conference on Computational Intelligence for Financial Engineering (pp. 253-259). New York.
17. Guyon, I., & Pereira, F. (1995). Design of a linguistic postprocessor using variable memory length Markov models. Proceedings of the International Conference on Document Analysis and Recognition (pp. 454-457). Montreal, Canada: IEEE Computer Society Press.
18. Hammer, B. (1997). On the generalization of Elman networks. In W. Gerstner, A. Germond, M. Hasler, & J.-D. Nicaud (Eds.), Artificial Neural Networks - ICANN'97 (pp. 409-414). Berlin: Springer-Verlag.
19. Hammer, B. (1999). On the learnability of recursive data. Mathematics of Control, Signals, and Systems, 12, 62-79.
20. Hammer, B. (2001). Generalization ability of folding networks. IEEE Transactions on Knowledge and Data Engineering, 13(2), 196-206.
21. Haussler, D. (1992). Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation, 100, 78-150.
22. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780.
23. Hornik, K. (1993). Some new results on neural network approximation. Neural Networks, 6, 1069-1072.
24. Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2, 359-366.
25. Karpinski, M., & Macintyre, A. (1995). Polynomial bounds for the VC dimension of sigmoidal neural networks. In Proceedings of the 27th Annual ACM Symposium on the Theory of Computing (pp. 200-208).
26. Kohavi, Z. (1978). Switching and finite automata. New York: McGraw-Hill.
27. Kohonen, T. (1997). Self-organizing maps. Berlin: Springer-Verlag.
28. Koiran, P., & Sontag, E. D. (1997). Vapnik-Chervonenkis dimension of recurrent neural networks. In Proceedings of the 3rd European Conference on Computational Learning Theory (pp. 223-237).
29. Kolen, J. F. (1994a). Recurrent networks: State machines or iterated function systems? In Proceedings of the 1993 Connectionist Models Summer School (pp. 203-210). Mahwah, NJ: Erlbaum.
30. Kolen, J. F. (1994b). The origin of clusters in recurrent neural state space. In Proceedings of the 1993 Connectionist Models Summer School (pp. 508-513). Mahwah, NJ: Erlbaum.
31. Krogh, A. (1997). Two methods for improving performance of a HMM and their application for gene finding. In Proceedings of the 5th International Conference on Intelligent Systems for Molecular Biology (pp. 179-186). Menlo Park, CA: AAAI Press.
32. Laird, P., & Saul, R. (1994). Discrete sequence prediction and its applications. Machine Learning, 15, 43-68.
33. Maass, W., & Orponen, P. (1998). On the effect of analog noise in discrete-time analog computation. Neural Computation, 10(5), 1071-1095.
34. Maass, W., & Sontag, E. D. (1999). Analog neural nets with gaussian or other common noise distributions cannot recognize arbitrary regular languages. Neural Computation, 11, 771-782.
35. Mackey, M. C., & Glass, L. (1977). Oscillations and chaos in physiological control systems. Science, 197, 287-289.
36. Nadas, J. (1984). Estimation of probabilities in the language model of the IBM speech recognition system. IEEE Transactions on ASSP, 4, 859-861.
37. Neal, R., & Hinton, G. (1998). A view of the EM algorithm that justifies incremental, sparse, and other variants. In M. Jordan (Ed.), Learning in graphical models (pp. 355-368). Norwell, MA: Kluwer.
38. Omlin, C. W., & Giles, C. L. (1996a). Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM, 43(6), 937-972.
39. Omlin, C. W., & Giles, C. L. (1996b). Stable encoding of large finite-state automata in recurrent networks with sigmoid discriminants. Neural Computation, 8, 675-696.
40. Robinson, T., Hochberg, M., & Renals, S. (1996). The use of recurrent networks in continuous speech recognition. In C.-H. Lee & F. K. Song (Eds.), Advanced topics in automatic speech and speaker recognition. Norwell, MA: Kluwer.
41. Ron, D., Singer, Y., & Tishby, N. (1996). The power of amnesia. Machine Learning, 25, 117-150.
42. Sejnowski, T., & Rosenberg, C. (1987). Parallel networks that learn to pronounce English text. Complex Systems, 1, 145-168.
43. Shawe-Taylor, J., Bartlett, P. L., Williamson, R., & Anthony, M. (1998). Structural risk minimization over data dependent hierarchies. IEEE Transactions on Information Theory, 44(5), 1926-1940.
44. Siegelmann, H. T., & Sontag, E. D. (1994). Analog computation, neural networks, and circuits. Theoretical Computer Science, 131, 331-360.
45. Siegelmann, H. T., & Sontag, E. D. (1995). On the computational power of neural networks. Journal of Computer and System Sciences, 50, 132-150.
46. Sontag, E. D. (1992). Feedforward nets for interpolation and classification. Journal of Computer and System Sciences, 45, 20-48.
47. Sontag, E. D. (1998). VC dimension of neural networks. In C. Bishop (Ed.), Neural networks and machine learning (pp. 69-95). Berlin: Springer-Verlag.
48. Sun, R. (2001), Introduction to sequence learning. In R. Sun & C. L. Giles (Eds.), Sequence learning: Paradigms, algorithms, and applications (pp. 1-10). Berlin: Springer-Verlag.
49. Tiňo, P., Čerňanský, M., & Beňušková, L. (2002a). Markovian architectural bias of recurrent neural networks. In P. Sinčák, J. Vaščák, V. Kvasnička, & J. Pospichal (Eds.), Intelligent technologies - Theory and applications: Frontiers in AI and applications - 2nd Euro-International Symposium on Computational Intelligence (pp. 17-23). Amsterdam: IOS Press.
50. Tiňo, P., Čerňanský, M., & Beňušková, L. (2002b). Markovian architectural bias of recurrent neural networks (Tech. Rep. No. NCRG/2002/008). NCRG, Aston University, UK.
51. Tiňo, P., & Dorffner, G. (2001). Predicting the future of discrete sequences from fractal representations of the past. Machine Learning, 45(2), 187-218.
52. Tiňo, P., & Hammer, B. (2002). Architectural bias of recurrent neural networks - fractal analysis. In J. R. Dorronsoro (Ed.), Int. Conf. on Artificial Neural Networks (ICANN 2002) (pp. 1359-1364). Berlin: Springer-Verlag.
53. Tiňo, P., & Šajda, J. (1995). Learning and extracting initial Mealy machines with a modular neural network model. Neural Computation, 4, 822-844.
54. Vidyasagar, M. (1997). A theory of learning and generalization. Berlin: Springer-Verlag.
55. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., & Lang, K. (1989). Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(3), 328-339.