Comments on "Constructive learning of recurrent neural networks: Limitations of recurrent cascade correlation and a simple solution

IEEE Transactions on Neural Networks

Volumn 7, Issue 4, 1996, Pages 1047-1051

  Kremer, Stefan C ^a  

^a Industry Canada   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER ARCHITECTURE; FINITE AUTOMATA; LEARNING SYSTEMS; SIGNAL PROCESSING; THEOREM PROVING;

CONSTRUCTIVE LEARNING; RECURRENT CASCADE CORRELATION; RECURRENT NEURAL NETWORKS;

NEURAL NETWORKS;

EID: 0030190837     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.508949     Document Type: Review

References (49)

1. S. Fahlman, "The recurrent cascade-correlation architecture," in Advances in Neural Information Processing Syst. 3, R. Lippmann, J. Moody, and D. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, pp. 190-196.
2. M. Tomita, "Dynamic construction of finite-state automata from examples using hill-climbing," in Proc. 4th Annu. Cognitive Sci. Conf., Ann Arbor, MI, 1982, pp. 105-108.
3. S. C. Kremer, "Finite state automata that recurrent cascade-correlation cannot represent," in Advances in Neural Information Processing Syst. 8, D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Eds. Cambridge, MA: MIT Press, 1996.
4. N. Alon, A. K. Dewdney, and T. J. Ott, "Efficient simulation of finite automata by neural nets," J. Assoc. Comput. Machinery, vol. 38, no. 2, pp. 495-514, 1991.
5. R. Alquezar and A. Sanfeliu, "An algebraic framework to represent finite state automata in single-layer recurrent neural networks," Neural Computa., vol. 7, no. 5, pp. 931-949, 1995.
6. A. D. Back and A. C. Tsoi, "FIR and IIR synapses, a new neural-network architecture for time-series modelling," Neural Computa., vol. 3, no. 3, pp. 337-350, 1991.
7. P. Baldi and S. S. Venkatesh, "Number of stable points for spin-glasses and neural networks of higher orders," Phys. Rev. Lett., vol. 58, no. 9, pp. 913-915, 1987.
8. J. Batali, "Recurrent neural networks, context-free grammars, and evolution," Dep. Cognitive Sci. Univ. California San Diego, La Jolla, CA, Tech. Rep., 1995.
9. C. Chen and V. Honavar, "A neural-network architecture for syntax analysis," Dep. Computer Sci., Iowa State Univ., Ames, Iowa, Tech. Rep. ISU-CS-TR 95-18, 1995.
10. D. Chen, C. L. Giles, G. Z. Sun, H. H. Chen, Y. C. Lee, and M. W. Goudreau, "Constructive learning of recurrent neural networks," in Proc. 1993 IEEE Int. Conf. Neural Networks, vol. III, pp. 1196-1201, Piscataway, NJ, 1993.
11. A. Cleeremans, D. Servan-Schreiber, and J. McClelland, "Finite-state automata and simple recurrent neural networks," Neural Computa., vol. 1, no. 3, pp. 372-381, 1989.
12. D. S. Clouse, C. L. Giles, B. G. Horne, and G. W. Cottrell, "Learning large Debruijn automata with feedforward neural networks," Comput. Sci. Eng., Univ. California San Diego, La Jolla, CA, Tech. Rep. CS94-398, 1994.
13. J. L. Elman, "Distributed representations, simple recurrent networks, and grammatical structure," Machine Learning, vol. 7, nos. 2/3, pp. 195-226, 1991.
14. S. Franklin and M. Garzon, "Neural computability," in Progress in Neural Networks, O. Omidvar, Ed. Norwood, NJ: Ablex, 1991.
15. P. Frasconi, M. Gori, M. Maggini, and G. Soda, "Unified integration of explicit rules and learning by example in recurrent networks," IEEE Trans. Knowledge Data Eng., vol. 7, no. 2, pp. 340-346, 1995.
16. _, "Representation of finite state automata in recurrent radial basis function networks," Machine Learning, vol. 23, pp. 5-32, 1996.
17. P. Frasconi, M. Gori, and G. Soda, "Local feedback multilayered networks," Neural Comput., vol. 4, pp. 120-130, 1992.
18. C. L. Giles, B. G. Horne, and T. Lin, "Learning a class of large finite-state machines with a recurrent neural network," Neural Networks, vol. 8, no. 9, p. 1359, 1995.
19. C. L. Giles, G. M. Kuhn, and R. J. Williams, "Dynamic recurrent neural networks: Theory and applications." IEEE Trans. Neural Networks, vol. 5, pp. 153-156, 1994.
20. C. L. Giles, C. B. Miller, D. Chen, H. H. Chen, G. Z. Sun, and Y. C. Lee, "Learning and extracting finite state automata with second-order recurrent neural networks," Neural Computa., vol. 4, no. 3, pp. 393-405, 1992.
21. C. L. Giles and C. W. Omlin, "Pruning recurrent neural networks for improved generalization performance," IEEE Trans. Neural Networks, vol. 5, no. 5, pp. 848-851, 1994.
22. C. L. Giles, G. Z. Sun, H. H. Chen, Y. C. Lee, and D. Chen, "Higher-order recurrent networks and grammatical inference," in Advances in Neural Information Processing Systems 2, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 380-387.
23. M. W. Goudreau, C. L. Giles, S. T. Chakradhar, and D. Chen, "First-order vs. second-order single layer recurrent neural networks," IEEE Trans. Neural Networks, vol. 5, pp. 511-513, 1994.
24. B. G. Horne and C. L. Giles, "An experimental comparison of recurrent neural network," in Advances in Neural Information Processing Systems 7, G. Tesauro, D. Touretzky, and T. Leen, Eds. Cambridge, MA: MIT Press, 1995, pp. 697-704.
25. B. G. Horne and D. Hush, "Bounds on the complexity of recurrent neural-network implementations of finite state machines, in Advances in Neural Information Processing Systems 6, San Mateo, CA: Morgan Kaufmann, 1994, pp. 359-366.
26. Z. Kohavi, Switching and Finite Automata Theory, 2nd ed. New York: McGraw-Hill, 1978.
27. Y. C. Lee, G. Doolen, H. H. Chen, G. Z. Sun, T. Maxwell, H. Y. Lee, and C. L. Giles, "Machine learning using a higher order correlational network," Physica D, vol. 22-D, nos. 1-3, pp. 276-306, 1986.
28. S. Lucas and R. Damper, "Syntactic neural networks," Connection Sci., vol. 2, pp. 199-225, 1990.
29. P. Manolios and R. Fanelli, "First order recurrent neural networks and deterministic finite state automata," Neural Computa., vol. 6, no. 6, pp. 1154-1172, 1994.
30. M. L. Minsky, "Computation: Finite and infinite machines," in Neural Networks: Automata Made Up of Parts. Englewood Cliffs, NJ: Prentice-Hall, 1967.
31. M. C. Mozer and J. Bachrach, "SLUG: A connectionist architecture for inferring the structure of finite-state environments," Machine Learning, vol. 7, nos. 2/3, pp. 139-160, 1991.
32. K. S. Narendra and K. Parthasarathy, "Identification and control of dynamical systems using neural networks," IEEE Trans. Neural Networks, vol. 1, p. 4, 1990.
33. I. Noda and M. Nagao, "A learning method tor recurrent networks based on minimization of finite automata," in Proc. Int. Joint Conf. Neural Networks 1992, vol. I, pp. 27-32, June 1992.
34. C. W. Omlin and C. L. Giles, "Constructing deterministic finite-state automata in recurrent neural networks," J. ACM, accepted.
35. _, "Stable encoding of large finite-state automata in recurrent neural networks with sigmoid discriminants," Neural Computa., vol. 8, no. 4, p. 675-696, 1996.
36. M. W. Pedersen and L. K. Hansen, "Recurrent networks: Second-order properties and pruning," in Advances in Neural Information Processing Systems 7, G. Tcsauro, D. Tourctzky, and T. Leen, Eds. Cambridge, MA: MIT Press, 1995.
37. J. B. Pollack, "Recursive distributed representations," Artificial Intell., vol. 46, pp. 77-105, 1990.
38. _, "The induction of dynamical recognizers," Machine Learning, vol. 7, nos. 2/3, pp. 227-252, 1991.
39. P. S. Sastry, G. Santharam, and K. P. Unnikrishnan, "Memory neuron networks for identification and control of dynamical systems," IEEE Trans. Neural Networks, vol. 5, pp. 306-319, 1994.
40. H. T. Siegelmann, B. G. Horne, and C. L. Giles, "Computational capabilities of recurrent narx neural networks," IEEE Trans. Syst., Man, Cybern., accepted.
41. H. T. Siegelmann and E. D. Sontag, "On the computational power of neural nets," J. Comput. Syst. Sci., vol. 50, pp. 132-150, 1995.
42. G. Z. Sun, H. H. Chen, Y. C. Lee, and C. L. Giles, "Turing equivalence of neural networks with second-order connection weights," in Proc. 1991 IEEE INNS Int. Joint Conf. Neural Networks vol. II, Seattle, pp. 357-362, Piscataway, NJ, 1991.
43. G. Z. Sun, C. L. Giles, H. H. Chen, and Y. C. Lee, "The neural-network pushdown automaton: Model, stack, and learning simulations, Inst. Advanced Comput. Studies, Univ. Maryland, College Park, MD, Tech. Rep. UMIACS-TR-93-77, 1993.
44. P. Tino and J. Sajda, "Learning and extracting initial mealy machines with a modular neural-network model," Neural Computata., vol. 7, no. 4, pp. 822-844, 1995.
45. B. de Vries and J. Principe, "The gamma model - a new neural network for temporal processing," Neural Networks, vol. 5, no. 4, pp. 565-576, 1992.
46. R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks," Neural Computa., vol. 4, no. 3, p. 406, 1992.
47. B. M. Yamauchi and R. D. Beer, "Sequential behavior and learning in evolved dynamical neural networks," Adaptive Behavior, vol. 2, no. 3, pp. 219-246, 1994.
48. Z. Zeng, R. M. Goodman, and P. Smyth, "Learning finite state machines with self-clustering recurrent networks," Neural Computa., vol. 5, no. 6, pp. 976-990, 1993.
49. _, "Discrete recurrent neural networks for grammatical inference," IEEE Trans. Neural Networks, vol. 5, pp. 320-330, 1994.