Extracting finite-state representations from recurrent neural networks trained on chaotic symbolic sequences

IEEE Transactions on Neural Networks

Volumn 10, Issue 2, 1999, Pages 284-302

  Tiňo, Peter ^a   Köteles, Miroslav ^b  

^a AUSTRIAN RESEARCH INSTITUTE FOR ARTIFICIAL INTELLIGENCE   (Austria)

^b SLOVAK UNIVERSITY OF TECHNOLOGY   (Slovakia)

Author keywords

Chaotic symbolic sequences; Entropy spectra; Fractal geometry; Recurrent neural network; Stochastic automata

Indexed keywords

CHAOS THEORY; COMPUTATIONAL GEOMETRY; FRACTALS; INFORMATION THEORY; KNOWLEDGE ACQUISITION; LEARNING SYSTEMS; RANDOM PROCESSES; RESPONSE TIME (COMPUTER SYSTEMS); SEQUENTIAL MACHINES;

CHAOTIC SYMBOLIC SEQUENCES; FINITE STATE REPRESENTATION;

RECURRENT NEURAL NETWORKS;

EID: 0033097317     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.750555     Document Type: Article

References (44)

1. M. R. Anderberg, Cluster Analysis for Applications. New York: Academic, 1973.
2. F. Aurenhammer, "Voronoi diagrams - Survey of a fundamental geometric data structure," ACM Comput. Surveys, vol. 3, pp. 345-405, 1991.
3. M. F. Barnsley, Fractals Everywhere. New York: Academic, 1988.
4. C. Beck and F. Schlogl, Thermodynamics of Chaotic Systems. Cambridge, U.K.: Cambridge Univ. Press, 1995.
5. G. E. Box and G. M. Jenkins, Time Series Analysis, Forecasting and Control. San Francisco, CA: Holden Day, 1970.
6. J. Bruske and G. Sommer, "Dynamic cell structure learns perfectly topology preserving map," Neural Comput., vol. 7, no. 4, pp. 845-865, 1995.
7. M. P. Casey, "The dynamics of discrete-time computation, with application to recurrent neural networks and finite-state machine extraction," Neural Comput., vol. 8, no. 6, pp. 1135-1178, 1996.
8. A. Cleeremans, D. Servan-Schreiber, and J. L. McClelland, "Finite-state automata and simple recurrent networks," Neural Comput., vol. 1, no. 3, pp. 372-381, 1989.
9. J. P. Crutchfield, "Reconstructing language hierarchies," in Information Dynamics, H. Atmanspacher and H. Scheingraber, Eds. New York: Plenum, 1991, pp. 45-60.
10. _, "The calculi of emergence: Computation, dynamics and induction," in Proc. Oji Int. Seminar: Complex Syst., Physica D, Namazu, Japan, 1993, Special Issue; SFI Tech. Rep. SFI 94-03-016, 1994.
11. J. P. Crutchfield and K. Young, "Inferring statistical complexity," Phys. Rev. Lett., vol. 63, pp. 105-108, July 1989.
12. _, "Computation at the onset of chaos," in Complexity, Entropy, and the Physics of Information, SFI Studies in the Sciences of Complexity, vol. 8, W. H. Zurek, Ed. Reading, MA: Addison-Wesley, 1990, pp. 223-269.
13. J. L. Elman, "Finding structure in time," Cognitive Sci., vol. 14, pp. 179-211, 1990.
14. P. Frasconi, M. Gori, M. Maggini, and G. Soda, "Insertion of finite-state automata in recurrent radial basis function networks," Machine Learning, vol. 23, pp. 5-32, 1996.
15. C. L. Giles, C. B. Miller, D. Chen, H. H. Chen, G. Z. Sun, and Y. C. Lee, "Learning and extracting finite-state automata with second-order recurrent neural networks," Neural Comput., vol. 4, no. 3, pp. 393-405, 1992.
16. J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation. Reading, MA: Addison-Wesley, 1991.
17. J. Jeffrey, "Chaos game representation of gene structure," Nucleic Acids Res., vol. 18, no. 8, pp. 2163-2170, 1990.
18. A. I. Khinchin, Math. Foundations of Inform. Theory. New York: Dover, 1957.
19. T. Kohonen, "The self-organizing map," Proc. IEEE, vol. 78, pp. 1464-1479, 1990.
20. A. Lapedes, C. Barnes, C. Burks, R. Farber, and K. Sirotkin, "Application of neural networks and other machine learning algorithms to DNA sequence analysis," in Computers and DNA, SFI Studies in the Sciences of Complexity, vol. VII, G. Bell and T. Marr, Eds. Reading, MA: Addison-Wesley, 1989, pp. 157-182.
21. W. Li, "The study of correlation structures of DNA sequences: A critical review," Comput. Chemistry, vol. 21, no. 4, pp. 257-272, 1997.
22. H. B. Mann and D. R. Whitney, "On a test of whether one of two random variables is stochastically larger than the other," Ann. Math. Statist., vol. 18, 1947.
23. P. Manolios and R. Fanelli, "First order recurrent neural networks and deterministic finite-state automata," Neural Comput., vol. 6, pp. 1155-1173, 1994.
24. J. L. McCauley, Chaos, Dynamics and Fractals: An Algorithmic Approach to Deterministic Chaos. Cambridge, U.K.: Cambridge Univ. Press, 1994.
25. C. Moore, "Dynamical recognizers: Real-time language recognition by analog computers," Dept. Theoretical Comput. Sci., Santa Fe Institute, Santa Fe, NM, Tech. Rep. Working Paper 96-05-023, 1996.
26. P. Tiňo, "Spatial representation of symbolic sequences through iterative function systems," IEEE Trans. Syst., Man, Cybern., pt. B, to be published, 1999.
27. P. Tiňo and G. Dorffner, "Constructing finite-context sources from fractal representations of symbolic sequences," Austrian Res. Inst. Artificial Intell., Austria, Tech. Rep. TR-98-18, 1998.
28. P. Tiňo, B. G. Horne, C. L. Giles, and P. C. Collingwood, "Finite-state machines and recurrent neural networks - Automata and dynamical systems approaches," in Neural Networks and Pattern Recognition, J. E. Dayhoff and O. Omidvar, Eds. New York: Academic, 1998, pp. 171-220.
29. P. Tiňo and J. Sajda, "Learning and extracting initial mealy machines with a modular neural network model," Neural Comput., vol. 7, no. 4, pp. 822-844, 1995.
30. J. L. Oliver, P. Bernaola-Galván, J. Guerrero-Garcia, and R. Román Roldan, "Entropic profiles of dna sequences through chaos-game-derived images," J. Theor. Biol., vol. 160, pp. 457-470, 1993.
31. C. W. Omlin and C. L. Giles, "Extraction of rules from discrete-time recurrent neural networks," Neural Networks, vol. 9, no. 1, pp. 41-51, 1996.
32. J. C. Principe and J.-M. Kuo, "Dynamic modeling of chaotic time series with neural networks," in Advances in Neural Inform. Processing Syst., vol. 7, G. Tesauro, D. Touretzky, and T. Leen, Eds. Cambridge, MA: MIT Press, 1995, pp. 311-318.
33. J. C. Principe, A. Rathie, and J.-M. Kuo, "Prediction of chaotic time series with neural networks and the issue of dynamic modeling," Int. J. Bifurcation and Chaos, vol. 2, no. 4, pp. 989-996, 1992.
34. A. Renyi, "On the dimension and entropy of probability distributions," Acta Math. Hung., vol. 10, p. 193, 1959.
35. R. Roman-Roldan, P. Bernaola-Galvan, and J. L. Oliver, "Entropic feature for sequence pattern through iteration function systems," Pattern Recognition Lett., vol. 15, pp. 567-573, 1994.
36. D. Ron, Y. Singer, and N. Tishby, "The power of amnesia," Machine Learning, vol. 25, 1996.
37. J. Schmidhuber and S. Heil, "Sequential neural text compression," IEEE Trans. Neural Networks, vol. 7, pp. 142-146, 1996.
38. E. E. Snyder and G. D. Stormo, "Identification of coding regions in genomic DNA sequences: An application of dynamic programming and neural networks," Nucleic Acids Res., vol. 21, pp. 607-613, 1993.
39. J. Tani and N. Fukurama, "Embedding a grammatical description in deterministic chaos: An experiment in recurrent neural learning," Biol. Cybern., vol. 72, pp. 365-370, 1995.
40. E. A. Wan, "Time series prediction by using a connectionist network with internal delay lines," in Time Series Prediction, A. S. Weigend and N. A. Gershenfeld, Eds. Reading, MA: Addison-Wesley, 1994, pp. 195-217.
41. R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks," Neural Comput., vol. 4, no. 3, pp. 406-414, 1992.
42. A. S. Weigend, B. A. Huberman, and D. E. Rumelhart, "Predicting the future: A connectionist approach," Int. J. Neural Syst., vol. 1, pp. 193-209, 1990.
43. K. Young and J. P. Crutchfield, "Fluctuation spectroscopy," in Chaos, Solitons, and Fractals, W. Ebeling, Ed., 1993, special issue on complexity.
44. J. Ziv and N. Merhav, "A measure of relative entropy between individual sequences with application to universal classification," IEEE Trans. Inform. Theory, vol. 39, pp. 1270-1279, 1993.