Symbolic representation of recurrent neural network dynamics

IEEE Transactions on Neural Networks and Learning Systems

Volumn 23, Issue 10, 2012, Pages 1649-1658

  Huynh, Thuan Q ^a   Reggia, James A ^a  

^a Institute of Advanced Computer Science   (United States)

Author keywords

Finite state machines; hidden layer representation; penalty function; recurrent neural networks

Indexed keywords

BACKPROPAGATION LEARNING; COMPUTATIONAL EXPERIMENT; ERROR BACK-PROPAGATION; FEED-FORWARD NETWORK; HIDDEN LAYERS; PENALTY FUNCTION; SYMBOLIC REPRESENTATION; TEMPORAL SEQUENCES;

ASYNCHRONOUS MACHINERY; BACKPROPAGATION; CONTEXT FREE LANGUAGES; DYNAMICAL SYSTEMS; EXTRACTION; FINITE AUTOMATA; RECURRENT NEURAL NETWORKS; RESEARCH;

NETWORK LAYERS;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; FEEDBACK SYSTEM; PROCEDURES; STATISTICAL MODEL; SYMBOLISM;

ALGORITHMS; COMPUTER SIMULATION; FEEDBACK; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED; SYMBOLISM;

EID: 84876436918     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2012.2210242     Document Type: Article

References (24)

1. J. Elman, "Finding structure in time," Cognit. Sci., vol. 14, no. 2, pp. 179-211, 1990.
2. H. Jacobsson, "Rule extraction from recurrent neural networks: A taxonomy and review," Neural Comput., vol. 17, no. 6, pp. 1223-1263, 2005.
3. P. Tino, C. Schittenkopf, and G. Dorffner, "Financial volatility trading using recurrent neural networks," IEEE Trans. Neural Netw., vol. 12, no. 4, pp. 865-874, Jul. 2001.
4. J. Chen and N. Chaudhari, "Segmented-memory recurrent neural networks," IEEE Trans. Neural Netw., vol. 20, no. 8, pp. 1267-1280, Aug. 2009.
5. M. Jordan, "Attractor dynamics and parallelism in a connectionist sequential machine," in Proc. 8th Annu. Conf. Cognit. Sci. Soc., 1986, pp. 531-546.
6. J. Jung and J. Reggia, "Evolutionary design of neural network architectures using a descriptive encoding language," IEEE Trans. Evol. Comput., vol. 10, no. 6, pp. 676-688, Dec. 2006.
7. T. Lin, B. Horne, and C. Giles, "How embedded memory in recurrent neural network architectures helps learning long-term temporal dependencies," Neural Netw., vol. 11, no. 5, pp. 861-868, 1998.
8. T. Huynh and J. Reggia, "Guiding hidden layer representations for improved rule extraction from neural networks," IEEE Trans. Neural Netw., vol. 22, no. 2, pp. 264-275, Feb. 2011.
9. H. Jacobsson, "Rule extraction from recurrent neural networks," Ph.D. dissertation, Dept. Comput. Sci., Univ. Sheffield, Sheffield, U.K., 2006.
10. A. Tickle, R. Andrews, M. Golea, and J. Diederich, "The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks," IEEE Trans. Neural Netw., vol. 9, no. 6, pp. 1057-1068, Nov. 1998.
11. P. Tino and M. Koteles, "Extracting finite-state representations from recurrent neural networks trained on chaotic symbolic sequences," IEEE Trans. Neural Netw., vol. 10, no. 2, pp. 284-302, Mar. 1999.
12. Z. Zeng, R. Goodman, and P. Smyth, "Learning finite state machines with self-clustering recurrent networks," Neural Comput., vol. 5, no. 6, pp. 976-990, 1993.
13. S. Kremer, R. Neco, and M. Forcada, "Constrained second-order recurrent networks for finite-state automata induction," in Proc. 8th Int. Conf. Artif. Neural Netw., vol. 98. 1998, pp. 529-534.
14. S. Das and M. Mozer, "Dynamic on-line clustering and state extraction: An approach to symbolic learning," Neural Netw., vol. 11, no. 1, pp. 53-64, 1998.
15. S. Haykin, Neural Networks: A Comprehensive Foundation. Upper Saddle River, NJ: Prentice-Hall, 1994.
16. M. Riedmiller and H. Braun, "RPROP: A fast adaptive learning algorithm," in Proc. Int. Symp. Comput. Inf. Sci., 1992, pp. 279-286.
17. T. Kanungo, D. Mount, N. Netanyahu, C. Piatko, R. Silverman, and A. Wu, "An efficient k-means clustering algorithm: Analysis and implementation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 881-892, Jul. 2002.
18. C. Liou and W. Cheng, "Resolving hidden representations," in Proc. Int. Conf. Neural Inf. Process., 2008, pp. 254-263.
19. M. Gori, M. Maggini, E. Martinelli, and G. Soda, "Inductive inference from noisy examples using the hybrid finite state filter," IEEE Trans. Neural Netw., vol. 9, no. 3, pp. 571-575, May 1998.
20. S. Frank, "Learn more by training less: Systematicity in sentence processing by recurrent networks," Connect. Sci., vol. 18, no. 3, pp. 287-302, 2006.
21. S. Frank and M. Cernansky, "Generalization and systematicity in echo state networks," in Proc. 30th Annu. Conf. Cognit. Sci. Soc., 2008, pp. 733-738.
22. S. Frank and H. Jacobsson, "Sentence-processing in echo state networks: A qualitative analysis by finite state machine extraction," Connect. Sci., vol. 22, no. 2, pp. 135-155, 2010.
23. F. Van der Velde, G. T. van der V. van der Kleij, and M. de Kamps, "Lack of combinatorial productivity in language processing with simple recurrent networks," Connect. Sci., vol. 16, no. 1, pp. 21-46, 2004.
24. M. Tomita, "Dynamic construction of finite-state automata from examples using hill-climbing," in Proc. 4th Annu. Cognit. Sci. Conf., 1982, pp. 105-108.