Segmented-memory recurrent neural networks

IEEE Transactions on Neural Networks

Volumn 20, Issue 8, 2009, Pages 1267-1280

  Chen, Jinmiao ^a,b   Chaudhari, Narendra S ^a  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b AGENCY FOR SCIENCE   (Singapore)

Author keywords

Gradient descent; Information latching; Long term dependencies; Recurrent neural networks (RNNs); Segmented memory; Vanishing gradient

Indexed keywords

GRADIENT DESCENT; INFORMATION LATCHING; LONG-TERM DEPENDENCIES; SEGMENTED MEMORY; VANISHING GRADIENT;

LEARNING ALGORITHMS; REINFORCEMENT LEARNING;

RECURRENT NEURAL NETWORKS;

PROTEIN;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; CHEMISTRY; FORECASTING; GENETICS; HUMAN; MEMORY; METHODOLOGY; PROTEIN SECONDARY STRUCTURE; SEQUENCE ANALYSIS;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; FORECASTING; HUMANS; MEMORY; NEURAL NETWORKS (COMPUTER); PROTEIN STRUCTURE, SECONDARY; PROTEINS; SEQUENCE ANALYSIS, PROTEIN;

EID: 68949200806     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2009.2022980     Document Type: Article

References (49)

1. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157-166, Mar. 1994.
2. R. Boné, M. Crucianu, and J. Beauville, "Learning long-term dependencies by the selective addition of time-delayed connections to recurrent neural networks," Neurocomputing, vol. 48, pp. 251-266, 2002.
3. J. Chen and N. S. Chaudhari, "Statistical analysis of long-range interactions in proteins," in Proc. Int. Conf. Bioinf. Comput. Biol. Las Vegas, NV, Jun. 2006, pp. 296-302.
4. J. M. Chen and N. S. Chaudhari, "Learning long-term dependencies in segmented memory recurrent neural networks," in Advances in Neural Networks ISNN 2004, ser. Lecture Notes in Computer Science. Berlin, Germany: Springer, 2004, vol. 3173, pp. 362-369.
5. J. M. Chen and N. S. Chaudhari, "Improvement of bidirectional recurrent neural network for learning long-term dependencies," in Proc. 17th Int. Conf. Pattern Recognit., Los Alamitos, CA, Aug. 2004, vol. 4, pp. 593-596.
6. S. Y. Cho, Z. Chi, W. C. Siu, and A. C. Tsoi, "An improved algorithm for learning long-term dependency problems in adaptive processing of data structures," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 781-793, Jul. 2003.
7. S. Doboli and A. A. Minai, "Using latent attractors to discern temporal order," in Proc. IEEE Int. Joint Conf. Neural Netw., 2004, vol. 2, pp. 1469-1474.
8. S. Doboli and A. A. Minai, "Latent attractors: A general paradigm for context-dependent neural computation," in Trends in Neural Computation, K. Chen and L. Wang, Eds. New York: Springer-Verlag, 2007, vol. 35, pp. 135-169.
9. J. L. Elman, "Distributed representations, simple recurrent networks, and grammatical structure," Mach. Learn., vol. 7, no. 2/3, pp. 195-226, 1991.
10. A. Fiser, Z. Dosztanyi, and I. Simon, "The role of long-range interactions in defining the secondary structure of proteins is overestimated," Comput. Appl. Biosci., vol. 13, no. 3, pp. 297-301, 1997.
11. M. L. Forcada and R. C. Carrasco, "Learning the initial state of a second-order recurrent neural network during regular-language inference," Neural Comput., vol. 7, pp. 923-930, 1995.
12. R. W. Frick, "Explanations of grouping effects in immediate ordered recall," Cognition, vol. 17, pp. 551-562, 1989.
13. J. Gibrat, J. Garnier, and B. Robson, "Further developments of protein secondary structure prediction using information theory," J. Mol. Biol., vol. 198, pp. 425-443, 1987.
14. C. L. Giles, C. B. Miller, D. Chen, H. H. Chen, G. Z. Sun, and Y. C. Lee, "Learning and extracting finite state automata with second-order recurrent neural networks," Neural Comput., vol. 4, no. 3, pp. 393-405, 1992.
15. S. E. Hihi and Y. Bengio, "Hierarchical recurrent neural networks for long-term dependencies," in Advances in Neural Information Processing Systems, M. Perrone, M. Mozer, and D. D. Touretzky, Eds. Cambridge, MA: MIT Press, 1996, pp. 493-499.
16. G. J. Hitch, N. Burgess, J. N. Towse, and V. Culpin, "Temporal grouping effects in immediate recall: A working memory analysis," Quart. J. Exp. Psychol., vol. 49A, pp. 116-139, 1996.
17. S. Hochreiter, "Untersuchungen zu dynamischen neuronalen netzen," Diploma thesis, Institut fur Informatik, Technische Universitat Munchen, Munchen, Germany, 1991.
18. S. Hochreiter and J. Schmidhuber, "Lstm can solve hard long time lag problems," in Advances in Neural Information Processing Systems 9 M. C. Mozer, M. I. Jordan, and T. Petsche, Eds. Cambridge, MA: MIT Press, 1997, pp. 473-479.
19. S. Hua and Z. Sun, "A novel method of protein secondary structure prediction with high segment overlap measure: Support vector machine approach," J. Mol. Biol., vol. 308, pp. 397-407, 2001.
20. D. T. Jones, "Protein secondary structure prediction based on positionspecific scoring matrices," J. Mol. Biol., vol. 292, pp. 195-202, 1999.
21. K. W. C. Ku,M.W.Mak, and W. C. Siu, "A cellular genetic algorithm for training recurrent neural networks," in Proc. Int. Conf. Neural Netw. Signal Process., 1995, pp. 140-143.
22. T. Lin, B. G. Horne, and C. L. Giles, "How embedded memory in recurrent neural network architectures helps learning long-term temporal dependencies," Neural Netw., vol. 11, pp. 861-868, 1998.
23. T. Lin, B. G. Horne, P. Tino, and C. L. Giles, "Learning long-term dependencies in NARX recurrent neural networks," IEEE Trans. Neural Netw., vol. 7, no. 6, pp. 1329-1337, Nov. 1996.
24. S. Ma and C. Ji, "Fast training of recurrent networks based on the em algorithms," IEEE Trans. Neural Netw., vol. 9, no. 1, pp. 11-26, Jan. 1998.
25. B. Matthews, "Comparison of the predicted and observed secondary structure of t4 phage lysozyme," Biochim Biophys Acta, vol. 405, no. 2, pp. 442-451, 1975.
26. X. M. Pan, W. D. Niu, and Z. X.Wang, "What is the minimum number of residues to determine the secondary structural state," J. Protein Chem., vol. 18, no. 5, pp. 579-584, 1999.
27. G. V. Puskorius and L. A. Feldkamp, "Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 279-297, Mar. 1994.
28. A. J. Robinson and F. Fallside, "Static and dynamic error propagation networks with application to speech coding," in Proc. Neural Inf. Process. Syst., 1987, pp. 632-641.
29. B. Rost and C. Sander, "Prediction of protein secondary structure at better than 70% accuracy," J. Mol. Biol., vol. 232, pp. 584-599, 1993.
30. J. Ryan, "Temporal grouping, rehearsal and short-term memory," Quart. J. Exp. Psychol., vol. 21, pp. 148-155, 1969.
31. J. Schmidhuber, "Adaptive decomposition of time," in Artificial Neural Networks, T. Kohonen, K. Mikisara, O. Simula, and J. Kangas, Eds. New York: Elsevier, 1991, pp. 909-914.
32. J. Schmidhuber, "Learning complex, extended sequences using the principle of history compression," Neural Comput., vol. 4, no. 2, pp. 234-242, 1992.
33. J. Schmidhuber, M. C. Mozer, and D. Prelinger, "Continuous history compression," in Proc. Int. Workshop Neural Netw., H. Huning, S. Neuhauser, M. Raus, and W. Ritschel, Eds., 1993, pp. 87-95.
34. N. N. Schraudolph, "Local gain adaptation in stochastic gradient descent," in Proc. 9th Int. Conf. Artif. Neural Netw., London, U.K., 1999, pp. 569-574.
35. N. N. Schraudolph, "Fast curvature matrix-vector products for secondorder gradient descent," Neural Comput., vol. 14, no. 7, pp. 1723-1738, 2000.
36. M. Schuster and K. K. Paliwal, "Bidirectional recurrent neural networks," IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673-2681, Nov. 1997.
37. F. T. Severin and M. K. Rigby, "Influence of digit grouping on memory for telephone numbers," J. Appl. Psychol., vol. 47, pp. 117-119, 1963.
38. Q. Song, Y. L. Wu, and Y. C. Soh, "Robust adaptive gradient-descent training algorithm for recurrent neural networks in discrete time domain," IEEE Trans. Neural Netw., vol. 19, no. 11, pp. 1841-1853, Nov. 2008.
39. S. Squartini, A. Hussain, and F. Piazza, "Attempting to reduce the vanishing gradient effect through a novel recurrent multiscale architecture," in Proc. Int. Joint Conf. Neural Netw., 2003, vol. 4, pp. 2819-2824.
40. B. Ster, "Latched recurrent neural network," Elektrotehni.ski vestnik, vol. 70, no. 1-2, pp. 46-51, 2003.
41. P. F. M. J. Verschure and P. Althaus, "A real-world rational agent: Unifying old and new AI," Cogn. Sci., vol. 27, pp. 561-590, 2003.
42. P. F. M. J. Verschure, T. Voegtlin, and R. J. Douglas, "Environmentally mediated synergy between perception and behavior in mobile robots," Nature, vol. 425, pp. 620-624, 2003.
43. D. Wang, "Anticipation model for sequential learning of complex sequences," in Sequence Learning: Paradigms, Algorithms, and Applications, ser. Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2000, vol. 1828, pp. 53-79.
44. W. A. Wickelgren, "Rehearsal grouping and hierarchical organization of serial position cues in short-term memory," Quart. J. Exp. Psychol. vol. 19, pp. 97-102, 1967.
45. R. J. Williams and Zipser, "Gradient based learning algorithms for recurrent connectionist networks," College Comput. Sci., Northeastem Univ., Boston, MA, Tech. Rep. NU-CCS-90-9, 1990.
46. Y. L. Wu, Q. Song, and S. Liu, "A normalized adaptive training of recurrent neural networks with augmented error gradient," IEEE Trans. Neural Netw., vol. 19, no. 2, pp. 351-356, Feb. 2008.
47. T. Yi and E. Lander, "Protein secondary structure prediction using nearest-neighbor methods," J. Mol. Biol., vol. 232, pp. 1117-1129, 1993.
48. A. Zemla, C. Venclovas, K. Fidelis, and B. Rost, "A modified definition of SOV, a segment-based measure for protein secondary structure prediction assessment," Proteins: Structure, Function, Genetics, vol. 34, no. 2, pp. 220-223, 1999.
49. H. G. Zhang, Z. S. Wang, and D. R. Liu, "Global asymptotic stability of recurrent neural networks with multiple time-varying delays," IEEE Trans. Neural Netw., vol. 19, no. 5, pp. 855-873, May 2008.