An improved algorithm for learning long-term dependency problems in adaptive processing of data structures

IEEE Transactions on Neural Networks

Volumn 14, Issue 4, 2003, Pages 781-793

  Cho, Siu Yeung ^a   Chi, Zheru ^a   Siu, Wan Chi ^a   Tsoi, Ah Chung ^b  

^a HONG KONG POLYTECHNIC UNIVERSITY   (Hong Kong)

^b UNIVERSITY OF WOLLONGONG   (Australia)

Author keywords

Adaptive processing of data structures; Backpropagation through structure (BPTS); Least squares method; Long term dependency

Indexed keywords

ALGORITHMS; BACKPROPAGATION; DATA STRUCTURES; LEAST SQUARES APPROXIMATIONS; NEURAL NETWORKS; OPTIMIZATION; PROBLEM SOLVING;

LONG-TERM DEPENDENCY PROBLEMS;

LEARNING SYSTEMS;

EID: 0042525850     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2003.813831     Document Type: Article

References (23)

1. A. C. Tsoi, "Adaptive Processing of Data Structure: An Expository Overview and Comments," Faculty Informatics, Univ. Wollongong, Wollongong, Australia, 1998.
2. _, "Gradient based learning methods," in Adaptive Processing of Sequences and Data Structures, C. L. Giles and M. Gori, Eds. New York: Springer-Verlag, 1998, pp. 27-62.
3. B. Hammer, Learning with Recurrent Neural Networks. New York: Springler-Verlag, 2000, vol. 254, Springer Lecture Notes in Control and Information Sciences.
4. A. Sperduti and A. Starita, "Supervised neural networks for classification of structures," IEEE Trans. Neural Networks, vol. 8, pp. 714-735, May 1997.
5. C. Goller and A. Kuchler, "Learning task-dependent distributed representations by back-propagation through structure," in Proc. IEEE Int. Conf. Neural Networks, 1996, pp. 347-352.
6. P. Frasconi, M. Gori, and A. Sperduti, "A general framework for adaptive processing of data structures," IEEE Trans. Neural Networks, vol. 9, pp. 768-785, Sept. 1998.
7. C. L. Giles and M. Gori, Adaptive Processing of Sequences and Data Structures. New York: Springer-Verlag, 1998.
8. P. Fraconi, M. Gori, A. Kuchler, and A. Sperduti, "From sequences to data structures: Theory and applications," in A Field Guild to Dynamical Recurrent Networks, J. Kolen and S. Kremer, Eds. New York: IEEE Press, 2001, ch. 19, pp. 351-374.
9. D. E. Rumelhart and J. L. McClelland, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Cambridge, MA: MIT Press, 1986.
10. M. Gori and A. Tesi, "On the problem of local minima in backpropagation," IEEE Trans. Pattern Anal. Machine Intell., vol. 14, pp. 76-86, Jan. 1992.
11. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Trans. Neural Networks, vol. 5, pp. 157-166, Mar. 1994.
12. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-80, 1997.
13. R. Battiti and G. Tecchiolli, "Training neural nets with the reactive tabu search," IEEE Trans. Neural Networks, vol. 6, pp. 1185-1200, Sept. 1995.
14. Y. Shang and B. W. Wah, "Global optimization for neural network training," IEEE Computer, vol. 29, pp. 45-54, Mar. 1996.
15. S. Y. Cho and T. W. S. Chow, "A layer-by-layer least squares based recurrent networks: Stalling and escape," Neural Processing Lett., vol. 7, no. 1, pp. 15-25, Feb. 1998.
16. S. Y. Cho and T. W. S. Cho, "Training multilayer neural networks using fast global learning algorithm - Least squares and penalized optimization methods," Neurocomput., vol. 25, pp. 115-131, 1999.
17. Z. Wang, Z. Chi, D. Feng, and S. Y. Cho, "Adaptive processing of tree-structure image representation," in Proc. 2nd IEEE Pacific-Rim Conf. Multimedia (PCM2001), Oct. 2001, pp. 989-995.
18. S. Y. Cho, Z. Chi, and W. C. Siu, "Image classification with adaptive processing of BSP image representation," presented at the ICSP2002.
19. B. Hammer and V. Sperschneider, "Neural networks can approximate mappings on structured objects," presented at the Proc. 2nd Int. Conf. Computational Intelligence Neuroscience, 1997.
20. Y. Bengio and P. Frasconi, "Input-output HMMs for sequence processing," IEEE Trans. Neural Networks, vol. 7, pp. 1231-49, Sept. 1996.
21. X. Wu, "Image coding by adaptive tree-structured segmentation, " IEEE Trans. Inform. Theory, vol. 38, pp. 1755-1767, Nov. 1992.
22. H. Radha, M. Vetterli, and R. Leonardi, "Image compression using binary space partitioning tree," IEEE Trans. Image Processing, vol. 5, pp. 1610-1624, Dec. 1996.
23. P. Salembier, "Binary partition tree as an efficient representation for image processing, segmentation, and information retrieval," IEEE Trans. Image Processing, vol. 9, pp. 561-576, Apr. 2000.