A delay damage model selection algorithm for narx neural networks

IEEE Transactions on Signal Processing

Volumn 45, Issue 11, 1997, Pages 2719-2730

  Lin, Tsung Nan ^a   Lee Giles, C ^a   Home, Bill G ^a   Kung, Sun Yuan ^a  

^a NEC LABORATORIES AMERICA   (United States)

Author keywords

Automata; Autoregressive; Embedding theory; Gradient descent training; Latching; Long term dependencies; Memory; NARX; Networks; Pruning; Recurrent neural networks; Tapped delay lines; Temporal sequences; Time series

Indexed keywords

AUTOMATA THEORY; DATA STORAGE EQUIPMENT; FINITE AUTOMATA; HEURISTIC METHODS; TIME SERIES ANALYSIS;

GRADIENT DESCENT METHOD; NONLINEAR AUTOREGRESSIVE MODELS WITH EXOGENOUS INPUTS (NARX) NETWORKS; TAPPED DELAY LINES;

NEURAL NETWORKS;

EID: 0031274628     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/78.650098     Document Type: Article

References (72)

1. H. Akaike, "A new look at the statistical model identification," IEEE Trans. Automat. Contr.. vol. AC-19, pp. 716-723, 1974.
2. W. Atmar, "Notes on the simulation of evolution," IEEE Trans. Neural Networks, vol. 5, pp. 130-148, 1994.
3. A. D. Back and A. C. Tsoi, "A comparison of discrete-time operator models for nonlinear system identification," in Advances in Neural Information Processing Systems 7, G. Tesauro, D. Touretzky, and T. Leen, Eds. Cambridge, MA: MIT Press, 1995, pp. 883-890.
4. U. Bodenhausen and A. Waibel, "The tempo 2 algorithm: Adjusting time-delays by supervised learning," in Advances in Neural Information Processing Systems 3. San Mateo, CA: Morgan Kaufmann, 1991, pp. 155-161.
5. Y. Cauvin, "A back-propagation algorithm with optimal use of hidden units," in Advances in Neural Information Processing Systems I, D. Touretzky, Ed., 1989, pp. 519-526.
6. S. Chen, S. A. Billings, and P. M. Grant, "Non-linear system identification using neural networks," Int. J. Contr., vol. 51, no. 6, pp. 1191-1214, 1990.
7. Y.-F. Cheng and D. M. Etter, "Analysis of an adaptive technique for modeling sparse systems," IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 254-264, Feb. 1989.
8. D. S. Clouse, C. L. Giles, B. G. Home, and G. W. Cottrell, "Time-delay neural networks: Representation and induction of finite state machines," IEEE Trans. Neural Networks, vol. 8, pp. 1065-1070, 1997.
9. J. Connor, L. E. Atlas, and D. R. Martin, "Recurrent networks and NARMA modeling," in Advances in Neural Information Processing Systems 4, 3. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds., 1992, pp. 301-308.
10. Y. Le Cun, J. S. Denker, and S. A. Solla, "Handwritten digit recognition with a backpropagation network," Advances in Neural Information Processing Systems, 1990, vol. 2, pp. 396104.
11. -, "Optimal brain damage," in Advances in Neural Information Processing Systems, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, vol. 2, pp. 598-605.
12. G. Cybenko, "Approximation by superpositions of a sigmoidal function," Math. Contr., Signals, Syst., vol. 2, no. 4, pp. 303-314, 1989.
13. J. E. Dayhoff, D.-T. Lin, and P. A. Ligomenides, "A learning algorithm for adaptive time-delays in a temporal neural network," Syst. Res. Center, Univ. Maryland, College Park, Tech. Rep. SRC-TR-92-59, 1992.
14. S. P. Day and M. R. Davenport, "Continuous-time temporal backpropagation with adaptable time delays," IEEE Trans. Neural Networks, vol. 4, pp. 348-354, 1993.
15. D. M. Etter and S. D. Stearns, "Adaptive estimation of time delays in sampled data systems," IEEE Trans. AcousL, Speech, Signal Processing, vol. ASSP-29, pp. 582-587, 1981.
16. D. B. Fogel, "An information criterion for optimal neural network selection," IEEE Trans. Neural Networks, vol. 2, pp. 490-497, 1991.
17. -, "Using evolutionary programming for modeling: An ocean acoustic example," IEEE J. Ocean. Eng., vol. 17, pp. 333-340, 1992.
18. ___, "Evolutionary programming: An introduction and some current directions," Statist. Comput., vol. 4, pp. 113-129, 1994.
19. K. Funahashi, " On the approximate realization of continuous mappings by neural networks," Neural Networks, vol. 2, no. 3, pp. 183-192, 1989.
20. C. L. Giles, B. G. Home, and T. Lin, "Learning a class of large finite state machines with a recurrent neural network," Inst. Adv. Comput. Studies, Univ. Maryland, College Park, Tech. Rep. UMIACS-TR-94-94 and CS-TR-3328, 1994.
21. ___, "Learning a class of large finite state machines with a recurrent neural network," Neural Networks, vol. 8, no. 9, pp. 1359-1365, 1995.
22. C. L. Giles, T.-N. Lin, and B. G. Home, "Remembering the past: The role of embedded memory in recurrent neural network architectures," in Proc. Neural Networks Signal Processing VII 1997 IEEE Workshop,
23. J. Principe, L. Giles, N. Morgan, and E. Wilson, Eds. Piscataway, NJ: IEEE, 1997, pp. 34-43.
24. C. L. Giles and C. W. Omlin, "Pruning recurrent neural networks for improved generalization performance," IEEE Trans. Neural Networks, vol. 5, pp. 848-851, 1994.
25. F. Girosi and T. Poggio, "Networks and the best approximation property," Biolog. Cybern., vol. 63, pp. 169-176, 1990.
26. E. J. Hannan and B. G. Quinn, "The determination of the order of an autoregression," J. Royal Stat. Soc. B., vol. 41, pp. 190-195, 1979.
27. B. Hassibi and D. G. Stork, "Second order derivatives for network pruning: Optimal brain surgeon," in Advances in Neural Information Processing Systems 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993.
28. B. G. Hörne and C. L. Giles, "An experimental comparison of recurrent neural networks," in Advances in Neural Information Processing Systems
29. G. Tesauro, D. Touretzky, and T. Leen, Eds. Cambridge, MA: MIT Press, 1995, pp. 697-704.
30. K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Networks, vol. 2, no. 5, pp. 359-366, 1989.
31. B. Irie and S. Miyaké, "Capabilities of three-layered perceptrons," in Proc. IEEE Int. Conf. Neural Networks, San Diego, CA, 1988, vol. 1, pp. 641-648.
32. E. D. Karnin, "A simple procedure for pruning back-propagation trained neural networks," IEEE Trans. Neural Networks, vol. 1, pp. 239-242, 1990.
33. A. Krogh and J. A. Hertz, "A simple weight decay can improve generalization," in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds., 1992, pp. 950-957.
34. K. J. Lang, A. H. Waibel, and G. E. Hinton, "A time-delay neural network architecture for isolated word recognition," Neural Networks, vol. 3, no. 1, pp. 2-4, 1990.
35. A. Lapedes and R. Farber, "Nonlinear signal processing using neural networks: Prediction and signal modeling," Los Alamos Nat. Labs., Los Alamos, NM, Tech. Rep. LA-UR-87-2662, 1987.
36. I. J. Leontaritis and S. A. Billings, "Input-output parametric models for nonlinear systems: Part I: Deterministic nonlinear systems," Int. J. Contr., vol. 41, no. 2, pp. 303-328, 1985.
37. D. T. Lin, J. E. Dayhoff, and P. A. Ligomenides, "Trajectory production with the adaptive time-delay neural network," Neural Networks, vol. 8, no. 3, pp. 447-461, 1995.
38. T.-N. Lin, B. G. Hörne, P. Tino, and C. L. Giles, "Learning long-term dependencies in narx recurrent neural networks," IEEE Trans. Neural Networks, vol. 7, pp. 1329-1338, 1996.
39. T.-N. Lin, B. G. Hörne, and C. L. Giles, "How memory orders effect the performance of narx networks," Inst. Adv. Comput. Studies, Univ. Maryland, College Park, Tech. Rep. UMIACS-TR-96-76 and CS-TR-3706, 1996.
40. T. Lin, B. G. Hörne, P. Tino, and C. L. Giles, "Learning long-term dependencies is not as difficult with narx recurrent neural networks," in Advances in Neural Information Processing Systems 8. Cambridge, MA: MIT Press, 1996.
41. L. Ljung, System Identification: Theory for the User. Englewood Cliffs, NJ: Prentice-Hall, 1987.
42. G. A. N. Mbamalu and M. E. El-Hawary, "Load forecasting via suboptimal seasonal autoregressive models and iteratively reweighted least squares estimation," IEEE Trans. Power Syst., vol. 8, pp. 343-348, 1993.
43. D. C. Montgomery and E. A. Peck, Introduction to Linear Regression Analysis. New York: Wiley, 1982.
44. J. E. Moody, "The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems," in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 847-854.
45. M. C. Mozer and P. Smolensky, "Skeletonization: A technique for trimming the fat from a network via relevance assessment," Connection Sei., vol. 11, pp. 3-26, 1989.
46. K. S. Narendra and K. Parthasarathy, "Identification and control of dynamical systems using neural networks," IEEE Trans. Neural Networks, vol. 1, pp. 4-27, 1990.
47. G. Nave and A. Cohen, "Ecg compression using long-term prediction," IEEE Trans. Biomed. Eng., vol. 40, pp. 877-885, 1993.
48. R. Mane, "On the dimension of the compact invariant sets of certain nonlinear maps," in Lecture Notes in Math. no. 898. New York: Springer-Verlag, 1981, p. 230.
49. S. J. Nowlan and G. E. Hinton, "Simplifying neural networks by soft weight-sharing," Neural Comput., vol. 4, no. 4, pp. 473193, 1992.
50. E. Ott, T. Sauer, and J. Yorke, Coping with Chaos. New York: Wiley, 1994.
51. N. Packard, J. Crutchfield, D. Farmer, and R. Shaw, "Geometry from a time series," Phys. Rev. Lett., vol. 45, pp. 712-715, 1980.
52. M. W. Pedersen and L. K. Hansen, "Recurrent networks: Second order properties and pruning," in Advances in Neural Information Processing Systems 7, G. Tesauro, D. Touretzky, and T. Leen, Eds. Cambridge, MA: MIT Press, 1995.
53. S.-Z. Qin, H.-T. Su, and T. J. McAvoy, "Comparison of four neural net learning methods for dynamic system identification," IEEE Trans. Neural Networks, vol. 3, pp. 122-130, 1992.
54. R. Reed, "Pruning algorithms-A survey," IEEE Trans. Neural Networks, vol. 4, pp. 740-747, 1993.
55. J. Rissanen, "Universal coding, information, prediction, and estimation," IEEE Trans. Inform. Theory, vol. IT-30, pp. 629-636, 1984.
56. T. Sauer, J. Yorke, and M. Casdagli, "Embedology," J. Statist. Phys., vol. 65, pp. 579-616, 1991.
57. R. Shibata, "Various model selection techniques in time series analysis," in Handbook of Statistics. Amsterdam, The Netherlands: Else vier, 1985, vol. 5, pp. 179-187.
58. H. T. Siegelmann, B. G. Hörne, and C. L. Giles, "Computational capabilities of recurrent narx neural networks," IEEE Trans. Syst., Man Cybern., pt. B, vol. 27, p. 208, Apr. 1997.
59. H.-T. Su and T. J. McAvoy, "Identification of chemical processes using recurrent networks," in Proc. Amer. Contr. Conf., 1991, vol. 3, pp. 2314-2319.
60. H.-T. Su, T. J. McAvoy, and P. Werbos, "Long-term predictions of chemical processes using recurrent neural networks: A parallel training approach," Ind. Eng. Chem. Res., vol. 31, pp. 1338-1352, 1992.
61. C. Svarer, L. K. Hansen, and J. Larsen, "On design and evaluation of tapped-delay neural architectures," in Proc. IEEE Int. Conf. Neural Networks, 1992, pp. 46-51.
62. F. Takens, "Detecting strange attractors in turbulence," in Lecture Notes Math. no. 898. New York: Springer-Verlag, 1981, pp. 366-381.
63. S. T. Venkataraman, "On encoding nonlinear oscillations in neural networks for locomotion," in Proc. Eighth Yale Workshop Adaptive Learning Syst., 1994, pp. 14-20.
64. B. de Vries and J. Principe, "The gamma model-A new neural network for temporal processing," Neural Networks, vol. 5, no. 4, pp. 565-576, 1992.
65. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang, "Phoneme recognition using time-delay neural networks," IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 328-339, 1989.
66. A. S. Weigend, B. A. Huberman, and D. E. Rumelhart, "Prediction the future: A connectionist approach," Int. J. Neural Syst., vol. 1, no. 3, pp. 193-209, 1990.
67. A. S. Weigend and N. A. Gershenfeld, Time Series Prediction: Forecasting the Future and Understanding the Past. Reading, MA: Addison-Wesley, 1994.
68. A. S. Weigend, B. A. Huberman, and D. E. Rumelhart, "Predicting sunspots and exchange rates with connectionist networks," in Nonlinear Modeling and Forecasting, SFI Studies in the Sciences of Complexity, M. Casdagli and S. Eubank, Eds. Reading, MA: Addison-Wesley, 1991, vol. 12.
69. A. S. Weigend, D. E. Rumelhart, and B. A. Huberman, "Generalization by weight-elimination with application to forecasting," in Advances in Neural Information Processing Systems 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, pp. 875-882.
70. R. J. Williams and D. Zipser, "Gradient-based learning algorithms for recurrent networks and their computational complexity," in BackPropagation: Theory, Architectures and Applications, Y. Chauvin and D. E. Rumelhart, Eds. Hillsdale, NJ: Lawrence Erlbaum, 1995, ch. 13, pp. 43386.
71. G.-H. Yu and Y.-C. Lin, "A methodology for selecting subset autoregressive time series models," J. Time Series Anal., vol. 12, no. 4, pp. 363-373, 1989.
72. E. A. Wan, "Combining fossil and sunspot data: Committee predictions," in Proc. Int. Conf. Neural Networks, 1997, pp. 2176-2180.