Model structure selection using an integrated forward orthogonal search algorithm assisted by squared correlation and mutual information

International Journal of Modelling, Identification and Control

Volumn 3, Issue 4, 2008, Pages 341-356

  Wei, Hua Liang ^a   Billings, Stephen A ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Correlation; Hypothesis tests; Identification; Model selection; Mutual information; NARX NARMAX model

Indexed keywords

CORRELATION METHODS; IDENTIFICATION (CONTROL SYSTEMS); LINEAR SYSTEMS; RELIGIOUS BUILDINGS;

HYPOTHESIS TESTS; MODEL SELECTION; MODEL SELECTION CRITERIA; MODEL STRUCTURE SELECTION; MUTUAL INFORMATIONS; NON-LINEAR SYSTEM IDENTIFICATION; ORTHOGONAL LEAST SQUARES; SEARCH ALGORITHMS;

NONLINEAR SYSTEMS;

EID: 53649108664     PISSN: 17466172     EISSN: 17466180     Source Type: Journal    
DOI: 10.1504/IJMIC.2008.020543     Document Type: Article

References (82)

1. Aguirre, L.A., Barroso, M.F.S., Saldanha, R.R. and Mendes, E.M.A.M. (2004) 'Imposing steady-state performance on identified nonlinear polynomial models by means of constrained parameter estimation', IEE Proceedings-Control Theory and Applications, Vol. 151, No. 2, pp.174-179.
2. Aguirre, L.A. and Billings, S.A. (1994) 'Validating identified nonlinear models with chaotic dynamics', International Journal of Bifurcation and Chaos, Vol. 4, pp.109-125.
3. Aguirre, L.A. and Billings, S.A. (1995a) 'Dynamical effects of overparametrization in nonlinear models', Physica D, Vol. 80, pp.26-40.
4. Aguirre, L.A. and Billings, S.A. (1995b) 'Improved structure selection for nonlinear models based on term clustering', International Journal of Control, Vol. 62, pp.569-587.
5. Aguirre, L.A. and Billings, S.A. (1995c) 'Retrieving dynamical invariants from chaotic data using NARMAX models', International Journal of Bifurcation and Chaos, Vol. 5, pp.449-474.
6. Aguirre, L.A., Correa, M.V. and Cassini, C.C.S. (2002) 'Nonlinearities in NARX polynomial models: Representation and estimation', IEE Proceedings-Control Theory and Applications, Vol. 149, No. 4, pp.343-348.
7. Aguirre, L.A., Donoso-Garcia, P.F. and Santos, R. (2000) 'Use of a priori information in the identification of global nonlinear models - a case study using a buck converter', IEEE Transactions on Circuits and Systems I-Fundamental Theory and Applications, Vol. 47, No. 7, pp.1081-1085.
8. Akaike, H. (1974) 'A new look at the statistical model identification', IEEE Transactions on Automatic Control, Vol. 19, pp.716-723.
9. Allen, D.M. (1974) 'The relationship between variable selection and data augmentation and a method for prediction', Technometrics, Vol. 16, pp.125-127.
10. Anders, U. and Korn, O. (1999) 'Model selection in neural networks', Neural Networks, Vol. 12, pp.309-323.
11. Barron, A.R. and Xiao, X.Y. (1991) 'Discussion: Multivariate adaptive regression splines', Annals of Statistics, Vol. 19, No. 1, pp.67-82.
12. Battiti, R. (1994) 'Using mutual information for selecting features in supervised neural net learning', IEEE Transactions on Neural Networks Vol. 5, pp.537-550.
13. Billings, S.A. and Aguirre, L.A. (1995) 'Effects of the sampling time on the dynamics and identification of nonlinear models', International Journal of Bifurcation and Chaos, Vol. 5, pp.1541-1556.
14. Billings, S.A. and Chen, S. (1998) 'The determination of multivariable nonlinear models for dynamic systems using neural networks', in C.T. Leondes (Ed). Neural Network Systems Techniques and Applications, San Diego: Academic Press, pp.231-278.
15. Billings, S.A., Chen, S. and Backhouse, R.J. (1989a) 'The identification of linear and non-linear models of a turbo charged diesel-engine', Mechanical Systems and Signal Processing, Vol. 3, No. 2, pp.123-142.
16. Billings, S.A., Chen, S. and Korenberg, M.J. (1989b) 'Identification of MIMO non-linear systems using a forward-regression orthogonal estimator', International Journal of Control, Vol. 49, pp.2157-2189.
17. Billings, S.A., Korenberg, M. and Chen, S. (1988) 'Identification of nonlinear output-affine systems using an orthogonal least-squares algorithm', International Journal of Systems Science, Vol. 19, pp.1559-1568.
18. Billings, S.A. and Wei, H.L. (2005a) 'A new class of wavelet networks for nonlinear system identification', IEEE Transactions on Neural Networks, Vol. 16, pp.862-874.
19. Billings, S.A. and Wei, H.L. (2005b) 'The wavelet-NARMAX representation: a hybrid model structure combining the polynomial models and multiresolution wavelet decompositions', International Journal of Systems Science, Vol. 36, pp.137-152.
20. Billings, S.A. and Wei, H.L. (2007) 'Sparse model identification using a forward orthogonal regression algorithm aided by mutual information', IEEE Transactions on Neural Networks, Vol. 18, No. 1, pp.306-310.
21. Billings, S.A. and Wei, H.L. (2008) 'An adaptive orthogonal search algorithm for model subset selection and nonlinear system identification', International Journal of Control, Vol. 81, No. 5, pp.714-724.
22. Bomberger, J.D. and Seborg, D.E. (1998) 'Determination of model order for NARX models directly from input-output data', Journal of Process Control, Vol. 8, pp.459-468.
23. Breiman, L. (1995) 'Better subset regression using the nonnegative garrote', Technometrics, Vol. 37, pp.373-384.
24. Breiman, L. and Freedman, D. (1983) 'How many variables should be entered in a regression equation', Journal of the American Statistical Association, Vol. 78, pp. 131-136.
25. Chen, S., Billings, S.A. and Luo, W. (1989) 'Orthogonal least-squares methods and their application to non-linear system-identification', International Journal of Control, Vol. 50, pp.1873-1896.
26. Chen, S., Chng, E.S. and Alkadhimi, K. (1996) 'Regularized orthogonal least squares algorithm for constructing radial basis function networks', International Journal of Control, Vol. 64, pp.829-837.
27. Chen, S., Hong, X. and Harris, C.J. (2003) 'Sparse kernel regression modeling using combined locally regularized orthogonal least squares and D-optimality experimental design', IEEE Transactions on Automatic Control, Vol. 48, pp. 1029-1036.
28. Chen, S., Wang, X.X. and Brown, D.J. (2005) 'Sparse incremental regression modeling using correlation criterion with boosting search', IEEE Signal Processing Letters, Vol. 12, pp.198-201.
29. Cover, T.M. and Thomas, J.A. (1991) Elements of Information Theory New York: John Wiley & Sons.
30. Darbellay, G.A. and Vajda, I. (1999) 'Estimation of the information by an adaptive partitioning of the observation space', IEEE Transactions on Information Theory, Vol. 45, No. 4, pp.1315-1321.
31. Davis, G., Mallat, S. and Zhang, Z. (1994) 'Adaptive time-frequency decompositions', Optical Engineering, Vol. 33, pp.2183-2191.
32. Feil, B., Abonyi, J. and Szeifert, F. (2004) 'Model order selection of nonlinear input-output models - a clustering based approach', Journal of Process Control, Vol. 14, No. 6, pp.593-602.
33. Friedman, J.H. and Silverman, B.W. (1989) 'Flexible parsimonious smoothing and additive modeling', Technometrics, Vol. 31, No. 1, pp.3-21.
34. Golub, G.H., Heath, M. and Wahha, G. (1979) 'Generalized cross-validation as a method for choosing a good ridge parameter', Technometrics, Vol. 21, pp.215-223.
35. Haber, R. and Unbehauen, H. (1990) 'Structure identification of nonlinear dynamic-systems - a survey on input output approaches', Automatica, Vol. 26, pp.651-677.
36. Harris, C.J., Hong, X. and Gan, Q. (2002) Adaptive Modelling, Estimation and Fusion from Data: A Neurofuzzy Approach, Berlin; London: Springer-Verlag.
37. Haykin, S. (1999) Neural Networks: A Comprehensive Foundation, 2nd edition, New York: Macmillan; Oxford: Maxwell, Macmillan International.
38. Haynes, B.R. and Billings, S.A. (1994) 'Global analysis and model validation in nonlinear-system identification', Nonlinear Dynamics Vol. 5, No. 1, pp.93-130.
39. Hocking, R.R. (1976) 'Analysis and selection of variables in lineax-regression', Biometrics, Vol. 32, pp.l-49.
40. Hocking, R.R. (1983) 'Developments in linear-regression methodology: 1959-1982', Technometrics, Vol. 25, pp.219-230.
41. Hong, X. and Chen, S. (2005) 'M-estimator and D-optimality model construction using orthogonal forward least regression', IEEE Transactions on Systems, Man, and Cybernetics, Part B, Vol. 30, pp.155-162.
42. Hong, X. and Harris, C.J. (2002) 'Nonlinear model structure design and construction using orthogonal least squares and D-optimality design', IEEE Transactions on Neural Networks, Vol. 13, pp.1245-1250.
43. Huang, J.H.Z. and Yang, L.J. (2004) 'Identification of non-linear additive autoregressive models', Journal of Royal Statistical Society Series B-Statistical Methodology, Part 2, Vol. 66, pp.463-477.
44. Korenberg, M., Billings, S.A., Liu, Y.R and McIlroy, P.J. (1988) 'Orthogonal parameter estimation algorithm for non-linear stochastic systems', International Journal of Control, Vol. 48, pp.193-210.
45. Leontaritis, I.J. and Billings, S.A. (1985) 'Input-output parametric models for non-linear systems, part I: Deterministic non-linear systems', International Journal of Control, Vol. 41, pp.303-344.
46. Leontaritis, I.J. and Billings, S.A. (1987) 'Experimental-design and identifiability for nonlinear-systems', International Journal of Systems Science, Vol. 18, pp.189-202.
47. Lind, I. and Ljung, L. (2005) 'Regressor selection with the analysis of variance method', Automatica, Vol. 41, pp.693-700.
48. Ljung, L. (1987) System Identification: Theory for the User, Englewood Cliffs: Prentice-Hall.
49. Madax, J., Abonyi, J. and Szeifert, F. (2005) 'Genetic programming for the identification of nonlinear input-output models', Industrial and Engineering Chemistry Research, Vol. 44, No. 9, pp.3178-3186.
50. Mallat, S.G. and Zhang, Z. (1993) 'Matching pursuits with time-frequency dictionaries', IEEE Transactions on Signal Processing, Vol. 41, pp.3397-3415.
51. Mao, K.Z. and Billings, S.A. (1997) 'Algorithms for minimal model structure detection in nonlinear dynamic system identification', International Journal of Control, Vol. 68, pp.311-330.
52. Mendes, E. and Billings, S.A. (1993) Orthogonal Estimator with Intelligent Structure Detection: Users Guide, Department of ACSE, The University of Sheffield, UK.
53. Miller, A.J. (1990) Subset Selection in Regression, London: Chapman and Hall.
54. Moddemeijer, R. (1989) 'On estimation of entropy and mutual information of continuous distributions', Signal Processing, Vol. 16, No. 3, pp.233-246.
55. Moddemeijer, R. (1999) 'A statistic to estimate the variance of the histogram-based mutual information estimator based on dependent pairs of observations', Signal Processing, Vol. 75, No. 1, pp.51-63.
56. Montgomery, D.C., Peck, E.A. and Vining, G.G. (2001) Introduction to Linear Regression Analysis, 3rd edition, New York: John Wiley & Sons.
57. Moody, J.E. (1992) 'The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems', in J.E. Moody, S.J. Hanson and R.P. Lippmann (Eds). Advances in Neural Information Processing Systems (NIPS), San Mateo, CA: Morgan Kaufmann, pp.847-854.
58. Nelles, O. (2001) Nonlinear System Identification: From Classical Approach to Neural Networks and Fuzzy Models, Berlin: Springer-Verlag.
59. Orr, M.J.L. (1995) 'Regularization in the selection of radial basis function centers', Neural Computation, Vol. 7, pp.606-623.
60. Orr, M.J.L. (1998) 'Optimising the widths of radial basis functions', The Fifth Brazilian Symposium on Neural Networks, pp.26-29, Belo Horizonte, Brazil, 9-11 December.
61. Paninski, L. (2003) 'Estimation of entropy and mutual information', Neural Computation, Vol. 15, pp. 1191-1253.
62. Pearson, R.K. (1999) Discrete-Time Dynamic Models, Oxford: Oxford University Press.
63. Piroddi, L. and Spinelli, W. (2003) 'An identification algorithm for polynomial NARX models based on simulation error minimization', International Journal of Control, Vol. 76, pp.1767-1781.
64. Rech, G., Terasvirta, T. and Tschernig, R. (2001) 'A simple variable selection technique for nonlinear models', Communications in Statistics-Theory and Methods, Vol. 30, pp.1227-1241.
65. Rissanen, J. (1978) 'Modelling by shortest data description', Automatica, Vol. 14, pp.465-471.
66. Savit, R. and Green, M. (1991) 'Time series and dependent variables', Physica D, Vol. 50, pp.95-116.
67. Schwarz, G. (1978) 'Estimating the dimension of a model', The Annals of Statistics, Vol. 6, pp.461-464.
68. Shao, J. (1993) 'Linear-model selection by cross-validation', Journal of the American Statistical Association, Vol. 88, pp.486-494.
69. Sjoberg, J. and Ljung, L. (1995) 'Overtraining, regularization and searching for a minimum, with application to neural networks', International Journal of Control, Vol. 62, pp.1391-1407.
70. Söderström, T. and Stoica, R (1989) System Identification, New York: Prentice Hall.
71. Stark, J.A. and Fitzgerald, W.J. (1995) 'Parameter-based hypothesis tests for model selection', Signal Processing, Vol. 46, pp.169-178.
72. Stoica, P., Eykhoff, P., Janssen, P. and Soderstrom, T. (1986) 'Model-structure selection by cross-validation', International Journal of Control, Vol. 43, pp.1841-1878.
73. Stoica, P. and Selen, Y. (2004) 'Model-order selection: A review of information criterion rules', IEEE Signal Proceedings of Magazine, Vol. 21, pp.36-47.
74. Stone, M. (1974) 'Cross-validatory choice and assessment of statistical predictions', Journal of Royal Statistical Society Series B- Methodology, Vol. 36, No. 2, pp.111-147.
75. Tjostheim, D. and Auestad, B.H. (1994) 'Nonparametric identification of nonlinear time series: Projections', Journal of the American Statistical Association, Vol. 89, pp.1398-1409.
76. Vieu, P. (1995) 'Order choice in nonlinear autoregressive models', Statistics, Vol. 26, pp.307-328.
77. Wei, H.L., Billings, S.A. and Liu, J. (2004) 'Term and variable selection for nonlinear system identification', International Journal of Control, Vol. 77, pp.86-110.
78. Zheng, G.L. and Billings, S.A. (1996) 'Radial basis function networks configuration using mutual information and the orthogonal least squares algorithm', Neural Networks, Vol. 9, pp.1619-1637.
79. Zheng, G.L. and Billings, S.A. (1999) 'Qualitative validation and generalization in non-linear system identification', International Journal of Control, Vol. 72, No. 17, pp.1592-1608.
80. Zheng, G.L. and Billings, S.A. (2002) 'Effects of overparametrization in non-linear system identification and neural networks', International Journal of Systems Science, Vol. 33, No. 5, pp.331-349.
81. Zhu, D., Balikhin, M. and Billings, S.A. (2005) 'Variable and term selection for the identification of MIMO nonlinear systems', Research report no. 893, Department of Automatic Control and Systems Engineering, University of Sheffield.
82. Zhu, Q.M. and Billings, S.A. (1996) 'Fast orthogonal identification of nonlinear stochastic models and radial basis function neural networks', International Journal of Control, Vol. 64, No. 5, pp.871-886.