Orthogonal-least-squares regression: A unified approach for data modelling

Neurocomputing

Volumn 72, Issue 10-12, 2009, Pages 2670-2681

  Chen, S ^a   Hong, X ^b   Luk, B L ^c   Harris, C J ^a  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^b UNIVERSITY OF READING   (United Kingdom)

^c CITY UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Classification; Density estimation; Leave one out cross validation; Multiplicative nonnegative quadratic programming; Orthogonal least squares algorithm; Regression; Regularisation; Sparse kernel modelling

Indexed keywords

CLASSIFICATION; DENSITY ESTIMATION; LEAVE-ONE-OUT CROSS-VALIDATION; MULTIPLICATIVE NONNEGATIVE QUADRATIC PROGRAMMING; ORTHOGONAL-LEAST-SQUARES ALGORITHM; REGRESSION; REGULARISATION; SPARSE KERNEL MODELLING;

COMPUTATIONAL FLUID DYNAMICS; ESTIMATION; REGRESSION ANALYSIS;

PROBABILITY DENSITY FUNCTION;

ACCURACY; ALGORITHM; ARTICLE; CLASSIFICATION; KERNEL METHOD; MATHEMATICAL ANALYSIS; PRIORITY JOURNAL; PROBABILITY; RADIAL BASED FUNCTION; REGRESSION ANALYSIS; STATISTICAL MODEL;

EID: 67349249355     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neucom.2008.10.002     Document Type: Article

References (55)

1. Billings S.A., and Chen S. The determination of multivariable nonlinear models for dynamic systems. In: Leondes C.T. (Ed). Control and Dynamic Systems, Neural Network Systems Techniques and Applications vol. 7 (1998), Academic Press, San Diego, CA 231-278
2. Billings S.A., Chen S., and Backhouse R.J. The identification of linear and non-linear models of a turbocharged automotive diesel engine. Mech. Syst. Signal Process. 3 20 (1989) 123-142
3. Bishop C.M. Neural Networks for Pattern Recognition (1995), Oxford University Press, Oxford, UK
4. Bowman A.W., and Azzalini A. Applied Smoothing Techniques for Data Analysis (1997), Oxford University Press, Oxford, UK
5. Box G.E.P., and Jenkins G.M. Time Series Analysis, Forecasting and Control (1976), Holden Day, San Francisco, CA
6. Chapelle O., Vapnik V., Bousquet O., and Mukherjee S. Choosing multiple parameters for support vector machines. Machine Learning 46 1-3 (2002) 131-159
7. Chen S. Local regularization assisted orthogonal least squares regression. Neurocomputing 69 4-6 (2006) 559-585
8. Chen S., Billings S.A., and Luo W. Orthogonal least squares methods and their application to non-linear system identification. Int. J. Control 50 5 (1989) 1873-1896
9. Chen S., Chng E.S., and Alkadhimi K. Regularised orthogonal least squares algorithm for constructing radial basis function networks. Int. J. Control 64 5 (1996) 829-837
10. Chen S., Cowan C.F.N., and Grant P.M. Orthogonal least squares learning algorithm for radial basis function networks. IEEE Trans. Neural Networks 2 2 (1991) 302-309
11. Chen S., Hong X., and Harris C.J. Sparse kernel regression modelling using combined locally regularized orthogonal least squares and D-optimality experimental design. IEEE Trans. Autom. Control 48 6 (2003) 1029-1036
12. Chen S., Hong X., and Harris C.J. Sparse kernel density construction using orthogonal forward regression with leave-one-out test score and local regularization. IEEE Trans. Syst Man Cybern Part B 34 4 (2004) 1708-1717
13. Chen S., Hong X., and Harris C.J. An orthogonal forward regression technique for sparse kernel density estimation. Neurocomputing 71 4-6 (2008) 931-943
14. Chen S., Hong X., Harris C.J., and Sharkey P.M. Sparse modelling using orthogonal forward regression with PRESS statistic and regularization. IEEE Trans. Systems Man Cybern. Part B 34 2 (2004) 898-911
15. Chen S., and Wigger J. Fast orthogonal least squares algorithm for efficient subset model selection. IEEE Trans. Signal Process. 43 7 (1995) 1713-1715
16. Chen S., Wu Y., and Luk B.L. Combined genetic algorithm optimisation and regularised orthogonal least squares learning for radial basis function networks. IEEE Trans. Neural Networks 10 5 (1999) 1239-1243
17. Chen S.S., Donoho D.L., and Saunders M.A. Atomic decomposition by basis pursuit. SIAM Rev. 43 1 (2001) 129-159
18. A. Choudhury, Fast machine learning algorithms for large data, Ph.D. Thesis, Computational Engineering and Design Center, School of Engineering Sciences, University of Southampton, UK, 2002.
19. Cristianini N., and Shawe-Taylor J. An Introduction to Support Vector Machines: And Other Kernel-Based Learning Methods (2000), Cambridge University Press, Cambridge, UK
20. Duan K., Keerthi S.S., and Poo A.N. Evaluation of simple performance measures for tuning SVM hyperparameters. Neurocomputing 51 (2003) 41-59
21. Duda R.O., and Hart P.E. Pattern Classification and Scene Analysis (1973), Wiley, New York
22. Girolami M., and He C. Probability density estimation from optimally condensed data samples. IEEE Trans. Pattern Anal. Mach. Intell. 25 10 (2003) 1253-1264
23. S. Gunn, Support vector machines for classification and regression, Technical Report, ISIS Research Group, School of Electronics and Computer Science, University of Southampton, UK, 1998.
24. Hansen L.K., and Larsen J. Linear unlearning for cross-validation. Adv. Comput. Math. 5 (1996) 269-280
25. Hong X., Chen S., and Harris C.J. Fast kernel classifier construction using orthogonal forward selection to minimise leave-one-out misclassification rate. Proceedings of the 2nd International Conference on Intelligent Computing. Kunming, China (August 16-19, 2006) 106-114
26. Hong X., Chen S., and Harris C.J. A fast linear-in-the-parameters classifier construction algorithm using orthogonal forward selection to minimize leave-one-out misclassification rate. Int. J. Syst. Sci. 39 2 (2008) 119-125
27. Hong X., Chen S., and Harris C.J. A forward-constrained regression algorithm for sparse kernel density estimation. IEEE Trans. Neural Networks 19 1 (2008) 193-198
28. Hong X., Mitchell R.J., Chen S., Harris C.J., Li K., and Irwin G.W. Model selection approaches for non-linear system identification: a review. Int. J. Syst. Sci. 39 10 (2008) 925-946
29. Hong X., Sharkey P.M., and Warwick K. Automatic nonlinear predictive model construction algorithm using forward regression and the PRESS statistic. IEE Proc. Control Theory Appl. 150 3 (2003) 245-254
30. 〈http://www.ics.uci.edu/∼mlearn/MLRepository.html〉
31. 〈http://ida.first.fhg.de/projects/bench/benchmarks.htm〉
32. Lanckriet G.R.G., Cristianini N., Bartlett P., Ghaoui L.E., and Jordan M.I. Learning the kernel matrix with semidefinite programming. J. Mach. Learn. Res. 5 (2004) 27-72
33. K. Li, J. Peng, E.-W. Bai, Two-stage mixed discrete-continuous identification of radial basis function (RBF) neural models for nonlinear systems, IEEE Trans. Circuits Syst. (2008), to appear.
34. MacKay D.J.C. Bayesian interpretation. Neural Comput. 4 3 (1992) 415-447
35. McLachlan G., and Peel D. Finite Mixture Models (2000), Wiley, New York
36. Monari G., and Dreyfus G. Withdrawing an example from the training set: an analytic estimation of its effect on a non-linear parameterised model. Neurocomputing 35 (2000) 195-201
37. Monari G., and Dreyfus G. Local overfitting control via leverages. Neural Comput. 14 (2002) 1481-1506
38. S. Mukherjee, V. Vapnik, Support vector method for multivariate density estimation, Technical Report, A.I. Memo No. 1653, MIT AI Lab, 1999.
39. Myers R.H. Classical and Modern Regression with Applications. second ed. (1990), PWS Pub. Co., Boston, MA
40. Ong C.S., Smola A.J., and Williamson R.C. Hyperkernels. Neural Information Processing Systems vol. 15 (2002), MIT Press, Cambridge, MA
41. Parzen E. On estimation of a probability density function and mode. Ann. Math. Stat. 33 (1962) 1066-1076
42. Peng J., Li K., and Huang D.S. A hybrid forward algorithm for RBF neural network construction. IEEE Trans. Neural Networks 17 6 (2006) 1439-1451
43. Rätsch G., Onoda T., and Müller K.R. Soft margins for AdaBoost. Mach. Learn. 42 3 (2001) 287-320
44. Ripley B.D. Pattern Recognition and Neural Networks (1996), Cambridge University Press, Cambridge, UK
45. Schölkopf B., and Smola A.J. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond (2002), MIT Press, Cambridge, MA
46. Schölkopf B., Smola A.J., Williamson R.C., and Bartlett P.L. New support vector algorithms. Neural Comput. 12 5 (2000) 1207-1245
47. F. Sha, L.K. Saul, D.D. Lee, Multiplicative updates for nonnegative quadratic programming in support vector machines, Technical Report, MS-CIS-02-19, University of Pennsylvania, USA, 2002.
48. Silverman B.W. Density Estimation for Statistics and Data Analysis (1986), Chapman & Hall, London
49. Stone M. Cross validation choice and assessment of statistical predictions. J. R. Stat. Soc. Ser. B 36 (1974) 117-147
50. Suykens J.A.K., Van Gestel T., De Brabanter J., De Moor B., and Vandewalle J. Least Squares Support Vector Machines (2002), World Scientific Publishing Co., Singapore
51. Tipping M.E. Sparse Bayesian learning and the relevance vector machine. J. Mach. Learn. Res. 1 (2001) 211-244
52. Vapnik V. The Nature of Statistical Learning Theory (1995), Springer, New York
53. Vapnik V., and Mukherjee S. Support vector method for multivariate density estimation. In: Solla S., Leen T., and Müller K.R. (Eds). Advances in Neural Information Processing Systems (2000), MIT Press, Cambridge, MA 659-665
54. Vincent P., and Bengio Y. Kernel matching pursuit. Mach. Learn. 48 1 (2002) 165-187
55. Weston J., Gammerman A., Stitson M.O., Vapnik V., Vovk V., and Watkins C. Support vector density estimation. In: Schölkopf B., Burges C., and Smola A.J. (Eds). Advances in Kernel Methods-Support Vector Learning (1999), MIT Press, Cambridge, MA 293-306