Regression based D-optimality experimental design for sparse kernel density estimation

Neurocomputing

Volumn 73, Issue 4-6, 2010, Pages 727-739

  Chen, S ^a   Hong, X ^b   Harris, C J ^a  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^b UNIVERSITY OF READING   (United Kingdom)

Author keywords

D optimality; Optimal experimental design; Orthogonal forward regression; Parzen window estimate; Probability density function; Sparse kernel modelling

Indexed keywords

D-OPTIMALITY; FORWARD REGRESSION; OPTIMAL EXPERIMENTAL DESIGNS; PARZEN WINDOW ESTIMATE; SPARSE KERNELS;

ALGORITHMS; DESIGN; EIGENVALUES AND EIGENFUNCTIONS; NUMERICAL METHODS; OPTIMIZATION; ORTHOGONAL FUNCTIONS; QUADRATIC PROGRAMMING; REGRESSION ANALYSIS; STATISTICS; WINDOWS;

PROBABILITY DENSITY FUNCTION;

ACCURACY; ANALYTIC METHOD; ARTICLE; CALCULATION; COMPUTER LANGUAGE; CONTROLLED STUDY; DENSITY FUNCTIONAL THEORY; EXPERIMENTAL DESIGN; INTERMETHOD COMPARISON; KERNEL METHOD; MATHEMATICAL COMPUTING; PERFORMANCE; PRIORITY JOURNAL; PROCESS DEVELOPMENT; PROCESS MODEL; PROCESS OPTIMIZATION; QUANTITATIVE ANALYSIS; REGRESSION ANALYSIS; STATISTICAL MODEL;

EID: 75749084353     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neucom.2009.11.002     Document Type: Article

References (48)

1. Duda R.O., and Hart P.E. Pattern Classification and Scene Analysis (1973), Wiley, New York
2. Bishop C.M. Neural Networks for Pattern Recognition (1995), Oxford University Press, Oxford, UK
3. Silverman B.W. Density Estimation (1996), Chapman & Hall, London
4. Bowman A.W., and Azzalini A. Applied Smoothing Techniques for Data Analysis (1997), Oxford University Press, Oxford, UK
5. Wang H. Robust control of the output probability density functions for multivariable stochastic systems with guaranteed stability. IEEE Trans. Autom. Control 44 (1998) 2103-2107
6. Chen S., Samingan A.K., Mulgrew B., and Hanzo L. Adaptive minimum-BER linear multiuser detection for DS-CDMA signals in multipath channels. IEEE Trans. Signal Process. 49 (2001) 1240-1247
7. McLachlan G., and Peel D. Finite Mixture Models (2000), Wiley, New York
8. Dempster A.P., Laird N.M., and Rubin D.B. Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. B 39 (1977) 1-38
9. J.A. Bilmes, A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models, Technical Report ICSI-TR-97-021, University of Berkeley, USA, 1997.
10. Efron B., and Tibshirani R.J. An Introduction to Bootstrap (1993), Chapman & Hall, London
11. Yang Z.R., and Chen S. Robust maximum likelihood training of heteroscedastic probabilistic neural networks. Neural Networks 11 (1998) 739-747
12. Svensén M., and Bishop C.M. Robust Bayesian mixture modelling. Neurocomputing 64 (2005) 235-252
13. Archambeau C., and Verleysen M. Robust Bayesian clustering. Neural Networks 20 (2007) 129-138
14. Parzen E. On estimation of a probability density function and mode. Ann. Math. Stat. 33 (1962) 1066-1076
15. Weston J., Gammerman A., Stitson M.O., Vapnik V., Vovk V., and Watkins C. Support vector density estimation. In: Schölkopf B., Burges C., and Smola A.J. (Eds). Advances in Kernel Methods-Support Vector Learning (1999), MIT Press, Cambridge, MA 293-306
16. S. Mukherjee, V. Vapnik, Support vector method for multivariate density estimation, Technical Report A.I. Memo No. 1653, MIT AI Lab, USA, 1999.
17. Vapnik V., and Mukherjee S. Support vector method for multivariate density estimation. In: Solla S., Leen T., and Müller K.R. (Eds). Advances in Neural Information Processing Systems (2000), MIT Press, Cambridge, MA 659-665
18. L. Song, X. Zhang, A. Smola, A. Gretton, B. Schöolkopf, Tailoring density estimation via reproducing kernel moment matching, in: Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland, July 5-9, 2008, pp. 992-999.
19. Girolami M., and He C. Probability density estimation from optimally condensed data samples. IEEE Trans. Pattern Anal. Mach. Intell. 25 (2003) 1253-1264
20. A. Choudhury, Fast machine learning algorithms for large data, Ph.D. Thesis, Computational Engineering and Design Center, School of Engineering Sciences, University of Southampton, Southampton, UK, August 2002.
21. Chen S., Hong X., and Harris C.J. Sparse kernel regression modeling using combined locally regularized orthogonal least squares and D-optimality experimental design. IEEE Trans. Autom. Control 48 (2003) 1029-1036
22. Chen S., Hong X., Harris C.J., and Sharkey P.M. Sparse modeling using orthogonal forward regression with PRESS statistic and regularization. IEEE Trans. Syst. Man Cybern. Part B 34 (2004) 898-911
23. Chen S., Hong X., and Harris C.J. Sparse kernel density construction using orthogonal forward regression with leave-one-out test score and local regularization. IEEE Trans. Syst. Man Cybern. Part B 34 (2004) 1708-1717
24. Chen S., Hong X., and Harris C.J. An orthogonal forward regression techniques for sparse kernel density estimation. Neurocomputing 71 (2008) 931-943
25. F. Sha, L.K. Saul, D.D. Lee, Multiplicative updates for nonnegative quadratic programming in support vector machines, Technical Report MS-CIS-02-19, University of Pennsylvania, USA, 2002.
26. Hong X., Chen S., and Harris C.J. A forward-constrained regression algorithm for sparse kernel density estimation. IEEE Trans. Neural Networks 19 (2008) 193-1981
27. Atkinson A.C., and Donev A.N. Optimum Experimental Designs (1992), Clarendon Press, Oxford, UK
28. Hong X., and Harris C.J. Neurofuzzy design and model construction of nonlinear dynamical processes from data. IEE Proc. Control Theory Appl. 148 (2001) 530-538
29. Hong X., and Harris C.J. Nonlinear model structure detection using optimum design and orthogonal least squares. IEEE Trans. Neural Networks 12 (2001) 435-439
30. Hong X., and Harris C.J. Nonlinear model structure design and construction using orthogonal least squares and D-optimality design. IEEE Trans. Neural Networks 13 (2002) 1245-1250
31. Hong X., Harris C.J., Chen S., and Sharkey P.M. Robust nonlinear model identification methods using forward regression. IEEE Trans. Syst. Man Cybern. Part A 33 (2003) 514-523
32. Stone M. Cross validation choice and assessment of statistical predictions. J. R. Stat. Soc. Ser. B 36 (1974) 111-147
33. Myers R.H. Classical and Modern Regression with Applications. second ed. (1990), PWS-KEN, Boston
34. Glivenko-Cantelli theorem. [Online] Available 〈http://en.wikipedia.org/wiki/Glivenko-Cantelli_theorem〉.
35. Schölkopf B., Platt J.C., Shawe-Taylor J., Smola A.J., and Williamson R.C. Estimating the support of a high-dimensional distribution. Neural Comput. 13 (2001) 1443-1471
36. Chen S., and Wigger J. Fast orthogonal least squares algorithm for efficient subset model selection. IEEE Trans. Signal Process. 43 (1995) 1713-1715
37. Ripley B.D. Pattern Recognition and Neural Networks (1996), Cambridge University Press, Cambridge, UK
38. [Online] Available 〈http://www.stats.ox.ac.uk/PRNN〉.
39. [Online] Available 〈http://ida.first.fhg.de/projects/bench/benchmarks.htm〉.
40. Rätsch G., Onoda T., and Müller K.R. Soft margins for AdaBoost. Mach. Learn. 42 (2001) 287-320
41. S. Chen, X. Hong, C.J. Harris, Orthogonal forward selection for constructing the radial basis function network with tunable nodes, in: Proceedings of the 2005 International Conference Intelligent Computing, Hefei, China, August 23-26, 2005, pp. 777-786.
42. S. Chen, X. Hong, C.J. Harris, Construction of RBF classifiers with tunable units using orthogonal forward selection based on leave-one-out misclassification rate, in: Proceedings of the 2006 International Joint Conference on Neural Networks, Vancouver, Canada, July 16-21, 2006, pp. 6390-6394.
43. Wang X.X., Chen S., Lowe D., and Harris C.J. Parsimonious least squares support vector regression using orthogonal forward selection with the generalised kernel mode. Int. J. Modelling, Identification Control 1 (2006) 245-256
44. Wang X.X., Chen S., Lowe D., and Harris C.J. Sparse support vector regression based on orthogonal forward selection for the generalised kernel model. Neurocomputing 70 (2006) 462-474
45. Peng J.-X., Li G., and Irwin G.W. A new Jacobian matrix for optimal learning of single-layer neural networks. IEEE Trans. Neural Networks 19 (2008) 119-129
46. Chen S., Hong X., Luk B.L., and Harris C.J. Construction of tunable radial basis function networks using orthogonal forward selection. IEEE Trans. Syst. Man Cybern. Part B 39 (2009) 457-466
47. Li K., Peng J.-X., and Bai E.-W. Two-stage mixed discretecontinuous identification of radial basis function (RBF) neural models for nonlinear systems. IEEE Trans. Circuits Syst. Part I 56 (2009) 630-643
48. S. Chen, X. Hong, C.J. Harris, Probability density estimation with tunable kernels using orthogonal forward regression, IEEE Trans. Syst. Man Cybern. Part B (2010), to appear.