Probability density estimation with tunable kernels using orthogonal forward regression

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 40, Issue 4, 2010, Pages 1101-1114

  Chen, Sheng ^a   Hong, Xia ^b   Harris, Chris J ^a  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^b UNIVERSITY OF READING   (United Kingdom)

Author keywords

Leave one out (LOO) cross validation; multiplicative nonnegative quadratic programming (MNQP); orthogonal forward regression (OFR); Parzen window (PW) estimate; probability density function (pdf); sparse kernel density (KD) estimate; tunable kernels

Indexed keywords

CROSS VALIDATION; FORWARD REGRESSION; LEAVE-ONE-OUT; MULTIPLICATIVE NONNEGATIVE QUADRATIC PROGRAMMING (MNQP); PARZEN WINDOWS; PROBABILITY DENSITY FUNCTION (PDF); SPARSE KERNELS;

COVARIANCE MATRIX; ESTIMATION; OPTIMIZATION; ORTHOGONAL FUNCTIONS; PROBABILITY DISTRIBUTIONS; QUADRATIC PROGRAMMING; REGRESSION ANALYSIS;

PROBABILITY DENSITY FUNCTION;

ARTICLE; COMPUTER SIMULATION; REGRESSION ANALYSIS; SAMPLE SIZE; STATISTICAL ANALYSIS; STATISTICAL DISTRIBUTION; STATISTICAL MODEL;

COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; MODELS, STATISTICAL; REGRESSION ANALYSIS; SAMPLE SIZE; STATISTICAL DISTRIBUTIONS;

EID: 77954759393     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2009.2034732     Document Type: Article

References (43)

1. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973.
2. C. M. Bishop, Neural Networks for Pattern Recognition. Oxford, U.K.: Oxford Univ. Press, 1995.
3. B. W. Silverman, Density Estimation. London, U.K.: Chapman & Hall, 1996.
4. H. Wang, "Robust control of the output probability density functions for multivariable stochastic systems with guaranteed stability," IEEE Trans. Autom. Control, vol.44, no.11, pp. 2103-2107, Nov. 1999.
5. S. Chen, A. K. Samingan, B. Mulgrew, and L. Hanzo, "Adaptive minimum-BER linear multiuser detection for DS-CDMA signals in multipath channels," IEEE Trans. Signal Process., vol.49, no.6, pp. 1240- 1247, Jun. 2001. (Pubitemid 32512284)
6. G. McLachlan and D. Peel, Finite Mixture Models. New York: Wiley, 2000.
7. A. P. Dempster, N.M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Stat. Soc. B, vol.39, no.1, pp. 1-38, 1977.
8. J. A. Bilmes, "A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models," Univ. California, Berkeley, Tech. Rep., ICSI-TR-97-1021, 1997.
9. B. Efron and R. J. Tibshirani, An Introduction to Bootstrap. London, U.K.: Chapman & Hall, 1993.
10. Z. R. Yang and S. Chen, "Robust maximum likelihood training of heteroscedastic probabilistic neural networks," Neural Netw., vol.11, no.4, pp. 739-747, Jun. 1998.
11. M. Svensén and C. M. Bishop, "Robust Bayesian mixture modelling," Neurocomputing, vol.64, pp. 235-252, Mar. 2005.
12. C. Archambeau and M. Verleysen, "Robust Bayesian clustering," Neural Netw., vol.20, no.1, pp. 129-138, Jan. 2007.
13. E. Parzen, "On estimation of a probability density function and mode," Ann. Math. Stat., vol.33, no.3, pp. 1066-1076, 1962.
14. J. Weston, A. Gammerman, M. O. Stitson, V. Vapnik, V. Vovk, and C. Watkins, "Support vector density estimation," in Advances in Kernel Methods-Support Vector Learning, B. Schölkopf, C. Burges, and A. J. Smola, Eds. Cambridge, MA: MIT Press, 1999, pp. 293-306.
15. S. Mukherjee and V. Vapnik, "Support vector method for multivariate density estimation," MIT AI Lab, Cambridge, MA, Tech. Rep., A.I. Memo No. 1653, 1999.
16. V. Vapnik and S. Mukherjee, "Support vector method for multivariate density estimation," in Advances in Neural Information Processing Systems, S. Solla, T. Leen, and K. R. Müller, Eds. Cambridge, MA: MIT Press, 2000, pp. 659-665.
17. M. Girolami and C. He, "Probability density estimation from optimally condensed data samples," IEEE Trans. Pattern Anal.Mach. Intell., vol.25, no.10, pp. 1253-1264, Oct. 2003.
18. A. Choudhury, "Fast machine learning algorithms for large data," Ph.D. dissertation, Comput. Eng. Design Center, School Eng. Sci., Univ. Southampton, Southampton, U.K., 2002.
19. S. Chen, X. Hong, and C. J. Harris, "Sparse kernel density construction using orthogonal forward regression with leave-one-out test score and local regularization," IEEE Trans. Syst.,Man, Cybern. B, Cybern., vol.34, no.4, pp. 1708-1717, Aug. 2004.
20. S. Chen, X. Hong, and C. J. Harris, "An orthogonal forward regression techniques for sparse kernel density estimation," Neurocomputing, vol.71, no.4-6, pp. 931-943, Jan. 2008. (Pubitemid 351168423)
21. F. Sha, L. K. Saul, and D. D. Lee, "Multiplicative updates for nonnegative quadratic programming in support vector machines," Univ. Pennsylvania, Philadelphia, PA, Tech. Rep. MS-CIS-02-19, 2002.
22. D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning. Reading, MA: Addison-Wesley, 1989.
23. K. F. Man, K. S. Tang, and S. Kwong, Genetic Algorithms: Concepts and Design. London, U.K.: Springer-Verlag, 1998.
24. L. Ingber, "Simulated annealing: Practice versus theory," Math. Comput. Model., vol.18, no.11, pp. 29-57, 1993.
25. S. Chen and B. L. Luk, "Adaptive simulated annealing for optimization in signal processing applications," Signal Process., vol.79, no.1, pp. 117- 128, Nov. 1999.
26. J. Kennedy and R. C. Eberhart, "Particle swarm optimization," in Proc. IEEE Int. Conf. Neural Netw., Piscataway, NJ, Nov. 27-Dec. 1, 1995, vol.4, pp. 1942-1948.
27. J. Kennedy, R. C. Eberhart, and Y.-H. Shi, Swarm Intelligence. San Mateo, CA: Morgan Kaufmann, 2001.
28. S. Chen, X. X. Wang, and C. J. Harris, "Experiments with repeating weighted boosting search for optimization in signal processing applications," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol.35, no.4, pp. 682-693, Aug. 2005.
29. X. Hong, P. M. Sharkey, and K.Warwick, "Automatic nonlinear predictive model construction algorithm using forward regression and the PRESS statistic," Proc. Inst. Elect. Eng.-Control Theory Appl., vol.150, no.3, pp. 245-254, May 2003.
30. S. Chen, X. Hong, C. J. Harris, and P.M. Sharkey, "Sparse modeling using orthogonal forward regression with PRESS statistic and regularization," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol.34, no.2, pp. 898-911, Apr. 2004.
31. B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, "Estimating the support of a high-dimensional distribution," Neural Comput., vol.13, no.7, pp. 1443-1471, Jul. 2001.
32. [Online]. Available: http://en.wikipedia.org/wiki/Glivenko-Cantelli- theorem
33. R. H. Myers, Classical and Modern Regression With Applications, 2nd ed. Boston, MA: PWS-KENT, 1990.
34. M. Stone, "Cross validation choice and assessment of statistical predictions," J. R. Stat. Soc. B, vol.36, pp. 111-147, 1974.
35. L. K. Hansen and J. Larsen, "Linear unlearning for cross-validation," Adv. Comput. Math., vol.5, no.1, pp. 269-280, Dec. 1996.
36. G. Monari and G. Dreyfus, "Local overfitting control via leverages," Neural Comput., vol.14, no.6, pp. 1481-1506, Jun. 2002.
37. S. Chen and J. Wigger, "Fast orthogonal least squares algorithm for efficient subset model selection," IEEE Trans. Signal Process., vol.43, no.7, pp. 1713-1715, Jul. 1995.
38. S. Chen, S. A. Billings, and W. Luo, "Orthogonal least squares methods and their applications to non-linear system identification," Int. J. Control, vol.50, no.5, pp. 1873-1896, Nov. 1989.
39. R. E. Schapire, "The strength of weak learnability," Mach. Learn., vol.5, no.2, pp. 197-227, Jun. 1990.
40. Y. Freund and R. E. Schapire, "A decision-theoretic generalization of online learning and an application to boosting," J. Comput. Syst. Sci., vol.55, no.1, pp. 119-139, Aug. 1997.
41. L. Breiman, "Prediction games and arcing algorithms," Neural Comput., vol.11, no.7, pp. 1493-1518, Oct. 1999.
42. R. Meir and G. Rätsch, "An introduction to boosting and leveraging," in Advanced Lectures in Machine Learning, S. Mendelson and A. Smola, Eds. New York: Springer-Verlag, 2003, pp. 119-184.
43. F. Schoen, "Stochastic techniques for global optimization: A survey of recent advances," J. Global Optim., vol.1, no.3, pp. 207-228, Sep. 1991.