An information theoretic approach of designing sparse kernel adaptive filters

IEEE Transactions on Neural Networks

Volumn 20, Issue 12, 2009, Pages 1950-1961

  Liu, Weifeng ^a   Park, Il ^b   Príncipe, José C ^b  

^a AMAZON   (United States)

^b University of Florida   (United States)

Author keywords

Information measure; Kernel adaptive filters; Online Gaussian processes; Online kernel learning; Sparsification; Surprise

Indexed keywords

GAUSSIAN PROCESSES; INFORMATION MEASURE; INFORMATION MEASURES; KERNEL LEARNING; SPARSIFICATION;

ADAPTIVE FILTERS; ELECTRIC FILTERS; GAUSSIAN DISTRIBUTION; GAUSSIAN NOISE (ELECTRONIC); INFORMATION THEORY;

ADAPTIVE FILTERING;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; HUMAN; INFORMATION SCIENCE; NONLINEAR SYSTEM; REGRESSION ANALYSIS; TIME;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; HUMANS; INFORMATION THEORY; LEAST-SQUARES ANALYSIS; NONLINEAR DYNAMICS; TIME FACTORS;

EID: 72149098116     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2009.2033676     Document Type: Article

References (41)

1. J. Quinonero-Candela and C. E. Rasmussen, "A unifying viewof sparse approximate Gaussian process regression," J. Mach. Learn. Res., vol.6, pp. 1939-1959, 2005.
2. C. K. I.Williams and M. Seeger, "Using the Nyström method to speed up kernel machines," in Advances in Neural Information Processing Systems, T. K. Leen, T. G. Dietterich, and V. Tresp, Eds. Cambridge, MA: MIT Press, 2001, vol.13, pp. 682-688.
3. S. Fine and K. Scheinberg, "Efficient SVM training using low-rank kernel representations," J. Mach. Learn. Res., vol.2, pp. 242-264, 2001.
4. D. MacKay, "Information-based objective functions for active data selection," Neural Comput., vol.4, no.4, pp. 590-604, 1992.
5. A. J. Smola and P. L. Bartlett, "Sparse greedy Gaussian process regression," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2001, vol.13, pp. 619-625.
6. M. Seeger and C.Williams, "Fast forward selection to speed up sparse Gaussian process regression," in Proc. Workshop Artif. Intell. Statist., 2003, pp. 205-212.
7. L. Hoegaerts, J. A. K. Suykens, J. Vandewalle, and B. De Moor, "Subset based least squares subspace regression in RKHS," Neurocomputing, vol.63, pp. 293-323, 2005.
8. G. Cauwenberghs and T. Poggio, "Incremental and decremental support vector machine learning," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2000, pp. 409-415.
9. J. Quinonero-Candela and O. Winther, "Incremental Gaussian processes," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2003, vol.15, pp. 1001-1008.
10. K. Crammer, J. Kandola, and Y. Singer, "Online classification on a budget," in Advances in Neural Information Processing Systems, S. Thrun, L. Saul, and B. Schölkopf, Eds. Cambridge, MA: MIT Press, 2004, vol.16, pp. 225-232.
11. J. Kivinen, A. Smola, and R. C. Williamson, "Online learning with kernels," IEEE Trans. Signal Process., vol.52, no.8, pp. 2165-2176, Aug. 2004.
12. W. Liu, P. Pokharel, and J. Príncipe, "The kernel least mean square algorithm," IEEE Trans. Signal Process., vol.56, no.2, pp. 543-554, Feb. 2008.
13. Y. Engel, S. Mannor, and R. Meir, "The kernel recursive least-squares algorithm," IEEE Trans. Signal Process., vol.52, no.8, pp. 2275-2285, Aug. 2004.
14. W. Liu and J. C. Príncipe, "The kernel affine projection algorithms," EURASIP J. Adv. Signal Process. 2008 [Online]. Available: http://www.hindawi.com/GetArticle.aspx?doi=10.1155/2008/784292
15. K. Slavakis and S. Theodoridis, "Sliding window generalized kernel affine projection algorithm using projection mappings," EURASIP J. Adv. Signal Process. 2008 [Online]. Available: http://www.hindawi. com/GetArticle.aspx?doi=10.1155/2008/735351
16. C. Richard, J. C. M. Bermudez, and P. Honeine, "Online prediction of time series data with kernels," IEEE Trans. Signal Process., vol.57, no.3, pp. 1058-1066, Mar. 2009.
17. W. Liu, I. Park, Y.Wang, and J. C. Príncipe, "Extended kernel recursive least squares algorithm," IEEE Trans. Signal Process., vol.57, no.10, pp. 3801-3814, Oct. 2009.
18. S. Haykin, Adaptive Filter Theory, 4th ed. Englewood Cliffs, NJ: Prentice-Hall, 2002.
19. J. Platt, "Aresource-allocating network for function interpolation," Neural Comput., vol.3, no.2, pp. 213-225, 1991.
20. L. Csato and M. Opper, "Sparse online Gaussian processes," Neural Comput., vol.14, pp. 641-668, 2002.
21. G. Palm, "Evidence, information, and surprise," Biol. Cybern., vol.42, no.1, pp. 57-68, 1981.
22. E. Pfaffelhuber, "Learning and information theory," Int. J. Neurosci., vol.3, no.2, pp. 83-88, 1972.
23. L. Itti and P. Baldi, "Bayesian surprise attracts human attention," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2006, vol.19, pp. 1-8.
24. N. Ranasinghe and W. Shen, "Surprise-based learning for developmental robotics," in Proc. ECSIS Symp. Learn. Adapt. Behav. Robot. Syst., 2008, pp. 65-70.
25. P. A. N. Bosman and D. Thierens, "Negative log-likelihood and statistical hypothesis testing as the basis of model selection in IDEAs," in Proc. Genetic Evol. Comput. Conf. (Late Breaking Papers), D.Whitley, Ed., Las Vegas, NV, 2000, pp. 51-58.
26. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. New York: Wiley-Interscience, 2000.
27. C. Dima and M. Hebert, "Active learning for outdoor obstacle detection," in Proc. Sci. Syst. I, Aug. 2005.
28. N. Aronszajn, "Theory of reproducing kernels," Trans. Amer. Math. Soc., vol.68, pp. 337-404, 1950.
29. C. J. C. Burges, "A tutorial on support vector machines for pattern recognition," Data Mining Knowl. Disc., vol.2, no.2, pp. 121-167, 1998.
30. B. Widrow and M. E. Hoff, "Adaptive switching circuits," IRE WESCON Convention Record, vol.4, pp. 96-104, 1960.
31. C. Saunders, A. Gammerman, and V. Vovk, "Ridge regression learning algorithm in dual variables," in Proc. 15th Int. Conf. Mach. Learn., 1998, pp. 515-521.
32. J. A. K. Suykens, T. V. Gestel, J. Brabanter, B. D. Moor, and J. Vandewalle, Least Squares Support Vector Machines. Singapore: World Scientific, 2002.
33. B. Schölkopf, B. Mika, C. J. C. Burges, P. Knirsch, K. Müller, G. Rätsch, and A. Smola, "Input space vs. feature space in kernel-based methods," IEEE Trans. Neural Netw., vol.10, no.5, pp. 1000-1017, Sep. 1999.
34. C. E. Shannon, "A mathematical theory of communication," in Bell Syst. Tech. J., Jul. 1948, pp. 379-423.
35. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley-Interscience, 1991.
36. T. Poggio and S. Smale, "The mathematics of learning: Dealing with data," Trans. Amer. Math. Soc., vol.50, pp. 537-544, Nov. 2003.
37. C. E. Rasmussen and C. Williams, Gaussian Processes for Machine Learning. Cambridge, MA: MIT Press, 2006.
38. L. Glass and M. Mackey, From Clocks to Chaos: The Rhythms of Life. Princeton, NJ: Princeton Univ. Press, 1988.
39. S. Mukherjee, E. Osuna, and F. Girosi, "Nonlinear prediction of chaotic time series using support vector machines," in Proc. IEEE Workshop Neural Netw. Signal Process. VII, J. Príncipe, L. Giles, N. Morgan, and E. Wilson, Eds., 1997, pp. 511-519.
40. F. Takens, "Detecting strange attractors in turbulence," Dyn. Syst. Turbulence, pp. 366-381, 1981.
41. P. Tans, "Trends in atmospheric carbon dioxide-Mauna loa," NOAA/ ESRL, 2008 [Online]. Available: www.esrl.noaa.gov/gmd/ccgg/trends