On accuracy of PDF divergence estimators and their applicability to representative data sampling

Entropy

Volumn 13, Issue 7, 2011, Pages 1229-1266

  Budka, Marcin ^a   Gabrys, Bogdan ^a   Musial, Katarzyna ^a  

^a BOURNEMOUTH UNIVERSITY   (United Kingdom)

Author keywords

Cross validation; Divergence estimation; Generalisation error estimation; Kullback Leibler divergence; Sampling

Indexed keywords

EID: 80051601039     PISSN: None     EISSN: 10994300     Source Type: Journal    
DOI: 10.3390/e13071229     Document Type: Article

References (49)

1. Budka, M.; Gabrys, B. Correntropy-based density-preserving data sampling as an alternative to standard cross-validation. In Proceedings of the International Joint Conference on Neural Networks, IJCNN 2010, part of the IEEE World Congress on Computational Intelligence, WCCI 2010, Barcelona, Spain, 18-23 July 2010; pp. 1437-1444.
2. Budka, M.; Gabrys, B. Density Preserving Sampling (DPS) for error estimation and model selection. IEEE Trans. Pattern Anal. Mach. Intell. submitted for publication, 2011.
3. Kohavi, R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, Montreal, Canada, 20-25 August 1995; Morgan Kaufmann: San Francisco, CA, USA, 1995; Volume 2, pp. 1137-1145.
4. Liu, W.; Pokharel, P.; Principe, J. Correntropy: A Localized Similarity Measure. In Proceedings of the International Joint Conference on Neural Networks, Vancouver, Canada, 16-21 July 2006; pp. 4919-4924.
5. Parzen, E. On estimation of a probability density function and mode. Ann. Math. Stat. 1962, 33, 1065-1076.
6. Duda, R.; Hart, P.; Stork, D. Pattern Classification, 2nd ed.; John Wiley & Sons: New York, NY, USA, 2001.
7. Akaike, H. A new look at the statistical model identification. IEEE Trans. Automat. Contr. 1974, 19, 716-723.
8. Seghouane, A.; Bekara, M. A small sample model selection criterion based on Kullback's symmetric divergence. IEEE Trans. Signal Process. 2004, 52, 3314-3323.
9. Stone, M. An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion. J. Roy. Stat. Soc. B 1977, 39, 44-47.
10. Nguyen, X.; Wainwright, M.; Jordan, M. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inform. Theor. 2010, 56, 5847-5861.
11. Jenssen, R.; Principe, J.; Erdogmus, D.; Eltoft, T. The Cauchy-Schwarz divergence and Parzen windowing: Connections to graph theory and Mercer kernels. J. Franklin Inst. 2006, 343, 614-629.
12. Turlach, B. Bandwidth selection in kernel density estimation: A review. CORE and Institut de Statistique 1993, 23-493.
13. Duin, R. On the choice of smoothing parameters for Parzen estimators of probability density functions. IEEE Trans. Comput. 1976, 100, 1175-1179.
14. Silverman, B. Density Estimation for Statistics and Data Analysis; Chapman & Hall/CRC Press: Boca Raton, FL, USA, 1998.
15. Sheather, S.J.; Jones,M.C. A Reliable Data-Based Bandwidth SelectionMethod for Kernel Density Estimation. J. Roy. Stat. Soc. B 1991, 53, 683-690.
16. Jones, M.C.; Marron, J.S.; Sheather, S.J. A Brief Survey of Bandwidth Selection for Density Estimation. J. Am. Stat. Assoc. 1996, 91, 401-407.
17. Raykar, V.C.; Duraiswami, R. Fast optimal bandwidth selection for kernel density estimation. In Proceedings of the 6th SIAM International Conference on Data Mining, Bethesda, Maryland, USA, 20-22 April 2006; Ghosh, J., Lambert, D., Skillicorn, D., Srivastava, J., Eds.; SIAM: Philadelphia, PA, USA, 2006; pp. 524-528.
18. Perez-Cruz, F. Kullback-Leibler divergence estimation of continuous distributions. In Proceedings of the IEEE International Symposium on Information Theory, Toronto, Canada, 6-11 July 2008; pp. 1666-1670.
19. Cichocki, A.; Amari, S. Families of Alpha-Beta-and Gamma-Divergences: Flexible and Robust Measures of Similarities. Entropy 2010, 12, 1532-1568.
20. Kullback, S. Information Theory and Statistics; Dover Publications Inc.: New York, NY, USA, 1997.
21. Kullback, S.; Leibler, R. On information and sufficiency. Ann. Math. Stat. 1951, 22, 79-86.
22. Le Cam, L.; Yang, G. Asymptotics in Statistics: Some Basic Concepts; Springer Verlag: New York, NY, USA, 2000.
23. Fukunaga, K.; Hayes, R. The reduced Parzen classifier. IEEE Trans. Pattern Anal. Mach. Intell. 1989, 11, 423-425.
24. Cardoso, J. Infomax and maximum likelihood for blind source separation. IEEE Signal Process. Lett. 1997, 4, 112-114.
25. Cardoso, J. Blind signal separation: statistical principles. Proc. IEEE 1998, 86, 2009-2025.
26. Ojala, T.; Pietik̈ainen, M.; Harwood, D. A comparative study of texture measures with classification based on featured distributions. Pattern Recogn. 1996, 29, 51-59.
27. Hastie, T.; Tibshirani, R. Classification by pairwise coupling. Ann. Stat. 1998, 26, 451-471.
28. Buccigrossi, R.; Simoncelli, E. Image compression via joint statistical characterization in the wavelet domain. IEEE Trans. Image Process. 1999, 8, 1688-1701.
29. Moreno, P.; Ho, P.; Vasconcelos, N. A Kullback-Leibler divergence based kernel for SVM classification inmultimedia applications. Adv. Neural Inform. Process. Syst. 2004, 16, 1385-1392.
30. MacKay, D. Information Theory, Inference, and Learning Algorithms; Cambridge University Press: Cambridge, UK, 2003.
31. Wang, Q.; Kulkarni, S.; Verdu, S. A nearest-neighbor approach to estimating divergence between continuous random vectors. In Proceedings of the IEEE International Symposium on Information Theory, Seattle, WA, USA, 9-14 July 2006; pp. 242-246.
32. Hershey, J.; Olsen, P. Approximating the Kullback-Leibler divergence between Gaussian mixture models. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Honolulu, Hawaii, 15-20 April 2007; Volume 4, pp. 317-320.
33. Seghouane, A.; Amari, S. The AIC criterion and symmetrizing the Kullback-Leibler divergence. IEEE Trans. Neural Network 2007, 18, 97-106.
34. Jeffreys, H. An invariant form for the prior probability in estimation problems. Proc. Roy. Soc. Lond. Math. Phys. Sci. A 1946, 186, pp. 453-461.
35. Lin, J. Divergence measures based on the Shannon entropy. IEEE Trans. Inform. Theor. 1991, 37, 145-151.
36. Dhillon, I.; Mallela, S.; Kumar, R. A divisive information theoretic feature clustering algorithm for text classification. J.Mach. Learn. Res. 2003, 3, 1265-1287.
37. Subramaniam, S.; Palpanas, T.; Papadopoulos, D.; Kalogeraki, V.; Gunopulos, D. Online outlier detection in sensor data using non-parametric models. In Proceedings of the 32nd international conference on Very large data bases, Seoul, Korea, 12-15 September 2006; VLDB Endowment: USA, 2006, pp. 187-198.
38. Rao, S.; Liu,W.; Principe, J.; de Medeiros Martins, A. Information theoretic mean shift algorithm. In Proceedings of the 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing, Arlington, VA, USA, 6-8 September 2006; pp. 155-160.
39. Principe, J.; Xu, D.; Fisher, J. Information theoretic learning. In Unsupervised Adaptive Filtering; Haykin, S., Ed.; John Wiley & Sons: Toronto, Canada, 2000; pp. 265-319.
40. Jenssen, R.; Erdogmus, D.; Principe, J.; Eltoft, T. The Laplacian spectral classifier. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, PA, USA, 18-23 March 2005; pp. 325-328.
41. Jenssen, R.; Erdogmus, D.; Hild, K.; Principe, J.; Eltoft, T. Optimizing the Cauchy-Schwarz PDF distance for information theoretic, non-parametric clustering. In Energy Minimization Methods in Computer Vision and Pattern Recognition; Rangarajan, A., Vemurl, B., Yuille, A., Eds.; Springer, Berlin, Germany; Lect. Notes Comput. Sci., 2005, 3257, 34-45.
42. Kapur, J. Measures of Information and Their Applications; John Wiley & Sons: New York, NY, USA, 1994.
43. Zhou, S.; Chellappa, R. Kullback-Leibler distance between two Gaussian densities in reproducing kernel Hilbert space. In Proceedings of the IEEE International Symposium on Information Theory, Chicago, IL, USA, 27 June-2 July 2004.
44. Kuncheva, L. Fuzzy Classifier Design; Physica Verlag: Heidelberg, Germany, 2000.
45. Ripley, B. Pattern Recognition and Neural Networks; Cambridge University Press: Cambridge, UK, 1996.
46. Ruta, D.; Gabrys, B. A framework for machine learning based on dynamic physical fields. Nat. Comput. 2009, 8, 219-237.
47. Minka, T. A family of algorithms for approximate Bayesian inference. PhD thesis, MIT, Cambridge, MA, USA, January 2001.
48. Paninski, L. Estimation of entropy and mutual information. Neural Comput. 2003, 15, 1191-1253.
49. Goldberger, J.; Gordon, S.; Greenspan, H. An efficient image similarity measure based on approximations of KL-divergence between two Gaussian mixtures. In Proceedings of the 9th IEEE International Conference on Computer Vision, Nice, France, 13-16 October 2003; Volume 1, pp. 487-493.