Bounds for the loss in probability of correct classification under model based approximation

Journal of Machine Learning Research

Volumn 7, Issue , 2006, Pages 2449-2480

  Ekdahl, Magnus ^a   Koski, Timo ^a  

^a LINKÖPING UNIVERSITY   (Sweden)

Author keywords

Bayesian networks; Kolmogorov distance of variation; Na ve bayes; Plug in classifier; Variational learning

Indexed keywords

APPROXIMATION THEORY; CLASSIFIERS; COMPUTER CONTROL; MATHEMATICAL MODELS; OPTIMAL CONTROL SYSTEMS; STATISTICAL METHODS;

BAYESIAN NETWORKS; KOLMOGOROV DISTANCE OF VARIATION; OPTIMAL CLASSIFIERS; PLUG-IN CLASSIFIER; VARIATIONAL LEARNING;

PATTERN RECOGNITION;

EID: 33845265891     PISSN: 15337928     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (62)

1. S.M. Ali and S.D. Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society. Series B, 28(1):131-142, 1966.
2. S. Anoulova, P. Fischer, S. Polt, and H.U. Simon. Probably almost Bayes decisions. Information and Computation, 129(1):63-71, 1996.
3. S. Arimoto. Information-theoretical considerations on estimation problems. Information and Control, 19:181-194, 1971.
4. R.R. Bahadur. On classification based on responses to n dichotomous items. In H. Solomon, editor, Studies in Item Analysis and Prediction, pages 169-177, Stanford, California, 1961a. Stanford University Press.
5. R.R. Bahadur. A representation of the joint distribution of responses to n dichotomous items. In H. Solomon, editor, Studies in Item Analysis and Prediction, pages 158-168, Stanford, California, 1961b. Stanford University Press.
6. L.C. Barbosa. Maximum likelihood sequence estimators: A geometric view. IEEE Transactions on Information Theory, 35(2), 1989.
7. B.K. Bhattacharyya and G.T. Toussaint. An upper bound on the probability of misclassification in terms of Matusita's measure of affinity. Annals of the Institute of Statistical Mathematics, 34: 161-165, 1982.
8. C. Boutilier, N. Friedman, M. Goldszmidt, and D. Koller. Context-specific independence in Bayesian networks. In Uncertainty in Artificial Intelligence (UAI-96), pages 115-123, 1996.
9. D.T. Brown. A note on approximations to discrete probability distributions. Information and Control, 2:386-392, 1959.
10. H.D. Brunk and D.A. Pierce. Estimation of discrete multivariate densities for computer-aided differential diagnosis. Biometrika, 61(3):493-499, 1974.
11. D.M. Chickering. Optimal structure identification with greedy search. Journal of Machine Learning Research, 3(Nov):507-554, 2002.
12. C.K. Chow and C.N. Liu. Approximating discrete probability distributions with dependency trees. IEEE Transactions on Information Theory, 14(3):462-467, 1968.
13. G. Cooper. The computational complexity of probabilistic inference using bayesian Belief networks. Artificial Intelligence, 42(2-3):393-405, 1990.
14. T.M. Cover and J.A Thomas. Elements of Information Theory, chapter 2.2, page 169. Wiley, 1991.
15. R.G. Cowell, A.P. Dawid, S.L. Lauritzen, and D.J. Spiegelhalter. Probabilistic Networks and Expert Systems. Springer, 1999.
16. H. Cramér. Mathematical Methods of Statistics, 11 th printing. Princeton University Press, 1966.
17. L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1996.
18. P. Domingos and M. Pazzani. On the optimality of the simple bayesian classifier under zero-one loss. Machine Learning, 29(2):103-130, 1997.
19. M. Ekdahl. Approximations of Bayes Classifiers for Statistical Learning of Clusters. Licentiate thesis, Linköpings Universitet, 2006.
20. M. Ekdahl and T. Koski. On the performance of model based approximations in classification. In Proceedings of The 23rd Annual Workshop of the Swedish Artificial Intelligence Society, pages 73-82. http://sais2006.cs.umu.se/, 2006.
21. J.H. Friedman. On bias, variance, 0/1-loss, and the curse-of- dimensionality. Data Mining and Knowledge Discovery, 1:55-77, 1997.
22. N. Friedman, D. Geiger, and M. Goldszmidt. Bayesian network classifiers. Machine Learning, 29 (2): 1-36, 1997.
23. N. Glick. Sample-based classification procedures derived from density estimators. Journal of the American Statistical Association, 67(337):116-122, 1972.
24. N. Glick. Sample based multinomial classification. Biometrics, 29(2):241-256, 1973.
25. L.A. Goodman and W.H. Kruskal. Measures of association for cross classifications. Journal of the American Statistical Association, 49(268):732-763, 1954.
26. M. Gyllenberg and T. Koski. Probabilistic models for bacterial taxonomy. International Statistical Review, 69:249-276, 2001.
27. D. Hand, H. Mannila, and P. Smyth. Principles of Data Mining. The MIT Press, 2001.
28. D.J. Hand and K. Yu. Idiot's Bayes-not so stupid after all? International Statistical Review, 69(3): 385-398, 2001.
29. W. Hoeffding. Stochastische Unabhängigkeit und funktionaler Zusammenhang. Skandinavisk Aktuarietidskrift, 25:200-227, 1942.
30. W. Hoeffding and J. Wolfowitz. Distinguishability of sets of distributions. The Annals of Mathematical Statistics, 29(3):700-718, 1958.
31. K. Huang, I. King, and M.R. Lyu. Finite mixture model of bounded semi-naive Bayesian network classifier. In Proceedings of the International Conference on Artificial Neural Networks (ICANN-2003), volume 2714 of Lecture Notes in Computer Science. Springer, 2003.
32. M.I. Jordan, Z. Ghahramani, T. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999.
33. T. Kailath. The divergence and Bhattacharyya distance measures in signal selection. IEEE Transactions on Communications Technology, 15(1):52-60, 1967.
34. G.K. Kaleh. Channel equalization for block transmission systems. IEEE Journal on Selected Areas in Communications, 13(1):150-121, 1995.
35. K.B. Korb and A.E. Nicholson. Bayesian Artificial Intelligence. Chapman and Hall, 2004.
36. H.H. Ku and S. Kuliback. Approximating discrete probability distributions. IEEE Transactions on Information Theory, 15:444-447, 1969.
37. S. Kullback. Information Theory and Statistics, chapter 9, page 190. Dover Publications Inc., 1997.
38. S.L. Lauritzen. Propagation of probabilities, means, and variances in mixed graphical association models. Journal of the American Statistical Association, 87:1098-1108, 1990.
39. P.M. Lewis. Approximating probability distributions to reduce storage requirements. Information and Control, 2:214-225, 1959.
40. D.K. McGraw and J.F. Wagner. Elliptically symmetric distributions. IEEE Transactions on Information Theory, 14(1):110-120, 1968.
41. D.H.II. Moore. Evaluation of five discrimination procedures for binary variables. Journal of the American Statistical Association, 68(342):399-404, 1973.
42. X. Nguyen, M.J. Wainwright, and M.I. Jordan. On divergences, surrogate loss functions, and decentralized detection. Technical Report 695, University of California, Berkeley, 2005.
43. J. Ott and R.A. Kronmal. Some classification procedures for multivariate binary data using orthogonal functions. Journal of the American Statistical Association, 71(354):391-399, 1976.
44. G.A. Pistone, E.A. Riccomagno, and H.P.A. Wynn. Gröbner bases and factorisation in discrete probability and Bayes. Statistics and Computing, 11(1):37-46, 2001.
45. E.J.G. Pitman. Some Basic Theory for Statistical Inference, chapter 2. Chapman and Hall, 1979.
46. B.D. Ripley. Pattern Recognition and Neural Networks. Cambridge University Press, 1996.
47. I. Rish, J. Hellerstein, and J. Thathachar. An analysis of data characteristics that affect Naive Bayes performance. Technical Report RC21993, IBM, 2001.
48. J. Van Ryzin. Bayes risk consistency of classification procedures using density estimation. Sankhya Series A, 28:261-270, 1966.
49. P. Scheinck. Symptom diagnosis Bayes's and Bahadur's distribution. International Journal of Biomedical Computing, 3:17-28, 1972.
50. M.J. Schervish. Theory of Statistics. Springer, second edition, 1995.
51. D. Slepian. On the symmetrized Kronecker power of a matrix and extensions of Mehler's formula for Hermite polynomials. SIAM Journal on Mathematical Analysis, 3:606-616, 1972.
52. H. Strasser. Mathematical Theory of Statistics, chapter 2.2. Walter de Gruyter, 1985.
53. J.L. Teugels. Some representations of the multivariate Bernoulli and binomial distributions. Journal of Multivariate Analysis, 33(1):256-268, 1990.
54. D.M. Titterington, G.D. Murray, L.S. Murray, D.J. Spiegelhalter, A.M. Skene, J.D.F. Habbema, and G.J. Gelpke. Comparison of discrimination techniques applied to a complex data set of head injured patients. Journal of the Royal Statistical Society., 144(2):145-175, 1981.
55. F. Topsoe. Some inequalities for information divergence and related measures of discrimination. IEEE Transactions on Information Theory, 46(4):1602-1609, 2000.
56. G.T. Toussaint. Polynomial representation of classifiers with independent discrete-valued features. IEEE Transactions on Computers, 21:205-208, 1972.
57. R. van Engelen. Approximating bayesian Belief networks by arc removal. IEEE Transactions on Pattern Analysis and Machine Intellignce, 19(8):916-920, 1997.
58. R. Vilata and I. Rish. A decomposition of classes via clustering to explain and improve Naive Bayes. In Machine Learning: ECML 2003: 14th European Conference on Machine Learning, pages 444-455, 2003.
59. T.R. Vilmansen. Feature evaluation with measures of probabilistic dependence. IEEE Transactions on Computers, 22(4):381-387, 1973.
60. T.R. Vilmansen. On dependence and discrimination in pattern recognition. IEEE Transactions on Computers, 21:1029-1031, 1971.
61. M.J Wainwright and M.I. Jordan. Graphical models, exponential families, and variational inference. Technical Report 649, UC Berkeley, Dept. of Statistics, 2003.
62. J. Williamson. Bayesian Nets and Causality: Philosophical and Computational Foundations. Oxford University Press, 2005.