Probability estimation with machine learning methods for dichotomous and multicategory outcome: Theory

Biometrical Journal

Volumn 56, Issue 4, 2014, Pages 534-563

  Kruppa, Jochen ^a   Liu, Yufeng ^b   Biau, Gérard ^c,d   Kohler, Michael ^e   König, Inke R ^a   Malley, James D ^f   Ziegler, Andreas ^a,g  

^a UNIVERSITY HOSPITAL SCHLESWIG HOLSTEIN   (Germany)

^b University of North Carolina at Chapel Hill   (United States)

^c SORBONNE UNIVERSITÉ   (France)

^d ECOLE NORMALE SUPÉRIEURE   (France)

^e DARMSTADT UNIVERSITY OF TECHNOLOGY   (Germany)

^f NATIONAL INSTITUTES OF HEALTH   (United States)

^g Ratzeburger Allee 160   (Germany)

Author keywords

Bagged nearest neighbor; Nonparametric regression; Probability estimation; Random forest; Support vector machine

Indexed keywords

NEAREST NEIGHBOR SEARCH; PROBLEM SOLVING; REGRESSION ANALYSIS;

BAGGED NEAR NEIGHBOR; MACHINE LEARNING APPROACHES; MACHINE LEARNING METHODS; MULTICATEGORY; MULTINOMIAL LOGISTIC REGRESSION; NEAREST-NEIGHBOUR; NON-PARAMETRIC REGRESSION; PROBABILITY ESTIMATION; RANDOM FORESTS; SUPPORT VECTORS MACHINE;

SUPPORT VECTOR MACHINES;

ARTIFICIAL INTELLIGENCE; NONPARAMETRIC TEST; PROBABILITY; REGRESSION ANALYSIS; SUPPORT VECTOR MACHINE; THEORETICAL MODEL;

ARTIFICIAL INTELLIGENCE; MODELS, THEORETICAL; PROBABILITY; REGRESSION ANALYSIS; STATISTICS, NONPARAMETRIC; SUPPORT VECTOR MACHINES;

EID: 84903648854     PISSN: 03233847     EISSN: 15214036     Source Type: Journal    
DOI: 10.1002/bimj.201300068     Document Type: Article

References (105)

1. Arminger, G. and Enache, D. (1996). Statistical models and artificial neural networks. In: Bock, H. H. and Polasek, W., (eds), Data Analysis and Informations Systems. Springer, Heidelberg, pp. 243-260.
2. Austin, P. C., Lee, D. S., Steyerberg, E. W. and Tu, J. V. (2012). Regression trees for predicting mortality in patients with cardiovascular disease: what improvement is achieved by using ensemble-based methods? Biometrical Journal 54, 657-673.
3. Barla, A., Jurman, G., Riccadonna, S., Merler, S., Chierici, M. and Furlanello, C. (2008). Machine learning methods for predictive proteomics. Briefings in Bioinformatics 9, 119-128.
4. Bartlett, P. L., Jordan, M. I. and McAuliffe, J. D. (2006). Convexity, classification and risk bounds. Journal of the American Statistical Association 101, 138-156.
5. Benichou, J., Gail, M. H. and Mulvihill, J. J. (1996). Graphs to estimate an individualized risk of breast cancer. Journal of Clinical Oncology 14, 103-110.
6. Biau, G. (2012). Analysis of a random forests model. Journal of Machine Learning Research 13, 1063-1095.
7. Biau, G., Cérou, F. and Guyader, A. (2010). On the rate of convergence of the bagged nearest neighbor estimate. Journal of Machine Learning Research 11, 687-712.
8. Biau, G. and Devroye, L. (2010). On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification. Journal of Multivariate Analysis 101, 2499-2518.
9. Biau, G., Devroye, L. and Lugosi, G. (2008). Consistency of random forests and other averaging classifiers. Journal of Machine Learning Research 9, 2039-2057.
10. Bickel, P., Ritov, Y. and Tsybakov, A. (2009). Simultaneous analysis of lasso and dantzig selector. Annals of Statistics 37, 1705-1732.
11. Billingsley, P. (1995). Probability and Measure, John Wiley & Sons, New York, NY.
12. Breiman, L. (1996). Bagging predictors. Machine Learning 24, 123-140.
13. Breiman, L. (2001). Random forests. Machine Learning 45, 5-32.
14. Breiman, L. (2004). Consistency for a simple model of random forests. Technical report 670. Statistics department, University of California at Berkeley, CA. http://www.stat.berkeley.edu/breiman/RandomForests/consistencyRFA.pdf, accessed July 09, 2013.
15. Bühlmann, P. and Yu, B. (2002). Analyzing bagging. Annals of Statistics 30, 927-961.
16. Buja, A. and Stuetzle, W. (2006). Observations on bagging. Statistica Sinica 16, 323-352.
17. Bunea, F., Tsybakov, A. and Wegkamp, M. (2007). Sparsity oracle inequalities for the lasso. Electronic Journal of Statistics 1, 169-194.
18. Candés, E. J. and Tao, T. (2007). The dantzig selector: statistical estimation when p is much larger than n. Annals of Statistics 35, 2313-2351.
19. Carlin, J. B., Wolfe, R., Coffey, C. and Patton, G. C. (1999). Analysis of binary outcomes in longitudinal studies using weighted estimating equations and discrete-time survival methods: prevalence and incidence of smoking in an adolescent cohort. Statistics in Medicine 18, 2655-2679.
20. Chapelle, O., Vapnik, V., Bousquet, O. and Mukherjee, S. (2002). Choosing multiple parameters for support vector machines. Machine Learning 46, 131-159.
21. Chen, X. and Ishwaran, H. (2012). Random forests for genomic data analysis. Genomics 99, 323-329.
22. Claus, E. B. (2000). Risk models in genetic epidemiology. Statistical Methods in Medical Research 9, 589-601.
23. Cover, T. M. (1968). Rates of convergence for nearest neighbor procedures. In: Kinariwala, B. K. and Kuo, F. F. (Eds), Proceedings of the Hawaii International Conference on System Sciences, 413-415. University of Hawaii Press, Honolulu, HI.
24. Cox, J. L., Vallis, T. M., Pfammatter, A., Szpilfogel, C., Carr, B. and O'Neill, B. J. (2013). A novel approach to cardiovascular health by optimizing risk management (anchor): behavioural modification in primary care effectively reduces global risk. Canadian Journal of Cardiology 29, 1400-1407.
25. Cucker, F. and Smale, S. (2002). On the mathematical foundations of learning. Bulletin of the American Mathematical Society 39, 1-49.
26. D'Agostino, R. B.. S. (2012). Cardiovascular risk estimation in 2012: lessons learned and applicability to the HIV population. Journal of Infectious Diseases 205 (Suppl 3), S362-S367.
27. Devroye, L. (1982). Any discrimination rule can have arbitrarily bad probability of error for finite sample size. IEEE Transactions on Pattern Analysis and Machine Intelligence 4, 154-157.
28. Devroye, L., Györfi, L., Krzyzak, A. and Lugosi, G. (1994). On the strong universal consistency of nearest neighbor regression function estimates. Annals of Statistics 22, 1371-1385.
29. Devroye, L., Györfi, L. and Lugosi, G. (1996). A Probabilistic Theory of Pattern Recognition, Springer, Berlin, DE.
30. Devroye, L. and Wagner, T. J. (1980). Distribution-free consistency results in nonparametric discrimination and regression function estimation. Annals of Statistics 8, 231-239.
31. Díaz-Uriarte, R. and Alvarez de Andrés, S. (2006). Gene selection and classification of microarray data using random forest. BMC Bioinformatics 7, 3.
32. Domeniconi, C. and Yan, B. (2004). Nearest neighbor ensemble. In: Kittler, J. and Petrou, M. and Nixon, M. S. and Hancock, E. R. (Eds.), Proceedings of the 17th International Conference on Pattern Recognition, 2004, 228-231. IEEE Computer Society Press, Cambridge, UK.
33. Friedman, J., Hastie, T. and Tibshirani, R. (2000). Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors. Annals of Statistics 28, 337-407.
34. Friedman, J. H. and Hall, P. (2007). On bagging and nonlinear estimation. Journal of Statistical Planning and Inference 137, 669-683.
35. Genton, M. G. (2002). Classes of kernels for machine learning: a statistics perspective. Journal of Machine Learning Research 2, 299-312.
36. Genuer, R. (2012) Variance reduction in purely random forests. Journal of Nonparametric Statistics 24, 543-562.
37. Genuer, R., Poggi, J. M. and Tuleau, C. (2008). Random forests: some methodological insights. Technical report 6729. Institut national de recherche en informatique et en automatique, centre de recherche inria saclay, Ãle-de-france, Orsay, FR. http://arxiv.org/pdf/0811.3619v1.pdf. Accessed May 14, 2012.
38. Gillmann, G. and Minder, C. E. (2009). On graphically checking goodness-of-fit of binary logistic regression models. Methods of Information in Medicine 48, 306-310.
39. Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102, 359-378.
40. Györfi, L., Kohler, M., Krzyzak, A. and Walk, H. (2002). A Distribution-Free Theory of Nonparametric Regression, Springer, New York, NY.
41. Hall, P. and Samworth, R. J. (2005). Properties of bagged nearest neighbour classifiers. Journal of Royal Statistical Society, Series B 67, 363-379.
42. Harrell, F. E. (2001). Regression Modeling Strategies. With Applications to Linear Models, Logistic Regression, and Survival Analysis, Springer, New York, NY.
43. Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd edn. Springer, New York, NY.
44. Hastie, T. J. and Tibshirani, R. J. (1990). Generalized Additive Models, Chapman and Hall, London, UK.
45. Kohler, M. (2002). Universal consistency of local polynomial kernel regression estimates. Annals of the Institute of Statistical Mathematics 54, 879-899.
46. Kohler, M. and Krzyzak, A. (2001). Nonparametric regression estimation using penalized least squares. IEEE Transactions on Information Theory 47, 3054-3058.
47. Kohler, M., Krzyzak, A. and Walk, H. (2006). Rates of convergence for partitioning and nearest neighbor regression estimates with unbounded data. Journal of Multivariate Analysis 97, 311-323.
48. Kohler, M., Krzyzak, A. and Walk, H. (2009). Optimal global rates of convergence for nonparametric regression with unbounded data. Journal of Statistical Planning and Inference 139, 1286-1296.
49. König, I. R., Malley, J. D., Pajevic, S., Weimar, C., Diener, H. C. and Ziegler, A. (2008). Patient-centered yes/no prognosis using learning machines. International Journal of Data Mining and Bioinformatics 2, 289-341.
50. König, I. R., Malley, J. D., Weimar, C., Diener, H. C., Ziegler, A. and on behalf of the German Stroke Study Collaborators (2007). Practical experiences on the necessity of external validation. Statistics in Medicine 26, 5499-5511.
51. Kruppa, J., Liu, Y., Diener, H. C., Holste, T., Weimar, C., König, I. R., and Ziegler, A. (2014). Probability estimation with machine learning methods for dichotomous and multi-category outcome: applications. Biometrical Journal 56, 564-583.
52. Kruppa, J., Schwarz, A., Arminger, G. and Ziegler, A. (2013). Consumer credit risk: individual probability estimates using machine learning. Expert Systems with Applications 40, 5125-5131.
53. Kruppa, J., Ziegler, A. and König, I. R. (2012). Risk estimation and risk prediction using machine learning methods. Human Genetics 131, 1639-1654.
54. Lee, B. K., Lessler, J. and Stuart, E. A. (2010). Improving propensity score weighting using machine learning. Statistics in Medicine 29, 337-346.
55. Lee, Y. K., Lin, Y. and Wahba, G. (2004). Multicategory support vector machines: theory and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association 99, 67-81.
56. Li, J., Jiang, B. and Fine, J. P. (2013). Multicategory reclassification statistics for assessing improvements in diagnostic accuracy. Biostatistics 14, 382-394.
57. Lin, Y. (2002). Support vector machines and the Bayes rule in classification. Data Mining and Knowledge Discovery 6, 259-275.
58. Lin, Y. and Jeon, Y. (2006). Random forests and adaptive nearest neighbors. Journal of the American Statistical Association 101, 578-590.
59. Liu, Y. and Yuan, M. (2011). Reinforced multicategory support vector machines. Journal of Computational and Graphical Statistics 20, 901-919.
60. Liu, Y., Zhang, H. H. and Wu, Y. (2011). Soft or hard classification? Large margin unified machines. Journal of the American Statistical Association 106, 166-177.
61. Lugosi, G. and Zeger, K. (1995). Nonparametric estimation via empirical risk minimization. IEEE Transactions on Information Theory 41, 677-687.
62. Malley, D. J., Malley, K. G. and Pajevic, S. (2011). Statistical Learning for Biomedical Data, Cambridge University Press, Cambridge, UK.
63. Malley, J. D., Kruppa, J., Dasgupta, A., Malley, K. G. and Ziegler, A. (2012). Probability machines. consistent probability estimation using nonparametric learning machines. Methods of Information in Medicine 51, 74-81.
64. Mease, D., Wyner, A. J. and Buja, A. (2007). Boosted classification trees and class probability/quantile estimation. Journal of Machine Learning Research 8, 409-439.
65. Meinshausen, N. (2006). Quantile regression forests. Journal of Machine Learning Research 7, 983-999.
66. Moguerza, J. M. and Muñoz, A. (2006). Support vector machines with applications. Statistical Science 21, 322-336.
67. Perk, J., De Backer, G., Gohlke, H., Graham, I., Reiner, Z., Verschuren, M., Albus, C., Benlian, P., Boysen, G., Cifkova, R., Deaton, C., Ebrahim, S., Fisher, M., Germano, G., Hobbs, R., Hoes, A., Karadeniz, S., Mezzani, A., Prescott, E., Ryden, L., Scherer, M., Syvanne, M., Scholte op Reimer, W. J., Vrints, C., Wood, D., Zamorano, J. L., Zannad, F., Fifth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice and European Association for Cardiovascular Prevention and Rehabilitation (2012). European Guidelines on cardiovascular disease prevention in clinical practice (version 2012): The Fifth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice (constituted by representatives of nine societies and by invited experts). European Heart Journal 33, 1635-1701.
68. Platt, J. C. (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Smola, A. and Bartlett, P. and Schölkopf, B. and Schuurmans, D. (Eds.), Advances in Large Margin Classifiers. MIT Press, Cambridge, MA, pp. 61-74.
69. Porzelius, C., Johannes, M., Binder, H. and Beissbarth, T. (2011). Leveraging external knowledge on molecular interactions in classification methods for risk prediction of patients. Biometrical Journal 53, 190-201.
70. Prentice, R. L. and Pyke, R. (1979). Logistic disease incidence models and case-control studies. Biometrika 66, 403-411.
71. Provost, F. and Domingos, P. (2003). Tree induction for probability-based ranking. Machine Learning 52, 199-215.
72. Royston, P. and Sauerbrei, W. (2005). Building multivariable regression models with continuous covariates in clinical epidemiology-with an emphasis on fractional polynomials. Methods of Information in Medicine 44, 561-571.
73. Royston, P. and Sauerbrei, W. (2008). Multivariable Model-Building: A Pragmatic Approach to Regression Analysis Based on Fractional Polynomials for Modelling Continuous Variables, John Wiley & Sons, New York, NY.
74. Sajda, P. (2006). Machine learning for detection and diagnosis of disease. Annual Review of Biomedical Engineering 8, 537-565.
75. Samworth, R. J. (2012). Optimal weighted nearest neighbour classifiers. Annals of Statistics 40, 2733-2763.
76. Schapire, R. E. (1990). The strength of weak learnability. Machine Learning 5, 197-227.
77. Schölkopf, B. and Smola, A. J. (2002). Learning with Kernels, MIT Press, Cambridge, MA.
78. Schwarz, D. F., König, I. R. and Ziegler, A. (2010). On safari to random jungle: a fast implementation of random forests for high dimensional data. Bioinformatics 26, 1752-1758.
79. Selvaraj, F. J., Mohamed, M., Omar, K., Nanthan, S., Kusiar, Z., Subramaniam, S. Y., Ali, N., Karanakaran, K., Ahmad, F., Low, W. H. and DISSEMINATE study group. (2012). The impact of a disease management program (coach) on the attainment of better cardiovascular risk control in dyslipidaemic patients at primary care centres (the DISSEMINATE study): a randomised controlled trial. BMC Family Practice 13, 97.
80. Smola, A. and Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing 14, 199-222.
81. Spiegelman, C. and Sacks, J. (1980). Consistent window estimation in nonparametric regression. Annals of Statistics 8, 240-246.
82. Stanski, H. R., Wilson, L. J. and Burrows, W. R. (1989). Survey of common verification methods in meteorology. World weather watch tech. rep. 8, wmo/td no. 358. world meteorological organization. http://www.cawcr.gov.au/projects/verification/Stanski_et_al/Stanski_et_al.html, accessed April 01, 2013.
83. Steinbach, M. and Tan, P. N. (2009). kNN: k-nearest neighbors. In: Wu, X. and Kumar, V. (Eds.), The Top Ten Algorithms in Data Mining, CRC Press, Boca Raton, FL, pp. 151-161.
84. Steinwart, I. and Scovel, C. (2007). Fast rates for support vector machines using gaussian kernels. Annals of Statistics 35, 575-607.
85. Steyerberg, E. W. (2009). Clinical Prediction Models, Springer, New York, NY.
86. Stone, C. J. (1977). Consistent nonparametric regression. Annals of Statistics 5, 595-645.
87. Stone, C. J. (1982). Optimal global rates of convergence for nonparametric regression. Annals of Statistics 10, 1040-1053.
88. Strobl, C., Malley, J. and Tutz, G. (2009). An introduction to recursive partitioning: rationale, application and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods 14, 323-348.
89. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of Royal Statistical Society, Series B 58, 267-288.
90. van Iterson, M., van Haagen, H. H. and Goeman, J. J. (2012). Resolving confusion of tongues in statistics and machine learning: a primer for biologists and bioinformaticians. Proteomics 12, 543-549.
91. Vapnik, V. (1998). Statistical Learning Theory, John Wiley & Sons, New York, NY.
92. Wahba, G. (2002). Soft and hard classification by reproducing kernel hilbert space methods. Proceedings of the National Academy of Sciences, USA 102, 12 332-12 337.
93. Walk, H. (2005). Strong universal consistency of smooth kernel regression estimates. Annals of the Institute of Statistical Mathematics 57, 665-685.
94. Wang, J., Shen, X. and Liu, Y. (2008). Probability estimation for large-margin classifiers. Biometrika 95, 149-167.
95. Wang, L. and Shen, X. (2007). On l-norm multi-class support vector machines: methodology and theory. Journal of the American Statistical Association 102, 595-602.
96. Wu, T. F., Lin, C. J. and Weng, R. C. (2004). Probability estimates for multi-class classification by pairwise coupling. Journal of Machine Learning Research 5, 975-1005.
97. Wu, Y. and Liu, Y. (2007). Robust truncated-hinge-loss support vector machines. Journal of the American Statistical Association 102, 974-983.
98. Wu, Y. and Liu, Y. (2011). Non-crossing large-margin probability estimation and its application to robust SVM via preconditioning. Statistical Methodology 8, 56-67.
99. Wu, Y. and Liu, Y. (2013). Adaptively weighted large margin classifiers. Journal of Computational and Graphical Statistics 22, 416-432.
100. Wu, Y., Zhang, H. H. and Liu, Y. (2010). Robust model-free multiclass probability estimation. Journal of the American Statistical Association 105, 424-436.
101. Xue, H., Yang, Q. and Chen, S. (2009). SVM: support vector machines. In: Wu, X. and Kumar, V., (Eds.), The Top Ten Algorithms in Data Mining. CRC Press, Boca Raton, FL, pp. 37-59.
102. Young, I. D. (2007). Introduction to Risk Calculation in Genetic Counseling, Oxford University Press, Oxford, UK.
103. Zhang, C. and Liu, Y. (2013). Multicategory large-margin unified machines. Journal of Machine Learning Research 14, 1349-1386.
104. Zhang, T. (2004a). Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research 5, 1225-1251.
105. Zhang, T. (2004b). Statistical behavior and consistency of classification methods based on convex risk minimization. Annals of Statistics 32, 56-85.