Class probability estimates are unreliable for imbalanced data (and how to fix them)

Proceedings - IEEE International Conference on Data Mining, ICDM

Volumn , Issue , 2012, Pages 695-704

  Wallace, Byron C ^a   Dahabreh, Issa J ^a  

^a BROWN UNIVERSITY   (United States)

Author keywords

Class imbalance; Probability estimates

Indexed keywords

ASYMMETRIC COSTS; BAYESIAN APPROACHES; CLASS IMBALANCE; CLASS PROBABILITIES; CLASSIFICATION SYSTEM; IMBALANCED DATA; POSTERIOR DISTRIBUTIONS; PROBABILITY ESTIMATE; RARE EVENT;

BAYESIAN NETWORKS; CALIBRATION; DATA MINING; ESTIMATION; UNCERTAINTY ANALYSIS;

PROBABILITY DISTRIBUTIONS;

EID: 84874066190     PISSN: 15504786     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICDM.2012.115     Document Type: Conference Paper

References (29)

1. Leo Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996.
2. N.E. Breslow, N.E. Day, et al. Statistical methods in cancer research. Vol. 1. The analysis of case-control studies., volume 1. Distributed for IARC by WHO, Geneva, Switzerland, 1980.
3. G.W. Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
4. N.V. Chawla. Data mining for imbalanced datasets: An overview. Data Mining and Knowledge Discovery Handbook, pages 875-886, 2010.
5. N.V. Chawla, K.W. Bowyer, L.O. Hall, and W.P. K. SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321-357, 2002.
6. D. Cieslak and N. Chawla. Analyzing pets on imbalanced datasets when training and testing class distributions differ. Advances in Knowledge Discovery and Data Mining, pages 519-526, 2008.
7. Gilles Cohen, Mélanie Hilario, Hugo Sax, Stéphane Hugonnet, and Antoine Geissbuhler. Learning from imbalanced data in surveillance of nosocomial infection. Artificial Intelligence in Medicine, 37(1):7-18, 2006.
8. I. Cohen and M. Goldszmidt. Properties and benefits of calibrated classifiers. Knowledge Discovery in Databases: PKDD 2004, pages 125-136, 2004.
9. C. Elkan. The foundations of cost-sensitive learning. In International Joint Conference on Artificial Intelligence, volume 17, pages 973-978, 2001.
10. D. Firth. Bias reduction of maximum likelihood estimates. Biometrika, 80(1):27-38, 1993.
11. D.P. Foster and R.A. Stine. Variable selection in data mining: Building a predictive model for bankruptcy. Journal of the American Statistical Association, 99(466):303-313, 2004.
12. Xinjian Guo, Yilong Yin, Cailing Dong, Gongping Yang, and Guantong Zhou. On the class imbalance problem. pages 192-201, 2008.
13. H. Haibo and E.A. Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1263-1284, 2009.
14. T. Hastie and R. Tibshirani. Classification by pairwise coupling. The annals of statistics, 26(2):451-471, 1998.
15. Jason Van Hulse, Taghi M. Khoshgoftaar, and Amri Napolitano. Experimental perspectives on learning from imbalanced data. pages 935-942, 2007.
16. Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent Data Analysis, 6(5):429-449, 2002.
17. G. King and L. Zeng. Logistic regression in rare events data. Political analysis, 9(2):137-163, 2001.
18. H.T. Lin, C.J. Lin, and R.C. Weng. A note on platts probabilistic outputs for support vector machines. Machine learning, 68(3):267-276, 2007.
19. P. McCullagh and J.A. Nelder. Generalized Linear Models. Chapman and Hall, 1987.
20. A. Niculescu-Mizil and R. Caruana. Obtaining calibrated probabilities from boosting. In Proc. 21st Conference on Uncertainty in Artificial Intelligence (UAI05), pages 413-420, 2005.
21. A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625-632. ACM, 2005.
22. J. Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
23. Foster Provost. Machine learning from imbalanced data sets 101. In Proc. of the AAAI Workshop on Learning from Imbalanced Data Sets, 2000.
24. S. Rabe-Hesketh and A. Skrondal. Multilevel and longitudinal modeling using Stata. Stata Corp, 2008.
25. B.C. Wallace, K. Small, C.E. Brodley, and T.A. Trikalinos. Class imbalance, redux. In Data Mining (ICDM), 2011 IEEE 11th International Conference on, pages 754-763. IEEE, 2011.
26. SD Walter. Small sample estimation of log odds ratios from logistic regression and fourfold tables. Statistics in medicine, 4(4):437-444, 1985.
27. Wu Yang and Xindong Wu. 10 challenging problems in data mining research. International Journal of Information Technology Decision Making, 5(4):597-604, 2006.
28. B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699. ACM, 2002.
29. J. Zhu and E. Hovy. Active learning for word sense disambiguation with methods for addressing the class imbalance problem. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 783-790, 2007.