A boosted decision tree approach using Bayesian hyper-parameter optimization for credit scoring

Expert Systems with Applications

Volumn 78, Issue , 2017, Pages 225-241

  Xia, Yufei ^a   Liu, Chuanzhe ^a   Li, YuYing ^a   Liu, Nana ^a  

^a CHINA UNIVERSITY OF MINING AND TECHNOLOGY   (China)

Author keywords

Bayesian hyper parameter optimization; Boosted decision tree; Credit scoring

Indexed keywords

DATA HANDLING; DECISION TREES;

AREA UNDER THE CURVES; BOOSTED DECISION TREES; CREDIT SCORING; CREDIT SCORING MODEL; DATA PREPROCESSING; EVALUATION MEASURES; HYPER-PARAMETER OPTIMIZATIONS; REDUNDANT VARIABLES;

OPTIMIZATION;

EID: 85013170820     PISSN: 09574174     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.eswa.2017.02.017     Document Type: Article

References (67)

1. Ala'raj, M., Abbod, M.F., Classifiers consensus system approach for credit scoring. Knowledge-Based Systems 104 (2016), 89–105.
2. Ala'raj, M., Abbod, M.F., A new hybrid ensemble credit scoring model based on classifiers consensus system approach. Expert Systems with Applications 64 (2016), 36–55.
3. Altman, E.I., Financial ratios, discriminant analysis and the prediction of corporate bankruptcy. The Journal of Finance 23 (1968), 589–609.
4. Baesens, B., Van Gestel, T., Viaene, S., Stepanova, M., Suykens, J., Vanthienen, J., Benchmarking state-of-the-art classification algorithms for credit scoring. Journal of the Operational Research Society 54 (2003), 627–635.
5. Bauer, E., Kohavi, R., An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning 36 (1999), 105–139.
6. Bergstra, J., Bengio, Y., Random search for hyper-parameter optimization. Journal of Machine Learning Research 13 (2012), 281–305.
7. Bergstra, J., Yamins, D., Cox, D.D., Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. Proceedings of the 12th Python in Science Conference, 2013, 13–20.
8. Bergstra, J.S., Bardenet, R., Bengio, Y., Kégl, B., Algorithms for hyper-parameter optimization. Proceedings of advances in neural information processing systems, 2011, 2546–2554.
9. Bishop, C.M., Pattern recognition and machine learning. 2006, Springer, New York.
10. Breiman, L., Bagging predictors. Machine Learning 24 (1996), 123–140.
11. Breiman, L., Random forests. Machine Learning 45 (2001), 5–32.
12. Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A., Classification and regression trees. 1984, CRC press.
13. Brillante, L., Gaiotti, F., Lovat, L., Vincenzi, S., Giacosa, S., Torchio, F., et al. Investigating the use of gradient boosting machine, random forest and their ensemble to predict skin flavonoid content from berry physical–mechanical characteristics in wine grapes. Computers and Electronics in Agriculture 117 (2015), 186–193.
14. Brown, I., Mues, C., An experimental comparison of classification algorithms for imbalanced credit scoring data sets. Expert Systems with Applications 39 (2012), 3446–3453.
15. Chang, C.-C., Lin, C.-J., LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2, 2011, 27.
16. Chen, S., Montgomery, J., Bolufé-Röhler, A., Measuring the curse of dimensionality and its effects on particle swarm optimization and differential evolution. Applied Intelligence 42 (2015), 514–526.
17. Chen, T., & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. arXiv preprint arXiv: 1603.02754.
18. Chen, T., & He, T. (2015). xgboost: EXtreme Gradient Boosting. R package version 0.4-2.
19. Chen, X., Zhou, L., Wan, D., Group social capital and lending outcomes in the financial credit market: An empirical study of online peer-to-peer lending. Electronic Commerce Research and Applications 15 (2016), 1–13.
20. Cortes, C., Vapnik, V., Support-vector networks. Machine Learning 20 (1995), 273–297.
21. Demšar, J., Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7 (2006), 1–30.
22. Demšar, J., Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research 7 (2006), 1–30.
23. Dietterich, T.G., An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine learning 40 (2000), 139–157.
24. Duin, R.P., Tax, D.M., Experiments with classifier combining rules. International workshop on multiple classifier systems, 2000, 16–29.
25. Elith, J., Leathwick, J.R., Hastie, T., A working guide to boosted regression trees. Journal of Animal Ecology 77 (2008), 802–813.
26. Finlay, S., Multiple classifier architectures and their application to credit risk assessment. European Journal of Operational Research 210 (2011), 368–378.
27. Florez-Lopez, R., Ramon-Jeronimo, J.M., Enhancing accuracy and interpretability of ensemble strategies in credit risk assessment. A correlated-adjusted decision forest proposal. Expert Systems with Applications 42 (2015), 5737–5753.
28. Freund, Y., Schapire, R.E., A desicion-theoretic generalization of on-line learning and an application to boosting. European conference on computational learning theory, 1995, 23–37.
29. Friedman, J.H., Greedy function approximation: A gradient boosting machine. Annals of statistics 29 (2001), 1189–1232.
30. Friedman, J.H., Stochastic gradient boosting. Computational Statistics & Data Analysis 38 (2002), 367–378.
31. Guelman, L., Gradient boosting trees for auto insurance loss cost modeling and prediction. Expert Systems with Applications 39 (2012), 3659–3667.
32. Guo, Y., Zhou, W., Luo, C., Liu, C., Xiong, H., Instance-based credit risk assessment for investment decisions in P2P lending. European Journal of Operational Research 249 (2016), 417–426.
33. Hand, D.J., Classifier technology and the illusion of progress. Statistical science 21 (2006), 1–14.
34. Hand, D.J., Measuring classifier performance: A coherent alternative to the area under the ROC curve. Machine learning 77 (2009), 103–123.
35. Hand, D.J., Henley, W.E., Statistical classification methods in consumer credit scoring: A review. Journal of the Royal Statistical Society: Series A (Statistics in Society) 160 (1997), 523–541.
36. Harris, T., Credit scoring using the clustered support vector machine. Expert Systems with Applications 42 (2015), 741–750.
37. Hastie, T., Tibshirani, R., Friedman, J., Boosting and additive trees. The elements of statistical learning, 2009, Springer, 337–387.
38. Huang, C.-L., Chen, M.-C., Wang, C.-J., Credit scoring with a data mining approach based on support vector machines. Expert Systems with Applications 33 (2007), 847–856.
39. Hutter, F., Hoos, H.H., Leyton-Brown, K., Sequential model-based optimization for general algorithm configuration. International conference on learning and intelligent optimization, 2011, 507–523.
40. Johnson, R., Zhang, T., Learning nonlinear functions using regularized greedy forest. IEEE transactions on pattern analysis and machine intelligence 36 (2014), 942–954.
41. Lee, T.-S., Chiu, C.-C., Chou, Y.-C., Lu, C.-J., Mining the customer credit using classification and regression tree and multivariate adaptive regression splines. Computational Statistics & Data Analysis 50 (2006), 1113–1130.
42. Lessmann, S., Baesens, B., Seow, H.-V., Thomas, L.C., Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research. European Journal of Operational Research 247 (2015), 124–136.
43. Lin, S.-W., Ying, K.-C., Chen, S.-C., Lee, Z.-J., Particle swarm optimization for parameter determination and feature selection of support vector machines. Expert Systems with Applications 35 (2008), 1817–1824.
44. Min, J.H., Lee, Y.-C., Bankruptcy prediction using support vector machine with optimal choice of kernel function parameters. Expert Systems with Applications 28 (2005), 603–614.
45. Nanni, L., Lumini, A., An experimental comparison of ensemble of classifiers for bankruptcy prediction and credit scoring. Expert Systems with Applications 36 (2009), 3028–3033.
46. Nascimento, D.S., Coelho, A.L., Canuto, A.M., Integrating complementary techniques for promoting diversity in classifier ensembles: A systematic study. Neurocomputing 138 (2014), 347–357.
47. Nie, G., Rowe, W., Zhang, L., Tian, Y., Shi, Y., Credit card churn forecasting by logistic regression and decision tree. Expert Systems with Applications 38 (2011), 15273–15285.
48. Pötzsch, S., Böhme, R., The role of soft information in trust building: Evidence from online social lending. International conference on trust and trustworthy computing, 2010, 381–395.
49. Paleologo, G., Elisseeff, A., Antonini, G., Subagging for credit scoring models. European Journal of Operational Research 201 (2010), 490–499.
50. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al. Scikit-learn: Machine learning in Python. Journal of Machine learning research 12 (2011), 2825–2830.
51. Serrano-Cinca, C., Gutiérrez-Nieto, B., The use of profit scoring as an alternative to credit scoring systems in peer-to-peer (P2P) lending. Decision Support Systems 89 (2016), 113–122.
52. Simon, H., Neural networks: A comprehensive foundation. 2008, Prentice-Hall of India, New Delhi.
53. Snoek, J., Larochelle, H., Adams, R.P., Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 2012, Curran Associates, 2951–2959.
54. Sutton, C.D., 11-classification and regression trees, bagging, and boosting. Handbook of Statistics 24 (2005), 303–329.
55. Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K., Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining, 2013, 847–855.
56. Tsai, C.-F., Hsu, Y.-F., Yen, D.C., A comparative study of classifier ensembles for bankruptcy prediction. Applied Soft Computing 24 (2014), 977–984.
57. Twala, B., Multiple classifier application to credit risk assessment. Expert Systems with Applications 37 (2010), 3326–3336.
58. Wang, G., Hao, J., Ma, J., Jiang, H., A comparative assessment of ensemble learning for credit scoring. Expert Systems with Applications 38 (2011), 223–230.
59. Wang, G., Ma, J., Huang, L., Xu, K., Two credit scoring models based on dual strategy ensemble trees. Knowledge-Based Systems 26 (2012), 61–68.
60. West, D., Neural network credit scoring models. Computers & operations research 27 (2000), 1131–1152.
61. Wiginton, J.C., A note on the comparison of logit and discriminant models of consumer credit behavior. Journal of Financial and Quantitative Analysis 15 (1980), 757–770.
62. Wolpert, D.H., Macready, W.G., No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1 (1997), 67–82.
63. Wu, T.-C., Hsu, M.-F., Credit risk assessment and decision making by a fusion approach. Knowledge-Based Systems 35 (2012), 102–110.
64. Yeh, C.-C., Lin, F., Hsu, C.-Y., A hybrid KMV model, random forests and rough set theory approach for credit rating. Knowledge-Based Systems 33 (2012), 166–172.
65. Zhang, Y., Haghani, A., A gradient boosting method to improve travel time prediction. Transportation Research Part C: Emerging Technologies 58 (2015), 308–324.
66. Zhang, Y., Jia, H., Diao, Y., Hai, M., Li, H., Research on Credit Scoring by Fusing Social Media Information in Online Peer-to-Peer Lending. Procedia Computer Science 91 (2016), 168–174.
67. Zięba, M., Tomczak, S.K., Tomczak, J.M., Ensemble boosted trees with synthetic features generation in application to bankruptcy prediction. Expert Systems with Applications 58 (2016), 93–101.