Cost-sensitive AdaBoost algorithm for ordinal regression based on extreme learning machine

IEEE Transactions on Cybernetics

Volumn 44, Issue 10, 2014, Pages 1898-1909

  Riccardi, Annalisa ^a   Fernández Navarro, Francisco ^a   Carloni, Sante ^a  

^a EUROPEAN SPACE AGENCY   (Netherlands)

Author keywords

Boosting; extreme learning machine; neural networks; ordinal regression; SAMME algorithm

Indexed keywords

BOOSTING; COST-SENSITIVE ADABOOST; EXTREME LEARNING MACHINE; ORDINAL REGRESSION;

NEURAL NETWORKS;

ALGORITHM; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; REGRESSION ANALYSIS;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; LEAST-SQUARES ANALYSIS; NEURAL NETWORKS (COMPUTER); REGRESSION ANALYSIS;

EID: 84904892728     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2014.2299291     Document Type: Article

References (63)

1. A. K. Jain, R. P. Duin, and J. Mao, "Statistical pattern recognition: A review," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 1, pp. 4-37, Jan. 2000.
2. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, 2nd ed. New York, NY, USA: Wiley, 2000.
3. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques (Data Management Systems), 2nd ed. San Mateo, CA, USA: Morgan Kaufmann, 2005.
4. V. Cherkassky and F. M. Mulier, Learning From Data: Concepts, Theory, and Methods. New York, NY, USA: Wiley, 2007.
5. B.-Y. Sun, J. Li, D. D. Wu, X.-M. Zhang, and W.-B. Li, "Kernel discriminant learning for ordinal regression," IEEE Trans. Knowl. Data Eng., vol. 22, no. 6, pp. 906-910, Jun. 2010.
6. C.-W. Seah, I. W. Tsang, and Y.-S. Ong, "Transductive ordinal regression," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1074-1086, Jul. 2012.
7. P. McCullagh, "Regression models for ordinal data,"J. Roy. Stat. Soc. Ser. B (Meth.), vol. 42, no. 2, pp. 109-142, 1980.
8. J. A. Anderson, "Regression and ordered categorical variables," J. Roy. Stat. Soc. Ser. B (Meth.), vol. 46, no. 1, pp. 1-30, 1984.
9. M. J. Mathieson, "Ordinal models for neural networks," in Proc. 3rd Int. Conf. Neural Netw. Capital Markets, 1996, pp. 523-536.
10. F. Fernández-Navarro, P. Campoy, M. De la Paz, C. Hervás-Martínez, and X. Yao, "Addressing the EU sovereign ratings using an ordinal regression approach," IEEE Trans. Cybern., vol. 43, no. 6, pp. 2228-2240, Dec. 2013.
11. R. Herbrich, T. Graepel, and K. Obermayer, "Large margin rank boundaries for ordinal regression," in Advances in Large Margin Classifiers, A. Smola, P. Bartlett, B. Schölkopf, and D. Schuurmans, Eds. Cambridge, MA, USA: MIT Press, 2000, pp. 115-132.
12. W. Chu and S. S. Keerthi, "Support vector ordinal regression," Neural Comput., vol. 19, no. 3, pp. 792-815, 2007.
13. W. Chu and Z. Ghahramani, "Gaussian processes for ordinal regression," J. Mach. Learn. Res., vol. 6, pp. 1019-1041, Dec. 2005.
14. W. Deng and L. Chen, "Color image watermarking using regularized extreme learning machine," Neural Netw. World, vol. 20, no. 3, pp. 317-330, 2010.
15. D. Becerra-Alonso, M. Carbonero-Ruz, F. J. Martínez-Estudillo, and A. C. Martínez-Estudillo, "Evolutionary extreme learning machine for ordinal regression," in Proc. Neural Information Processing, LNCS 7665. 2012, pp. 217-227.
16. Q.-Y. Zhu, A. K. Qin, P. N. Suganthan, and G.-B. Huang, "Evolutionary extreme learning machine," Pattern Recognit., vol. 38, no. 10, pp. 1759-1763, 2005.
17. Y. Liu, X. Yao, and T. Higuchi, "Ensembles with negative correlation learning," IEEE Trans. Evol. Comput., vol. 4, no. 4, pp. 380-387, Nov. 2000.
18. A. Chandra and X. Yao, "Divace: Diverse and accurate ensemble learning algorithm," in Proc. 5th Int. Conf. Intell. Data Eng. Autom. Learn., vol. 3177. 2005, pp. 619-625.
19. X. Zhu, P. Zhang, X. Lin, and Y. Shi, "Active learning from stream data using optimal weight classifier ensemble," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 40, no. 6, pp. 1607-1621, Dec. 2010.
20. D. Hernandez-Lobato, G. Martinez-Muoz, and A. Suarez, "Empirical analysis and evaluation of approximate techniques for pruning regression bagging ensembles," Neurocomputing, vol. 74, nos. 12-13, pp. 2250-2264, 2011.
21. A. L. Coelho and D. S. Nascimento, "On the evolutionary design of heterogeneous bagging models," Neurocomputing, vol. 73, nos. 16-18, pp. 3319-3322, 2010.
22. Z. Qi, Y. Xu, L. Wang, and Y. Song, "Online multiple instance boosting for object detection," Neurocomputing, vol. 74, no. 10, pp. 1769-1775, 2011.
23. E. Frank and M. Hall, "A simple approach to ordinal classification," in Proc. ECML, 2001, pp. 145-156.
24. W. Waegeman and L. Boullart, "An ensemble of weighted support vector machines for ordinal regression," Int. J. Comput. Syst. Sci. Eng., vol. 3, no. 1, pp. 47-51, 2009.
25. F. Fernández-Navarro, P. Gutierrez, C. Hervás-Martínez, and X. Yao, "Negative correlation ensemble learning for ordinal regression," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 11, pp. 1836-1849, Nov. 2013.
26. Y. Liu and X. Yao, "Negatively correlated neural networks can produce best ensembles," Aust. J. Intell. Inf. Process. Syst., vol. 4, no. 3, pp. 176-185, 1997.
27. Y. Liu and X. Yao, "Ensemble learning via negative correlation," Neural Netw., vol. 12, no. 10, pp. 1399-1404, 1999.
28. M. Pérez-Ortiz, P. Gutiérrez, and C. Hervás-Martínez, "Projection-based ensemble learning for ordinal regression," IEEE Trans. Cybern., 2014, DOI http://dx.doi.org/10.1109/TCYB.2013.2266336 in press.
29. H.-T. Lin and L. Li, "Large-margin thresholded ensembles for ordinal regression: Theory and practice," in Proc. 17th Int. Conf. Algorithmic Learn. Theory, 2006, pp. 319-333.
30. H.-T. Lin and L. Li, "Combining ordinal preferences by boosting," in Proc. ECML/PKDD, 2009, pp. 69-83.
31. J. Zhu, H. Zou, S. Rosset, and T. Hastie, "Multi-class AdaBoost," Stat. Interface, vol. 2, no. 1, pp. 349-360, 2009.
32. R. E. Schapire and Y. Singer, "Improved boosting algorithms using confidence-rated predictions," Mach. Learn., vol. 37, no. 3, pp. 297-336, 1999.
33. G.-B. Huang, D. Wang, and Y. Lan, "Extreme learning machines: A survey," Int. J. Mach. Learn. Cybern., vol. 2, no. 2, pp. 107-122, 2011.
34. G. Wang and P. Li, "Dynamic AdaBoost ensemble extreme learning machine," in Proc. ICACTE, vol. 3. 2010, pp. V3-54-V3-58.
35. H.-X. Tian and Z.-Z. Mao, "An ensemble ELM based on modified AdaBoost.RT algorithm for predicting the temperature of molten steel in ladle furnace," IEEE Trans. Autom. Sci. Eng., vol. 7, no. 1, pp. 73-80, Jan. 2010.
36. J.-H. Zhai, H.-Y. Xu, and X.-Z. Wang, "Dynamic ensemble extreme learning machine based on sample entropy," Soft Comput., vol. 16, no. 9, pp. 1493-1502, 2012.
37. Y. Sun, M. S. Kamel, A. K. C. Wong, and Y. Wang, "Cost-sensitive boosting for classification of imbalanced data," Pattern Recognit., vol. 40, no. 12, pp. 3358-3378, 2007.
38. H.-T. Lin and L. Li, "Reduction from cost-sensitive ordinal ranking to weighted binary classification," Neural Comput., vol. 24, no. 5, pp. 1329-1367, 2012.
39. G. B. Huang, Q. Y. Zhu, and C. K. Siew, "Extreme learning machine: A new learning scheme of feedforward neural networks," in Proc. IEEE Conf. Int. Conf. Neural Netw., vol. 2. Jul. 2004, pp. 985-990.
40. C. Zhaohu, R. Xuemei, and C. Qiang, "Camera calibration based on extreme learning machine," in Proc. Int. Conf. Commun., Electron. Autom. Eng., vol. 181. 2013, pp. 115-120.
41. H. Zhou, Y. Lan, Y. C. Soh, G.-B. Huang, and R. Zhang, "Credit risk evaluation with extreme learning machine," in Proc. IEEE Int. Conf. Syst., Man, Cybern., Oct. 2012, pp. 1064-1069.
42. J. Sánchez-Monedero, M. Cruz-Ramírez, F. Fernández-Navarro, J. C. Fernández, P. A. Gutiérrez, and C. Hervás-Martínez, "On the suitability of extreme learning machine for gene classification using feature selection," in Proc. ISDA, 2010, pp. 507-512.
43. G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, "Extreme learning machine: Theory and applications," Neurocomputing, vol. 70, nos. 1-3, pp. 489-501, 2006.
44. G.-B. Huang, H. Zhou, X. Ding, and R. Zhang, "Extreme learning machine for regression and multiclass classification," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 2, pp. 513-529, Apr. 2012.
45. V. N. Vapnik, The Nature of Statistical Learning Theory. Berlin, Germany: Springer, 1999.
46. R. Zhang, G.-B. Huang, N. Sundararajan, and P. Saratchandran, "Multicategory classification using an extreme learning machine for microarray gene expression cancer diagnosis," IEEE/ACM Trans. Comput. Biol. Bioinform., vol. 4, no. 3, pp. 485-495, Jul./Sep. 2007.
47. P. Bartlett, "The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network," IEEE Trans. Inf. Theory, vol. 44, no. 2, pp. 525-536, Mar. 1998.
48. J. Cao, Z. Lin, and G.-B. Huang, "Composite function wavelet neural networks with extreme learning machine," Neurocomputing, vol. 73, nos. 7-9, pp. 1405-1416, 2010.
49. F. Fernández-Navarro, C. Hervás-Martínez, J. Sánchez-Monedero, and P. A. Gutierrez, "MELM-GRBF: A modified version of the extreme learning machine for generalized radial basis function neural networks," Neurocomputing, vol. 74, no. 16, pp. 2502-2510, 2011.
50. A. E. Hoerl and R. W. Kennard, "Ridge regression: Biased estimation for nonorthogonal problems," Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
51. A. Asuncion and D. Newman. (2007). UCI Machine Learning Repository [Online]. Available: http://www.ics.uci.edu/mlearn/MLRepository.html
52. J. S. Cardoso and J. F. P. da Costa, "Learning to classify ordinal data: The data replication method," J. Mach. Learn. Res., vol. 8, pp. 1393-1429, Dec. 2007.
53. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, "The WEKA data mining software: An update," Special Interest Group Knowl. Discovery Data Mining Explorer Newslett., vol. 11, pp. 10-18, Nov. 2009.
54. W.-Y. Deng, Q.-H. Zheng, S. Lian, L. Chen, and X. Wang, "Ordinal extreme learning machine," Neurocomputing, vol. 74, nos. 1-3, pp. 447-456, 2010.
55. A. Castaño, F. Fernández-Navarro, and C. Hervás-Martínez, "PCAELM: A robust and pruned extreme learning machine approach based on principal component analysis," Neural Process. Lett., vol. 37, no. 3, pp. 377-392, 2013.
56. J. Demšar, "Statistical comparisons of classifiers over multiple data sets," J. Mach. Learn. Res., vol. 7, pp. 1-30, Dec. 2006.
57. M. Friedman, "A comparison of alternative tests of significance for the problem of m rankings," Ann. Math. Stat., vol. 11, no. 1, pp. 86-92, 1940.
58. Y. Hochberg and A. Tamhane, Multiple Comparison Procedures. New York, NY, USA: Wiley, 1987.
59. P. A. Gutiérrez, M. Pérez-Ortiz, F. Fernández-Navarro, J. Sánchez-Monedero, and C. Hervás-Martínez, "An experimental study of different ordinal regression methods and measures," in Hybrid Artificial Intelligent Systems, LNCS 7209. Berlin/Heidelberg, Germany: Springer, 2012, pp. 296-307.
60. G. Brown, J. L. Wyatt, and P. Tiňo, "Managing diversity in regression ensembles," J. Mach. Learn. Res., vol. 6, pp. 1621-1650, Dec. 2005.
61. H. He, S. Chen, K. Li, and X. Xu, "Incremental learning from stream data," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1901-1914, Dec. 2011.
62. H. Mohammed, J. Leander, M. Marbach, and R. Polikar, "Can AdaBoost. M1 learn incrementally? A comparison to Learn++ under different combination rules," in Proc. ICANN, vol. 4131. 2006, pp. 254-263.
63. M. D. Muhlbaier, A. Topalis, and R. Polikar, "Learn++.NC: Combining ensemble of classifiers with dynamically weighted consult-and-vote for efficient incremental learning of new classes," IEEE Trans. Neural Netw., vol. 20, no. 1, pp. 152-168, Jan. 2009.