Oblique Decision Tree Ensemble via Multisurface Proximal Support Vector Machine

IEEE Transactions on Cybernetics

Volumn 45, Issue 10, 2015, Pages 2165-2176

  Zhang, Le ^a   Suganthan, Ponnuthurai N ^a  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

Author keywords

Bias; ensemble; generalized eigenvectors; oblique; orthogonal; proximal support vector machine (SVM); Random Forest (RaF); Rotation Forest (RoF); Variance

Indexed keywords

CLASSIFICATION (OF INFORMATION); DECISION TREES; FACE RECOGNITION; SUPPORT VECTOR MACHINES;

BENCHMARK CLASSIFICATION; CLASSIFICATION ACCURACY; CLASSIFICATION RESULTS; GEOMETRIC PROPERTIES; OBLIQUE DECISION TREE; PROXIMAL SUPPORT VECTOR MACHINES; REGULARIZATION METHODS; SMALL SAMPLE SIZE PROBLEMS;

GEOMETRY;

EID: 84911466848     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2014.2366468     Document Type: Article

References (63)

1. L. Breiman, "Bias, variance, arcing classifiers," Dept. Stat., Univ. California, Berkeley, CA, USA, Tech. Rep. 460, 1996.
2. M. A. Wiering and H. van Hasselt, "Ensemble algorithms in reinforcement learning," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 930-936, Aug. 2008.
3. J. S. Goerss, "Tropical cyclone track forecasts using an ensemble of dynamical models," Mon. Weather Rev., vol. 128, no. 4, pp. 1187-1193, 2000.
4. T. G. Dietterich, "Ensemble methods in machine learning," in Multiple Classifier Systems. Berlin, Germany: Springer, 2000, pp. 1-15.
5. Z.-H. Zhou, F. Roli, J. Kittler, "Multiple classifier systems," in Proc. 2013 11th Int. Workshop Mult. Classifier Syst. (MCS), Nanjing, China.
6. P. Geurts, D. Ernst, L. Wehenkel, "Extremely randomized trees," Mach. Learn., vol. 63, no. 1, pp. 3-42, 2006.
7. C.-X. Zhang and J.-S. Zhang, "RotBoost: A technique for combining rotation forest and AdaBoost," Pattern Recognit. Lett., vol. 29, no. 10, pp. 1524-1536, 2008.
8. R. Kohavi and D. H. Wolpert, "Bias plus variance decomposition for zero-one loss functions," in Proc. Int. Conf. Mach. Learn. (ICML), Bari, Italy, 1996, pp. 275-283.
9. L. Breiman, "Random forests," Mach. Learn., vol. 45, no. 1, pp. 5-32, 2001.
10. J. J. Rodriguez, L. I. Kuncheva, C. J. Alonso, "Rotation forest: A new classifier ensemble method," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 10, pp. 1619-1630, Oct. 2006.
11. L. Breiman, "Bagging predictors," Mach. Learn., vol. 24, no. 2, pp. 123-140, 1996.
12. T. K. Ho, "The random subspace method for constructing decision forests," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 8, pp. 832-844, Aug. 1998.
13. H. Jiang et al., "Joint analysis of two microarray gene-expression data sets to select lung adenocarcinoma marker genes," BMC Bioinformat., vol. 5, no. 1, p. 81, 2004.
14. K.-Q. Shen, C.-J. Ong, X.-P. Li, Z. Hui, E. P. Wilder-Smith, "A feature selection method for multilevel mental fatigue EEG classification," IEEE Trans. Biomed. Eng., vol. 54, no. 7, pp. 1231-1237, Jul. 2007.
15. B. H. Menze et al., "A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data," BMC Bioinformat., vol. 10, no. 1, p. 213, 2009.
16. T. Hastie, R. Tibshirani, J. J. H. Friedman, The Elements of Statistical Learning, vol. 1. New York, NY, USA: Springer, 2001.
17. T. Hothorn, F. Leisch, A. Zeileis, K. Hornik, "The design and analysis of benchmark experiments," J. Comput. Graph. Stat., vol. 14, no. 3, pp. 675-699, 2005.
18. A. Liaw and M.Wiener, "Classification and regression by randomforest," R News, vol. 2, no. 3, pp. 18-22, 2002.
19. Y. Lin and Y. Jeon, "Random forests and adaptive nearest neighbors," J. Amer. Stat. Assoc., vol. 101, no. 474, pp. 578-590, 2006.
20. B. H. Menze, W. Petrich, F. A. Hamprecht, "Multivariate feature selection and hierarchical classification for infrared spectroscopy: Serum-based detection of bovine spongiform encephalopathy," Anal. Bioanal. Chem., vol. 387, no. 5, pp. 1801-1807, 2007.
21. K.-H. Liu and D.-S. Huang, "Cancer classification using rotation forest," Comput. Biol. Med., vol. 38, no. 5, pp. 601-610, 2008.
22. A. Ozcift, "SVM feature selection based rotation forest ensemble classifiers to improve computer-aided diagnosis of Parkinson disease," J. Med. Syst., vol. 36, no. 4, pp. 2141-2147, 2012.
23. J.-F. Xia, K. Han, D.-S. Huang, "Sequence-based prediction of protein-protein interactions by means of rotation forest and autocorrelation descriptor," Protein Peptide Lett., vol. 17, no. 1, pp. 137-145, 2010.
24. G. Stiglic and P. Kokol, "Effectiveness of rotation forest in metalearning based gene expression classification," in Proc. 2007 20th IEEE Int. Symp. Comput.-Based Med. Syst. (CBMS), Maribor, Slovenia, pp. 243-250.
25. R. E. Banfield, L. O. Hall, K. W. Bowyer, W. P. Kegelmeyer, "A comparison of decision tree ensemble creation techniques," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 1, pp. 173-180, Jan. 2007.
26. K. V. S. Murthy and S. L. Salzberg, "On growing better decision trees from data," Ph.D. dissertation, Dept. Comput. Sci., Johns Hopkins Univ., Baltimore, MD, USA, 1995.
27. N. Manwani and P. Sastry, "Geometric decision tree," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 1, pp. 181-192, Feb. 2012.
28. L. Breiman, Classification and Regression Trees. Boca Raton, FL, USA: CRC Press, 1993.
29. S. K. Murthy, S. Kasif, S. Salzberg, R. Beigel, "OC1: A randomized algorithm for building oblique decision trees," in Proc. 11th Nat. Conf. Artif. Intell. (AAAI), vol. 93. Washington, DC, USA, 1993, pp. 322-327.
30. W. Pedrycz and Z. A. Sosnowski, "Genetically optimized fuzzy decision trees," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 35, no. 3, pp. 633-641, Jun. 2005.
31. S.-H. Cha and C. Tappert, "A genetic algorithm for constructing compact binary decision trees," J. Pattern Recognit. Res., vol. 4, no. 1, pp. 1-13, 2009.
32. E. Cantu-Paz and C. Kamath, "Inducing oblique decision trees with evolutionary algorithms," IEEE Trans. Evol. Comput., vol. 7, no. 1, pp. 54-68, Feb. 2003.
33. J. M. Pangilinan and G. K. Janssens, "Pareto-optimality of oblique decision trees from evolutionary algorithms," J. Global Optim., vol. 51, no. 2, pp. 301-311, 2011.
34. L. Zhang, Y. Ren, P. Suganthan, "Towards generating random forests via extremely randomized trees," in Proc. IEEE 2014 Int. Joint Conf. Neural Netw. (IJCNN), Beijing, China, pp. 2645-2652.
35. X.-Z. Wang, J.-H. Zhai, S.-X. Lu, "Induction of multiple fuzzy decision trees based on rough set technique," Inf. Sci., vol. 178, no. 16, pp. 3188-3202, 2008.
36. X.-Z. Wang and C.-R. Dong, "Improving generalization of fuzzy IF-THEN rules by maximizing fuzzy entropy," IEEE Trans. Fuzzy Syst., vol. 17, no. 3, pp. 556-567, Jun. 2009.
37. L. Zhang and P. Suganthan, "Random forests with ensemble of feature spaces," Pattern Recognit., vol. 47, no. 10, pp. 3429-3437, 2014.
38. L. Zhang, W.-D. Zhou, T.-T. Su, L.-C. Jiao, "Decision tree support vector machine," Int. J. Artif. Intell. Tools, vol. 16, no. 1, pp. 1-15, 2007. [Online]. Available: http://search.ebscohost.com.ezlibproxy1.ntu.edu.sg/login.aspx?direct=true&db=cph&AN=24099026&site=ehost-live
39. L. Rokach and O. Maimon, "Top-down induction of decision trees classifiers-A survey," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 35, no. 4, pp. 476-487, Nov. 2005.
40. O. L. Mangasarian and E. W. Wild, "Multisurface proximal support vector machine classification via generalized eigenvalues," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 1, pp. 69-74, Jan. 2006.
41. L.-F. Chen, H.-Y. M. Liao, M.-T. Ko, J.-C. Lin, G.-J. Yu, "A new LDA-based face recognition system which can solve the small sample size problem," Pattern Recognit., vol. 33, no. 10, pp. 1713-1726, 2000.
42. X. Jiang, "Linear subspace learning-based dimensionality reduction," IEEE Signal Process. Mag., vol. 28, no. 2, pp. 16-26, Mar. 2011.
43. L. Breiman, Classification and Regression Trees. Belmont, CA, USA: Wadsworth Int. Group, 1984.
44. L. Bottou et al., "Comparison of classifier methods: A case study in handwritten digit recognition," in Proc. 12th IEEE IAPR Int. Conf. Pattern Recognit. B, Comput. Vis. Image Process., vol. 2. Jerusalem, Israel, 1994, pp. 77-82.
45. S. Knerr, L. Personnaz, G. Dreyfus, "Single-layer learning revisited: A stepwise procedure for building and training a neural network," in Neurocomputing. Berlin, Germany: Springer, 1990, pp. 41-50.
46. J. C. Platt, N. Cristianini, J. Shawe-Taylor, "Large margin DAGs for multiclass classification," in Proc. Adv. Neural Inf. Process. Syst., vol. 12. Denver, CO, USA, 1999, pp. 547-553.
47. T. G. Dietterich and G. Bakiri, "Solving multiclass learning problems via error-correcting output codes," J. Artif. Intell. Res., vol. 2, pp. 263-286, Jan. 1995.
48. X. Jiang, B. Mandal, A. Kot, "Eigenfeature regularization and extraction in face recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 3, pp. 383-394, Mar. 2008.
49. J. Marroquin, S. Mitter, T. Poggio, "Probabilistic solution of illposed problems in computational vision," J. Amer. Stat. Assoc., vol. 82, no. 397, pp. 76-89, 1987.
50. O. L. Mangasarian and R. Meyer, "Nonlinear perturbation of linear programs," SIAM J. Control Optim., vol. 17, no. 6, pp. 745-752, 1979.
51. T. Evgeniou, M. Pontil, T. Poggio, "Regularization networks and support vector machines," Adv. Comput. Math., vol. 13, no. 1, pp. 1-50, 2000.
52. K. K. Kandaswamy et al., "AFP-Pred: A random forest approach for predicting antifreeze proteins from sequence-derived properties," J. Theor. Biol., vol. 270, no. 1, pp. 56-62, 2011.
53. K. Kandaswamy, G. Pugalenthi, M. Hazrati, K.-U. Kalies, T. Martinetz, "BLProt: Prediction of bioluminescent proteins based on support vector machine and relieff feature selection," BMC Bioinformat., vol. 12, no. 1, p. 345, 2011.
54. X. He, S. Yan, Y. Hu, P. Niyogi, H.-J. Zhang, "Face recognition using Laplacianfaces," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 3, pp. 328-340, Mar. 2005.
55. D. Cai, X. He, J. Han, H.-J. Zhang, "Orthogonal Laplacianfaces for face recognition," IEEE Trans. Image Process., vol. 15, no. 11, pp. 3608-3614, Nov. 2006.
56. J. Demšar, "Statistical comparisons of classifiers over multiple data sets," J. Mach. Learn. Res., vol. 7, pp. 1-30, Jan. 2006.
57. L. I. Kuncheva and J. J. Rodriguez, "Classifier ensembles with a random linear oracle," IEEE Trans. Knowl. Data Eng., vol. 19, no. 4, pp. 500-508, Apr. 2007.
58. P. Nemenyi, Distribution-Free Multiple Comparisons. Princeton, NJ, USA: Princeton Univ., 1963. [Online]. Available: http://books.google.com.sg/books?id=nhDMtgAACAAJ
59. S. Geman, E. Bienenstock, R. Doursat, "Neural networks and the bias/variance dilemma," Neural Comput., vol. 4, no. 1, pp. 1-58, 1992.
60. E. B. Kong and T. G. Dietterich, "Error-correcting output coding corrects bias and variance," in Proc. Int. Conf. Mach. Learn. (ICML), Tahoe City, CA, USA, 1995, pp. 313-321.
61. J. H. Friedman, "On bias, variance, 0/1-Loss, the curse-ofdimensionality," Data Min. Knowl. Disc., vol. 1, no. 1, pp. 55-77, 1997.
62. G. M. James, "Variance and bias for general loss functions," Mach. Learn., vol. 51, no. 2, pp. 115-135, 2003.
63. M. Fernández-Delgado, E. Cernadas, S. Barro, D. Amorim, "Do we need hundreds of classifiers to solve real world classification problems?" J. Mach. Learn. Res., vol. 15, pp. 3133-3181, Oct. 2014. [Online]. Available: http://jmlr.org/papers/v15/delgado14a.html