Tree ensembles for predicting structured outputs

Pattern Recognition

Volumn 46, Issue 3, 2013, Pages 817-833

  Kocev, Dragi ^a   Vens, Celine ^b   Struyf, Jan ^b   Džeroski, Sašo ^a,c  

^a JO EF STEFAN INSTITUTE   (Slovenia)

^b UNIVERSITY OF LEUVEN   (Belgium)

^c Centre of Excellence for Integrated Approaches in Chemistry and Biology of Proteins CipKeBiP   (Slovenia)

Author keywords

Ensemble methods; Hierarchical multi label classification; Multi target classification; Multi target regression; Predictive clustering trees; Structured outputs

Indexed keywords

CLUSTERING TREES; ENSEMBLE METHODS; MULTI-LABEL; MULTITARGET; STRUCTURED OUTPUTS;

DECISION TREES;

FORECASTING;

EID: 84870255848     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2012.09.023     Document Type: Article

References (86)

1. Q. Yang, and X. Wu 10 challenging problems in data mining research International Journal of Information Technology & Decision Making 5 4 2006 597 604
2. H.-P. Kriegel, K. Borgwardt, P. Kröger, A. Pryakhin, M. Schubert, and A. Zimek Future trends in data mining Data Mining and Knowledge Discovery 15 2007 87 97
3. C. Silla, and A. Freitas A survey of hierarchical classification across different application domains Data Mining and Knowledge Discovery 22 1-2 2011 31 72
4. G.H. Bakir, T. Hofmann, B. Schölkopf, A.J. Smola, B. Taskar, S.V.N. Vishwanathan, Predicting Structured Data, Neural Information Processing, The MIT Press, 2007.
5. H. Blockeel, L.D. Raedt, J. Ramon, Top-down induction of clustering trees, in: Proceedings of the 15th International Conference on Machine Learning, Morgan Kaufmann, 1998, pp. 55-63.
6. J. Struyf, S. Džeroski, Constraint based induction of multi-objective regression trees, in: Proceedings of the 4th International Workshop on Knowledge Discovery in Inductive Databases KDID, Lecture Notes in Computer Science, vol. 3933, Springer, 2006, pp. 222-233.
7. D. Kocev, C. Vens, J. Struyf, S. Džeroski, Ensembles of multi-objective decision trees, in: ECML '07: Proceedings of the 18th European Conference on Machine Learning, Lecture Notes in Computer Science, vol. 4701, Springer, 2007, pp. 624-631.
8. C. Vens, J. Struyf, L. Schietgat, S. Džeroski, and H. Blockeel Decision trees for hierarchical multi-label classification Machine Learning 73 2 2008 185 214
9. I. Slavkov, V. Gjorgjioski, J. Struyf, and S. Džeroski Finding explained groups of time-course gene expression profiles with predictive clustering trees Molecular Biosystems 6 4 2010 729 740
10. L. Breiman, J. Friedman, R. Olshen, and C.J. Stone Classification and Regression Trees 1984 Chapman & Hall/CRC
11. L. Breiman Bagging predictors Machine Learning 24 2 1996 123 140
12. G. Seni, and J.F. Elder Ensemble Methods in Data Mining Improving Accuracy through Combining Predictions 2010 Morgan and Claypool Publishers
13. E. Bauer, and R. Kohavi An empirical comparison of voting classification algorithms bagging, boosting, and variants Machine Learning 36 1 1999 105 139
14. H. Blockeel, L. Schietgat, J. Struyf, S. Džeroski, A. Clare, Decision trees for hierarchical multilabel classification: a case study in functional genomics, in: Knowledge Discovery in Databases: PKDD 2006, Lecture Notes in Computer Science, vol. 4213, Springer, 2006, pp. 18-29.
15. L.K. Hansen, and P. Salamon Neural network ensembles IEEE Transactions on Pattern Analysis and Machine Intelligence 12 10 1990 993 1001
16. A. Verikas, A. Gelzinis, and M. Bacauskiene Mining data with random forests a survey and results of new tests Pattern Recognition 44 2 2011 330 349
17. L. Schietgat, C. Vens, J. Struyf, H. Blockeel, D. Kocev, and S. Džeroski Predicting gene function using hierarchical multi-label decision tree ensembles BMC Bioinformatics 11 2 2010 1 14
18. S. Džeroski, V. Gjorgjioski, I. Slavkov, J. Struyf, Analysis of time series data with predictive clustering trees, in: Knowledge Discovery in Inductive Databases, 5th International Workshop, KDID 2006, Revised Selected and Invited Papers, Lecture Notes in Computer Science, vol. 4747, Springer, 2007, pp. 63-80.
19. D. Demšar, S. Džeroski, T. Larsen, J. Struyf, J. Axelsen, M. Bruns-Pedersen, and P.H. Krogh Using multi-objective classification to model communities of soil Ecological Modelling 191 1 2006 131 143
20. G. Tsoumakas, and I. Katakis Multi label classification an overview International Journal of Data Warehouse and Mining 3 3 2007 1 13
21. P. Langley, Elements of Machine Learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1996.
22. H. Blockeel, and J. Struyf Efficient algorithms for decision tree cross-validation Journal of Machine Learning Research 3 2002 621 650
23. R.J. Quinlan C4.5 Programs for Machine Learning 1st ed. 1993 Morgan Kaufmann
24. A. Clare, Machine Learning and Data Mining for Yeast Functional Genomics, Ph.D. Thesis, University of Wales Aberystwyth, Aberystwyth, Wales, UK, 2003.
25. L. Kuncheva Combining Pattern Classifiers Methods and Algorithms 2004 Wiley-Interscience
26. H. Zouari, L. Heutte, and Y. Lecourtier Controlling the diversity in classifier ensembles through a measure of agreement Pattern Recognition 38 11 2005 2195 2199
27. Y. Freund, R.E. Schapire, Experiments with a new boosting algorithm, in: Proceedings of the 13th International Conference on Machine Learning - ICML, Morgan Kaufman, 1996, pp. 148-156.
28. L. Breiman Using iterated bagging to debias regressions Machine Learning 45 3 2001 261 277
29. L. Breiman Random forests Machine Learning 45 1 2001 5 32
30. T.K. Ho The random subspace method for constructing decision forests IEEE Transactions on Pattern Analysis and Machine Intelligence 20 8 1998 832 844
31. T.G. Dietterich, Ensemble methods in machine learning, in: Proceedings of the 1st International Workshop on Multiple Classifier Systems, Lecture Notes in Computer Science, vol. 1857, Springer, 2000, pp. 1-15.
32. I.H. Witten, and E. Frank Data Mining Practical Machine Learning Tools and Techniques 2005 Morgan Kaufmann
33. Intel ® SSE4 Programming Reference, D91561-003 Edition, 2007.
34. T. Gärtner, and S. Vembu On structured output training hard cases and an efficient alternative Machine Learning 76 2009 227 242
35. C. Kampichler, S. Džeroski, and R. Wieland Application of machine learning techniques to the analysis of soil ecological data bases relationships between habitat features and Collembolan community characteristics Soil Biology and Biochemistry 32 2 2000 197 209
36. A. Karalič, First Order Regression, Ph.D. Thesis, Faculty of Computer Science, University of Ljubljana, Ljubljana, Slovenia, 1995.
37. D. Stojanova, P. Panov, V. Gjorgjioski, A. Kobler, and S. Džeroski Estimating vegetation height and canopy cover from remotely sensed data with machine learning Ecological Informatics 5 4 2010 256 266
38. D. Stojanova, Estimating Forest Properties from Remotely Sensed Data by using Machine Learning, Master's Thesis, Jožef Stefan International Postgraduate School, Ljubljana, Slovenia, 2009.
39. D. Demšar, M. Debeljak, S. Džeroski, C. Lavigne, Modelling pollen dispersal of genetically modified oilseed rape within the field, in: The Annual Meeting of the Ecological Society of America, 2005.
40. A. Asuncion, D. Newman, UCI - Machine Learning Repository 〈 http://www.ics.uci.edu/∼mlearn/MLRepository.html 〉, 2007.
41. V. Gjorgjioski, S. Džeroski, M. White, Clustering Analysis of Vegetation Data, Technical Report 10065, Jožef Stefan Institute, 2008.
42. D. Kocev, S. Džeroski, M. White, G. Newell, and P. Griffioen Using single- and multi-target regression trees and ensembles to model a compound index of vegetation condition Ecological Modelling 220 8 2009 1159 1168
43. H. Blockeel, S. Džeroski, J. Grbović, Simultaneous prediction of multiple chemical parameters of river water quality with TILDE, in: Proceedings of the 3rd European Conference on PKDD, LNAI 1704, Springer, 1999, pp. 32-40.
44. S. Džeroski, D. Demšar, and J. Grbović Predicting chemical parameters of river water quality from bioindicator data Applied Intelligence 13 1 2000 7 17
45. K. Trohidis, G. Tsoumakas, G. Kalliris, I. Vlahavas, Multilabel classification of music into emotions, in: Proceedings of the 9th International Conference on Music Information Retrieval (ISMIR 2008), 2008, pp. 325-330.
46. M. Skrjanc, M. Grobelnik, and D. Zupanic Insights offered by data-mining when analyzing media space data Informatica (Slovenia) 25 3 2001 357 363
47. M. Boutell, J. Luo, X. Shen, and C. Brown Learning multi-label scene classification Pattern Recognition 37 9 2004 1757 1771
48. A. Elisseeff, J. Weston, A kernel method for multi-labelled classification, in: Advances in Neural Information Processing Systems, vol. 14, MIT Press, 2001, pp. 681-687.
49. I. Dimitrovski, D. Kocev, S. Loskovska, S. Džeroski, Hierchical annotation of medical images, in: Proceedings of the 11th International Multiconference - Information Society (IS 2008), IJS, Ljubljana, 2008, pp. 174-181.
50. ADIAC, Automatic Diatom Identification and Classification 〈 http://rbg-web2.rbge.org.uk/ADIAC/ 〉, 2008.
51. B. Klimt, Y. Yang, The enron corpus: a new dataset for email classification research, in: ECML '04: Proceedings of the 18th European Conference on Machine Learning, Lecture Notes in Computer Science, vol. 3201, Springer, 2004, pp. 217-226.
52. D.D. Lewis, Y. Yang, T.G. Rose, and F. Li RCV1 a new benchmark collection for text categorization research Journal of Machine Learning Research 5 2004 361 397
53. J. Rousu, C. Saunders, S. Szedmak, and J. Shawe-Taylor Kernel-based learning of hierarchical multilabel classification models Journal of Machine Learning Research 7 2006 1601 1626
54. Z. Barutcuoglu, R.E. Schapire, and O.G. Troyanskaya Hierarchical multi-label prediction of gene function Bioinformatics 22 7 2006 830 836
55. M. Sokolova, and G. Lapalme A systematic analysis of performance measures for classification tasks Information Processing & Management 45 4 2009 427 437
56. J. Demšar Statistical comparisons of classifiers over multiple data sets Journal of Machine Learning Research 7 2006 1 30
57. M. Friedman A comparison of alternative tests of significance for the problem of m rankings Annals of Mathematical Statistics 11 1940 86 92
58. R.L. Iman, and J.M. Davenport Approximations of the critical region of the Friedman statistic Communications in Statistics - Theory and Methods 9 6 1980 571 595
59. P.B. Nemenyi, Distribution-free Multiple Comparisons, Ph.D. Thesis, Princeton University, Princeton, NY, USA, 1963.
60. R. Caruana Multitask learning Machine Learning 28 1997 41 75
61. S. Thrun, and L. Pratt Learning to Learn 1998 Kluwer Academic Publishers
62. J. Baxter A model of inductive bias learning Journal of Artificial Intelligence Research 12 2000 149 198
63. T. Evgeniou, C.A. Micchelli, and M. Pontil Learning multiple tasks with kernel methods Journal of Machine Learning Research 6 2005 615 637
64. A. Caponnetto, C.A. Micchelli, M. Pontil, and Y. Ying Universal multi-task kernels Journal of Machine Learning Research 9 2008 1615 1646
65. S. Ben-David, and R.S. Borbely A notion of task relatedness yielding provable multiple-task learning guarantees Machine Learning 73 3 2008 273 287
66. R.K. Ando, T. Zhang, and P. Bartlett A framework for learning predictive structures from multiple tasks and unlabeled data Journal of Machine Learning Research 6 2005 1817 1853
67. W.H. Greene Econometric Analysis 6th ed. 2007 Prentice Hall
68. Y. Chang, J. Bai, K. Zhou, G.-R. Xue, H. Zha, and Z. Zheng Multi-task learning to rank for web search Pattern Recognition Letters 33 2 2012 173 181
69. D. Hernández-Lobato, J. Hernández-Lobato, T. Helleputte, P. Dupont, Expectation propagation for Bayesian multi-task feature selection, in: ECML '10: Proceedings of the 21st European Conference on Machine Learning, Lecture Notes in Computer Science, vol. 6321, Springer, 2010, pp. 522-537.
70. P.J. Brown, and J.V. Zidek Adaptive multivariate ridge regression The Annals of Statistics 8 1 1980 64 74
71. L. Breiman, and J. Friedman Predicting multivariate responses in multiple linear regression Journal of the Royal Statistical Society Series B (Statistical Methodology) 59 1 1997 3 54
72. B. Bakker, and T. Heskes Task clustering and gating for Bayesian multitask learning Journal of Machine Learning Research 4 2003 83 99
73. A. Wilson, A. Fern, S. Ray, P. Tadepalli, Multi-task reinforcement learning: a hierarchical Bayesian approach, in: ICML '07: Proceedings of the 24th International Conference on Machine Learning, ACM, 2007, pp. 1015-1022.
74. G. Liu, Z. Lin, and Y. Yu Multi-output regression on the output manifold Pattern Recognition 42 11 2009 2737 2743
75. A. Argyriou, T. Evgeniou, and M. Pontil Convex multi-task feature learning Machine Learning 73 2008 243 272
76. C.A. Micchelli, M. Pontil, Kernels for multi-task learning, in: Advances in Neural Information Processing Systems 17 - Proceedings of the 2004 Conference, 2004, pp. 921-928.
77. F. Cai, V. Cherkassky, SVM+ regression and multi-task learning, in: International Joint Conference on Neural Networks (IJCNN), 2009, pp. 418-424.
78. Y. Chen, and D. Xu Global protein function annotation through mining genome-scale data in yeast Saccharomyces cerevisiae Nucleic Acids Research 32 21 2004 6414 6424
79. W. Tian, L.V. Zhang, M. Taşan, F.D. Gibbons, O.D. King, J. Park, Z. Wunderlich, J.M. Cherry, and F.P. Roth Combining guilt-by-association and guilt-by-profiling to predict Saccharomyces cerevisiae gene function Genome Biology 9 S1 2008 S7+
80. G. Obozinski, G. Lanckriet, C. Grant, M.I. Jordan, and W.S. Noble Consistent probabilistic outputs for protein function prediction Genome Biology 9 S1 2008 S6+
81. Y. Guan, C.L. Myers, D.C. Hess, Z. Barutcuoglu, A.A. Caudy, and O.G. Troyanskaya Predicting gene function in a hierarchical context with an ensemble of classifiers Genome Biology 9 S1 2008 S3+
82. G. Valentini, M. Re, Weighted true path rule: a multilabel hierarchical algorithm for gene function prediction, in: Proceedings of the 1st International Workshop on Learning from Multi-Label Data, 2009, pp. 133-146.
83. J. Díez, J.J. del Coz, and A. Bahamonde A semi-dependent decomposition approach to learn hierarchical classifiers Pattern Recognition 43 11 2010 3795 3804
84. P. Geurts, L. Wehenkel, F. D?Alché-Buc, Kernelizing the output of tree-based methods, in: ICML '06: Proceedings of the 23rd International Conference on Machine Learning, ACM, 2006, pp. 345-352.
85. H. Blockeel, M. Bruynooghe, S. Džeroski, J. Ramon, J. Struyf, Hierarchical multi-classification, in: KDD-2002 Workshop Notes: MRDM 2002, Workshop on Multi-relational Data Mining, 2002, pp. 21-35.
86. Džeroski S. Towards a general framework for data mining. in: Džzeroski S., Struyf J. (Eds.). Knowledge Discovery in Inductive Databases, 5th International Workshop, KDID 2006, Revised Selected and Invited Papers, vol. 4747, 2007, pp. 259-300