A tutorial on support vector machine-based methods for classification problems in chemometrics

Analytica Chimica Acta

Volumn 665, Issue 2, 2010, Pages 129-145

  Luts, Jan ^a   Ojeda, Fabian ^a   Van de Plas Raf, R ^a   De Moor, Bart ^a   Van Huffel, Sabine ^a   Suykens, Johan A K ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Feature selection; Kernel logistic regression; Kernel based learning; Least squares support vector machine; Multi class probabilities; Support vector machine

Indexed keywords

FEATURE SELECTION; KERNEL LOGISTIC REGRESSION; KERNEL-BASED LEARNING; LEAST SQUARES SUPPORT VECTOR MACHINES; MULTI-CLASS; MULTI-CLASS PROBABILITIES;

FEATURE EXTRACTION; FUNCTION EVALUATION; GEARS; LOGISTICS; MOLECULAR BIOLOGY; REGRESSION ANALYSIS; VECTORS;

SUPPORT VECTOR MACHINES;

ARTICLE; AUTOMATED PATTERN RECOGNITION; CHEMOMETRICS; CLASSIFICATION; CLASSIFICATION ALGORITHM; CLASSIFIER; COMPUTER PREDICTION; COMPUTER PROGRAM; GENE REGULATORY NETWORK; GENOMICS; GLIOBLASTOMA; GLIOMA; HUMAN; KERNEL METHOD; MASS SPECTROMETRY; MENINGIOMA; METABOLOMICS; NEUROIMAGING; NONHUMAN; NUCLEAR MAGNETIC RESONANCE IMAGING; NUCLEAR MAGNETIC RESONANCE SPECTROSCOPY; PRIORITY JOURNAL; PROTEOMICS; RADIAL BASED FUNCTION; REGRESSION ANALYSIS; STATISTICAL ANALYSIS; SUPPORT VECTOR MACHINE; TUMOR CLASSIFICATION;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BRAIN NEOPLASMS; IMAGE INTERPRETATION, COMPUTER-ASSISTED; LEAST-SQUARES ANALYSIS; LOGISTIC MODELS; PATTERN RECOGNITION, AUTOMATED;

EID: 77952238401     PISSN: 00032670     EISSN: 18734324     Source Type: Journal    
DOI: 10.1016/j.aca.2010.03.030     Document Type: Article

References (86)

1. Alpaydin E. Introduction to Machine Learning 2004, MIT Press, Cambridge.
2. Bishop C.M. Pattern Recognition and Machine Learning 2006, Springer, New York.
3. Boser B.E., Guyon I.M., Vapnik V.N. A training algorithm for optimal margin classifiers. Proceedings of the Annual Workshop on Computational Learning Theory 1992, 144-152.
4. Vapnik V.N. The Nature of Statistical Learning Theory 1995, Springer, New York.
5. Cortes C., Vapnik V.N. Support-vector networks. Mach. Learn. 1995, 20:273-297.
6. Lukas L., Devos A., Suykens J.A.K., Vanhamme L., Howe F.A., Majos C., Moreno-Torres A., Van der Graaf M., Tate A.R., Arus C., Van Huffel S. Brain tumor classification based on long echo proton MRS signals. Artif. Intell. Med. 2004, 31:73-89.
7. Lee Y.-J., Mangasarian O.L., Wolberg W.H. Breast cancer survival and chemotherapy: a support vector machine analysis. DIMACS Ser. Discrete Math. Theor. Comput. Sci. 2000, 55:1-10.
8. Brown M.P.S., Grundy W.N., Lin D., Cristianini N., Sugnet C.W., Furey T.S., Ares M., Haussler D. Knowledge-based analysis of microarray gene expression data by using support vector machines. Proc. Natl. Acad. Sci. U. S. A. 2000, 97(1):262-267.
9. Luts J., Laudadio T., Idema A.J., Simonetti A.W., Heerschap A., Vandermeulen D., Suykens J.A.K., Van Huffel S. Nosologic imaging of the brain: segmentation and classification using MRI and MRSI. NMR Biomed. 2009, 22:374-390.
10. Holloway D.T., Kon M., DeLisi C. Machine learning for regulatory analysis and transcription factor target prediction in yeast. Syst. Synth. Biol. 2007, 1(1):25-46.
11. Burges C.J.C. A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 1998, 2:121-167.
12. Vapnik V.N. Statistical Learning Theory 1998, Wiley, New York.
13. Cristianini N., Shawe-Taylor J. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods 2000, Cambridge University Press, Cambridge.
14. Schölkopf B., Smola A.J. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond 2001, MIT Press, Cambridge.
15. Suykens J.A.K., Van Gestel T., De Brabanter J., De Moor B., Vandewalle J. Least Squares Support Vector Machines 2002, World Scientific, Singapore.
16. Abe S. Support Vector Machines for Pattern Classification 2005, Springer, New York.
17. Suykens J.A.K., Vandewalle J. Least squares support vector machine classifiers. Neural Process. Lett. 1999, 9(3):293-300.
18. Platt J.C. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers 2000, 61-74. MIT Press, Cambridge. P.J. Bartlett, B. Schölkopf, D. Schuurmans, A.J. Smola (Eds.).
19. Lin H.-T., Lin C.-J., Weng R.C. A note on Platt's probabilistic outputs for support vector machines. Mach. Learn. 2007, 68:267-276.
20. Zadrozny B., Elkan C. Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the International Conference on Knowledge Discovery and Data Mining 2002, 694-699.
21. Sollich P. Bayesian methods for support vector machines: evidence and predictive class probabilities. Mach. Learn. 2002, 46:21-52.
22. Van Gestel T., Suykens J.A.K., Lanckriet G., Lambrechts A., De Moor B., Vandewalle J. Bayesian framework for least squares support vector machine classifiers, Gaussian processes and kernel Fisher discriminant analysis. Neural Comput. 2002, 14:1115-1147.
23. MacKay D.J.C. Probable networks and plausible predictions - a review of practical Bayesian methods for supervised neural networks. Netw. Comput. Neural Syst. 1995, 6:469-505.
24. Crammer K., Singer Y. On the learnability and design of output codes for multiclass problems. Mach. Learn. 2002, 47:201-233.
25. Suykens J.A.K., Vandewalle J. Multiclass least squares support vector machines. Proceedings of the International Joint Conference on Neural Networks 1999, 900-903.
26. Dietterich T.G., Bakiri G. Solving multiclass learning problems via error-correcting output codes. J. Artif. Intell. Res. 1994, 2:263-286.
27. Allwein E.A., Schapire R.E., Singer Y. Reducing multiclass to binary: a unifying approach for margin classifiers. J. Mach. Learn. Res. 2001, 1:113-141.
28. Kreßel U.H.-G. Pairwise classification and support vector machines. Advances in Kernel Methods: Support Vector Learning 1999, 255-268. MIT Press, Cambridge. B. Schölkopf, C.J.C. Burges, A.J. Smola (Eds.).
29. Hastie T., Tibshirani R. Classification by pairwise coupling. Ann. Stat. 1998, 26:451-471.
30. Refregier P., Vallet F. Probabilistic approach for multiclass classification with neural networks. Proceedings of the International Conference on Artificial Networks 1991, 1003-1007.
31. Price D., Knerr S., Personnaz L., Dreyfus G. Pairwise neural network classifiers with probabilistic outputs. Proceedings of the Conference on Neural Information Processing Systems 1994, 1109-1116.
32. J. Friedman, Another Approach to Polychotomous classification, Technical Report, Stanford University (1996).
33. Wu T.-F., Lin C.-J., Weng R.C. Probability estimates for multi-class classification by pairwise coupling. J. Mach. Learn. Res. 2004, 5:975-1005.
34. Zhu J., Hastie T. Kernel logistic regression and the import vector machine. J. Comput. Graph. Stat. 2005, 14(1):185-205.
35. Karsmakers P., Pelckmans K., Suykens J.A.K. Multi-class kernel logistic regression: a fixed-size implementation. Proceedings of the International Joint Conference on Neural Networks 2007, 1756-1761.
36. Keerthi S.S., Duan K.B., Shevade S.K., Poo A.N. A fast dual algorithm for kernel logistic regression. Mach. Learn. 2005, 61:151-165.
37. Dietterich T.G. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput. -1923, 10:1895.
38. Wettschereck D., Aha D.W., Mohri T. A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms. Artif. Intell. Rev. 1997, 11:273-314.
39. Kohavi R., John G.H. Wrappers for feature subset selection. Artif. Intell. 1997, 97:273-324.
40. Blum A.L., Langley P. Selection of relevant features and examples in machine learning. Artif. Intell. 1997, 97:245-271.
41. John G.H., Kohavi R., Pfleger K. Irrelevant features and the subset selection problem. Proceedings of the International Conference on Machine Learning 1994, 121-129.
42. Neter J., Kutner M.H., Wasserman W., Nachtsheim C.J. Applied Linear Statistical Models 1996, McGraw-Hill, Boston.
43. Ginsberg M.L. Essentials of Artificial Intelligence 1993, Morgan Kaufmann, Palo Alto.
44. Guyon I.M., Weston J., Barnhill S., Vapnik V.N. Gene selection for cancer classification using support vector machines. Mach. Learn. 2002, 46:389-422.
45. Ojeda F., Suykens J.A.K., De Moor B. Low rank updated LS-SVM classifiers for fast variable selection. Neural Netw. 2008, 21(2-3):437-449.
46. Weston J., Mukherjee S., Chapelle O., Pontil M., Poggio T., Vapnik V.N. Feature selection for SVMs. Proceedings of the Conference on Neural Information Processing Systems 2000, 668-674.
47. Chapelle O., Vapnik V.N., Bousquet O., Mukherjee S. Choosing multiple parameters for support vector machines. Mach. Learn. 2002, 46(1-3):131-159.
48. Rasmussen C.E., Williams C.K.I. Gaussian Processes for Machine Learning 2006, MIT Press, Cambridge.
49. Weston J., Elisseeff A., Schölkopf B., Tipping M. Use of the zero-norm with linear models and kernel methods. J. Mach. Learn. Res. 2003, 3:1439-1461.
50. Tibshirani R. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B: Stat. Methodol. 1996, 58:267-288.
51. Cawley G.C. Leave-one-out cross-validation based model selection criteria for weighted LS-SVMs. Proceedings of the International Joint Conference on Neural Networks 2006, 1661-1668.
52. Cawley G.C., Talbot N.L.C. Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters. J. Mach. Learn. Res. 2007, 8:841-861.
53. Hanley J.A., McNeil B.J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982, 143:29-36.
54. Hand D.J., Till R.J. A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach. Learn. 2001, 45:171-186.
55. Vapnik V.N., Chapelle O. Bounds on error expectation for support vector machines. Neural Comput. 2000, 12(9):2013-2036.
56. Cawley G.C., Talbot N.L.C. Fast exact leave-one-out cross-validation of sparse least-squares support vector machines. Neural Netw. 2004, 17:1467-1475.
57. Hastie T., Rosset S., Tibshirani R., Zhu J. The entire regularization path for the support vector machine. J. Mach. Learn. Res. 2004, 5:1391-1415.
58. Alpaydin E. Combined 5×2cv F test for comparing supervised classification learning algorithms. Neural Comput. 1999, 11:1885-1892.
59. Nadeau C., Bengio Y. Inference for the generalization error. Mach. Learn. 2003, 52:239-281.
60. Bouckaert R.R. Choosing between two learning algorithms based on calibrated tests. Proceedings of the International Conference on Machine Learning 2003, 51-58.
61. Bouckaert R.R., Frank E. Evaluating the replicability of significance tests for comparing learning algorithms. Lect. Notes Comput. Sci. 2004, 3056:3-12.
62. Y. Grandvalet, Y. Bengio, Hypothesis Testing for Cross-Validation, Technical Report 1285, Departement d'Informatique et Recherche Operationnelle, Universite de Montreal (2006).
63. Yildiz O.T., Alpaydin E. Ordering and finding the best of K>2 supervised learning algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 2006, 28:392-402.
64. Demsar J. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 2006, 7:1-30.
65. Garcia S., Herrera F. An extension on " Statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons. J. Mach. Learn. Res. 2008, 9:2677-2694.
66. Leek J.T., Storey J.D. A general framework for multiple testing dependence. Proc. Natl. Acad. Sci. U. S. A. 2008, 105:18718-18723.
67. Cohen J. The earth is round (p<.05). Am. Psychol. 1994, 49:997-1003.
68. Sterne J.A.C., Smith G.D. Sifting the evidence - what's wrong with significance tests?. Br. Med. J. 2001, 322:226-231.
69. C.-C. Chang, C.-J. Lin, LIBSVM: A Library for Support Vector Machines, National Taiwan University, (2001). http://www.csie.ntu.edu.tw/~cjlin/libsvm/.
70. Karatzoglou A., Smola A., Hornik K., Zeileis A. kernlab - an S4 package for kernel methods in R. J. Stat. Softw. 2004, 11(9):1-20. http://www.jstatsoft.org/v11/i09/.
71. P. Sollich, C. Gold, Bayesian Support Vector Machine Hyperparameter Tuning, King's College London, (2005). http://www.mth.kcl.ac.uk/~psollich/BayesSVM/.
72. K. Pelckmans, J.A.K. Suykens, T. Van Gestel, J. De Brabanter, L. Lukas, B. Hamers, B. De Moor, J. Vandewalle, LS-SVMlab: A Matlab/C Toolbox for Least Squares Support Vector Machines, ESAT-SISTA, Katholieke Univiversiteit Leuven, Belgium, (2002). http://www.esat.kuleuven.be/sista/lssvmlab/.
73. Joachims T. Learning to Classify Text Using Support Vector Machines 2002, Kluwer.
74. Gasch A.P., Spellman P.T., Kao C.M., Carmel-Harel O., Eisen M.B., Storz G., Botstein D., Brown P.O. Genomic expression programs in the response of yeast cells to environmental changes. Mol. Biol. Cell 2000, 11(12):4241-4257.
75. Harbison C.T., Gordon D.B., Lee T.I., Rinaldi N.J., Macisaac K.D., Danford T.W., Hannett N.M., Tagne J.-B., Reynolds D.B., Yoo J., Jennings E.G., Zeitlinger J., Pokholok D.K., Kellis M., Rolfe P.A., Takusagawa K.T., Lander E.S., Gifford D.K., Fraenkel E., Young R.A. Transcriptional regulatory code of a eukaryotic genome. Nature 2004, 431(7004):99-104.
76. Lemmens K., Dhollander T., De Bie T., Monsieurs P., Engelen K., Smets B., Winderickx J., De Moor B., Marchal K. Inferring transcriptional modules from ChIP-chip, motif and microarray data. Genome Biol. 2006, 7(5):R37.
77. Qian J., Lin J., Luscombe N.M., Yu H., Gerstein M. Prediction of regulatory networks: genome-wide identification of transcription factor targets from gene expression data. Bioinformatics 2003, 19(15):1917-1926.
78. Mordelet F., Vert J.-P. SIRENE: supervised inference of regulatory networks. Bioinformatics 2008, 24(16):76-i82.
79. An S., Liu W., Venkatesh S. Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression. Pattern Recogn. 2007, 40(8):2154-2162.
80. Hu Z., Mellor J., Wu J., DeLisi C. VisANT: an online visualization and analysis tool for biological interaction data. BMC Bioinformatics 2004, 5(1):17.
81. Stoeckli M., Chaurand P., Hallahan D.E., Caprioli R.M. Imaging mass spectrometry: a new technology for the analysis of protein expression in mammalian tissues. Nat. Med. 2001, 7(4):493-496.
82. McDonnell L.A., Heeren R.M.A. Imaging mass spectrometry. Mass Spectrom. Rev. 2007, 26(4):606-643.
83. Van de Plas R., Ojeda F., Dewil M., Van Den Bosch L., De Moor B., Waelkens E. Prospective exploration of biochemical tissue composition via imaging mass spectrometry guided by principal component analysis. Proceedings of the Pacific Symposium on Biocomputing 12 2007, 458-469.
84. Van de Plas R., De Moor B., Waelkens E. Imaging mass spectrometry based exploration of biochemical tissue composition using peak intensity weighted PCA. IEEE/NIH Life Science Systems and Applications Workshop 2007.
85. Van de Plas R., Pelckmans K., De Moor B., Waelkens E. Spatial querying of imaging mass spectrometry data: a nonnegative least squares approach. Neural Information Processing Systems Workshop on Machine Learning in Computational Biology 2007.
86. Hanselmann M., Kothe U., Kirchner M., Renard B.Y., Amstalden E.R., Glunde K., Heeren R.M.A., Hamprecht F.A. Toward digital staining using imaging mass spectrometry and random forests. J. Proteome Res. 2009, 8(7):3558-3567.