Reliable estimation of prediction errors for QSAR models under model uncertainty using double cross-validation

Journal of Cheminformatics

Volumn 6, Issue 1, 2014, Pages

  Baumann, Désirée ^a   Baumann, Knut ^a  

^a TECHNICAL UNIVERSITY OF BRAUNSCHWEIG   (Germany)

Author keywords

Cross validation; Double cross validation; External validation; Internal validation; Prediction error; Regression

Indexed keywords

EID: 84916603462     PISSN: None     EISSN: 17582946     Source Type: Journal    
DOI: 10.1186/s13321-014-0047-1     Document Type: Article

References (72)

1. Kubinyi H: QSAR and 3D QSAR in drug design. Part 1: methodology. Drug Discov Today 1997, 2:457-467.
2. Baumann K: Cross-validation as the objective function of variable selection. Trends Anal Chem 2003, 22:395-406.
3. Todeschini R, Consonni V: Handbook of Molecular Descriptors. Berlin: Wiley-VCH; 2002.
4. Hastie T, Tibshirani R, Friedmann J: Elements of statistical Learning: Data Mining, Inference and Prediction. 2nd edition. New York: Springer; 2009.
5. Mosteller F, Turkey J: Data Analysis, Including Statistics. In The Handbook of Social Psychology. 2nd edition. Edited by Gardner L, Eliot A. MA, USA: Springer: Addison-Wesley, Reading; 1968:109-112.
6. Stone M: Cross-validatory choice and assessment of statistical predictions. J R Stat Soc Ser B Methodol 1974, 36:111-147.
7. Ganeshanandam S, Krzanowski WJ: On selecting variables and assessing their performance in linear discriminant analysis. Aust J Stat 1989, 31:433-447.
8. Jonathan P, Krzanowski WJ, McCarthy WV: On the use of cross-validation to assess performance in multivariate prediction. Stat Comput 2000, 10:209-229.
9. Ambroise C, McLachlan GJ: Selection bias in gene extraction on the basis of microarray gene-expression data. Proc Natl Acad Sci U S A 2002, 99:6562-6566.
10. Soeria-Atmadja D, Wallman M, Björklund AK, Isaksson A, Hammerling U, Gustafsson MG: External cross-validation for unbiased evaluation of protein family detectors: application to allergens. Proteins 2005, 61:918-925.
11. Lemm S, Blankertz B, Dickhaus T, Müller KR: Introduction to machine learning for brain imaging. Neuroimage 2011, 56:387-399.
12. Varma S, Simon R: Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics 2006, 7:91.
13. Okser S, Pahikkala T, Aittokallio T: Genetic variants and their interactions in disease risk prediction - machine learning and network perspectives. BioData Min 2013, 6:5.
14. Filzmoser P, Liebmann B, Varmuza K: Repeated double cross validation. J Chemom 2009, 23:160-171.
15. Wegner JK, Fröhlich H, Zell A: Feature selection for descriptor based classification models. 1. Theory and GA-SEC algorithm. J Chem Inf Comput Sci 2004, 44:921-930.
16. Anderssen E, Dyrstad K, Westad F, Martens H: Reducing over-optimism in variable selection by cross-model validation. Chemom Intell Lab Syst 2006, 84:69-74.
17. Gidskehaug L, Anderssen E, Alsberg B: Cross model validation and optimisation of bilinear regression models. Chemom Intell Lab Syst 2008, 93:1-10.
18. Krstajic D, Buturovic LJ, Leahy DE, Thomas S: Cross-validation pitfalls when selecting and assessing regression and classification models. J Cheminform 2014, 6:1-15.
19. Tetko IV, Sushko I, Pandey AK, Zhu H, Tropsha A, Papa E, Öberg T, Todeschini R, Fourches D, Varnek A: Critical assessment of QSAR models of environmental toxicity against Tetrahymena pyriformis: Focusing on applicability domain and overfitting by variable selection. J Chem Inf Model 2008, 48:1733-1746.
20. Gütlein M, Helma C, Karwath A, Kramer S: A large-scale empirical evaluation of cross-validation and external test set validation in (Q)SAR. Mol Inform 2013, 32:516-528.
21. Zucchini W: An introduction to model selection. J Math Psychol 2000, 44:41-61.
22. Broadhurst DI, Kell DB: Statistical strategies for avoiding false discoveries in metabolomics and related experiments. Metabolomics 2006, 2:171-196.
23. Bro R, Kjeldahl K, Smilde AK, Kiers HAL: Cross-validation of component models: a critical look at current methods. Anal Bioanal Chem 2008, 390:1241-1251.
24. Reunanen J: Overfitting in making comparisons between variable selection methods. J Mach Learn Res 2003, 3:1371-1382.
25. Hawkins DM: The problem of overfitting. J Chem Inf Comput Sci 2004, 44:1-12.
26. Cawley GC, Talbot NLC: On over-fitting in model selection and subsequent selection bias in performance evaluation. J Mach Learn Res 2010, 11:2079-2107.
27. Baumann K: Chance correlation in variable subset regression: Influence of the objective function, the selection mechanism, and ensemble averaging. QSAR Comb Sci 2005, 24:1033-1046.
28. Baumann K, Stiefl N: Validation tools for variable subset regression. J Comput Aided Mol Des 2004, 18:549-562.
29. Lukacs PM, Burnham KP, Anderson DR: Model selection bias and Freedman's paradox. Ann Inst Stat Math 2009, 62:117-125.
30. Johnson JB, Omland KS: Model selection in ecology and evolution. Trends Ecol Evol 2004, 19:101-108.
31. Miller A: Subset Selection in Regression. 2nd edition. New York: Chapmann & Hall/CRC; 2002.
32. Chirico N, Gramatica P: Real external predictivity of QSAR models: how to evaluate it? Comparison of different validation criteria and proposal of using the concordance correlation coefficient. J Chem Inf Model 2011, 51:2320-2335.
33. Gramatica P: Principles of QSAR models validation: internal and external. QSAR Comb Sci 2007, 26:694-701.
34. Scior T, Medina-Franco JL, Do Q-T, Martínez-Mayorga K, Yunes Rojas JA, Bernard P: How to recognize and workaround pitfalls in QSAR studies: a critical review. Curr Med Chem 2009, 16:4297-4313.
35. Aptula AO, Jeliazkova NG, Schultz TW, Cronin MTD: The better predictive model: High q2 for the training set or low root mean square error of prediction for the test set? QSAR Comb Sci 2005, 24:385-396.
36. Tropsha A, Gramatica P, Gombar VK: The importance of being earnest: validation is the absolute essential for successful application and interpretation of QSPR models. QSAR Comb Sci 2003, 22:69-77.
37. Justice AC, Covinsky KE, Berlin JA: Assessing the generalizability of prognostic information. Ann Intern Med 1999, 130:515-524.
38. Hawkins DM, Basak SC, Mills D: Assessing model fit by cross-validation. J Chem Inf Comput Sci 2003, 43:579-586.
39. Harrell Frank E: Model Validation. In Regression Modeling Strategies: With Application to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer Science and Business Inc; 2001:90.
40. Faber N, Klaas M: Estimating the uncertainty in estimates of root mean square error of prediction: application to determining the size of an adequate test set in multivariate calibration. Chemom Intell Lab Syst 1999, 49:79-89.
41. Roecker EB: Prediction error and its estimation for subset-selected models. Technometrics 1991, 33:459-468.
42. Hawkins DM, Kraker JJ: Determinstic fallacies and model validation. J Chem Inf Model 2010, 24:188-193.
43. Efron B, Tibshirani RJ: An Introduction to the Bootstrap. New York: Chapman & Hall/CRC; 1993.
44. Eklund M, Spjuth O, Wikberg JE: The C1C2: a framework for simultaneous model selection and assessment. BMC Bioinformatics 2008, 9:360-373.
45. Breiman L: Random forests. Mach Learn 2001, 45:5-32.
46. Baumann K, Albert H, von Korff M: A systematic evaluation of the benefits and hazards of variable selection in latent variable regression. Part I. Search algorithm, theory and simulations. J Chemom 2002, 16:339-350.
47. Arlot S, Celisse A: A survey of cross-validation procedures for model selection. Stat Surv 2010, 4:40-79.
48. Browne M: Cross-validation methods. J Math Psychol 2000, 44:108-132.
49. Shao J: Linear model selection by cross-validation. J Am Stat Assoc 1993, 88:486-494.
50. Briscoe E, Feldman J: Conceptual complexity and the bias/variance tradeoff. Cognition 2011, 118:2-16.
51. Freyhult E, Prusis P, Lapinsh M, Wikberg JE, Moulton V, Gustafsson MG: Unbiased descriptor and parameter selection confirms the potential of proteochemometric modelling. BMC Bioinformatics 2005, 6:50-64.
52. Lise S, Buchan D, Pontil M, Jones DT: Predictions of hot spot residues at protein-protein interfaces using support vector machines. PLoS ONE 2011, 6:e16774.
53. Statnikov A, Wang L, Aliferis CF: A comprehensive comparison of random forests and support vector machines for microarray-based cancer classification. BMC Bioinformatics 2008, 9:319.
54. Asgharzadeh S, Pique-Regi R, Sposto R, Wang H, Yang Y, Shimada H, Matthay K, Buckley J, Ortega A, Seeger RC: Prognostic significance of gene expression profiles of metastatic neuroblastomas lacking MYCN gene amplification. J Natl Cancer Inst 2006, 98:1193-1203.
55. Lottaz C, Spang R: Molecular decomposition of complex clinical phenotypes using biologically structured analysis of microarray data. Bioinformatics 2005, 21:1971-1978.
56. Smit S, van Breemen MJ, Hoefsloot HCJ, Smilde AK, Aerts JMFG, de Koster CG: Assessing the statistical validity of proteomics based biomarkers. Anal Chim Acta 2007, 592:210-217.
57. Tibshirani R: Regression shrinkage and selection via the lasso. J R Stat Soc Ser B Methodol 1996, 58:267-288.
58. Yuan M, Lin Y: On the non-negative garrotte estimator. J R Stat Soc Ser B Statistical Methodol 2007, 69:143-161.
59. Huuskonen J: Estimation of aqueous solubility for a diverse set of organic compounds based on molecular topology. J Chem Inf Comput Sci 2000, 40:773-777.
60. Yap CW: PaDEL-descriptor: an open source software to calculate molecular descriptors and fingerprints. J Comput Chem 2011, 32:1466-1474.
61. Zuber V, Strimmer K: High-dimensional regression and variable selection using CAR scores. Stat Appl Genet Mol Biol 2010, 10:25.
62. Guha R, Jurs PC: Development of QSAR models to predict and interpret the biological activity of artemisinin analogues. J Chem Inf Comput Sci 2004, 44:1440-1449.
63. Hong H, Xie Q, Ge W, Qian F, Fang H, Shi L, Su Z, Perkins R, Tong W: Mold (2), molecular descriptors from 2D structures for chemoinformatics and toxicoinformatics. J Chem Inf Model 2008, 48:1337-1344.
64. Golbraikh A, Tropsha A: Beware of q2! J Mol Graph Model 2002, 20:269-276.
65. Christensen R: Plane Answers to Complex Questions. 2nd edition. New York: Springer; 1996.
66. Clarke K: The phantom menace: omitted variable bias in econometric research. Confl Manag Peace Sci 2005, 22:341-352.
67. Marbach R, Heise HM: Calibration modeling by partial least-squares and principal component regression and its optimization using an improved leverage correction for prediction testing. Chemom Intell Lab Syst 1990, 9:45-63.
68. Efron B, Tibshirani R: Improvements on cross-validation: the .632+ bootstrap method. J Am Stat Assoc 1997, 92:548-560.
69. Breiman L, Spector P: Submodel selection and evaluation in regression. The X-random case. Int Stat Rev 1992, 60:291-319.
70. Xu H, Caramanis C, Mannor S: Robust regression and lasso. IEEE Trans Inf Theory 2010, 56:3561-3574.
71. Bühlmann P, van de Geer SA: Statistics for High-Dimensional Data Methods, Theory and Applications. New York: Springer; 2011.
72. R Development Core Team: R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2011.