Beware of Unreliable Q2! A Comparative Study of Regression Metrics for Predictivity Assessment of QSAR Models

Journal of Chemical Information and Modeling

Volumn 56, Issue 10, 2016, Pages 1905-1913

  Todeschini, Roberto ^a   Ballabio, Davide ^a   Grisoni, Francesca ^a  

^a UNIVERSITY OF MILANO BICOCCA   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

MOLECULAR GRAPHICS;

COMPARATIVE STUDIES; CROSS VALIDATION; MATHEMATICAL PRINCIPLE; MODEL DEVELOPMENT; MODEL PREDICTIVE; PREDICTIVITY; TEST MOLECULES; VALIDATION RESULTS;

COMPUTATIONAL CHEMISTRY;

ALGORITHM; CHEMICAL MODEL; COMPUTER SIMULATION; QUANTITATIVE STRUCTURE ACTIVITY RELATION;

ALGORITHMS; COMPUTER SIMULATION; MODELS, CHEMICAL; QUANTITATIVE STRUCTURE-ACTIVITY RELATIONSHIP;

EID: 84992723917     PISSN: 15499596     EISSN: 1549960X     Source Type: Journal    
DOI: 10.1021/acs.jcim.6b00277     Document Type: Article

References (36)

1. Golbraikh, A.; Shi, L. M.; Xiao, Z.; Xiao, Y. D.; Lee, K.-H.; Tropsha, A. Rational Selection of Training and Test Sets for the Development of Validated QSAR Models J. Comput.-Aided Mol. Des. 2003, 17, 241-253 10.1023/A:1025386326946
2. Tropsha, A.; Gramatica, P.; Gombar, V. K. The Importance of Being Earnest: Validation is the Absolute Essential for Successful Application and Interpretation of QSPR Models QSAR Comb. Sci. 2003, 22, 69-77 10.1002/qsar.200390007
3. Schmuker, M.; Givehchi, A.; Schneider, G. Impact Of Different Software Implementations on the Performance of the Maxmin Method for Diverse Subset Selection Mol. Diversity 2004, 8, 421-425 10.1023/B:MODI.0000047503.82461.e8
4. Fourches, D.; Barnes, J. C.; Day, N. C.; Bradley, P.; Reed, J. Z.; Tropsha, A. Cheminformatics Analysis of Assertions Mined from Literature that Describe Drug-Induced Liver Injury in Different Species Chem. Res. Toxicol. 2010, 23, 171-183 10.1021/tx900326k
5. Hsieh, J.-H.; Yin, S.; Liu, S.; Sedyckh, A.; Dokholyan, N. V.; Tropsha, A. Combined Application of Cheminformatics- and Physical Force Field-Based Scoring Functions Improves Binding Affinity Prediction for CSAR Data Sets J. Chem. Inf. Model. 2011, 51, 2027-2035 10.1021/ci200146e
6. Colmenarejo, G. In Silico Prediction of Drug-Binding Strengths to Human Serum Albumin Med. Res. Rev. 2003, 23, 275-301 10.1002/med.10039
7. 2: Simple, Unambiguous Assessment of the Prediction Accuracy of QSAR and QSPR Models J. Chem. Inf. Model. 2015, 55, 1316-1322 10.1021/acs.jcim.5b00206
8. Grisoni, F.; Consonni, V.; Vighi, M.; Villa, S.; Todeschini, R. Expert QSAR System for Predicting the Bioconcentration Factor under the REACH Regulation Environ. Res. 2016, 148, 507 10.1016/j.envres.2016.04.032
9. Todeschini, R.; Consonni, V. Molecular Descriptors for Chemoinformatics (2 volumes); WILEY-VCH: Weinheim (Germany), 2009.
10. OECD Guidance Document on the Validation of (Quantitative) Structure-Activity Relationship [(Q)SAR] Models. OECD Series on Testing and Assessment; 2014, 69.
11. 2 Parameter for QSAR Validation J. Chem. Inf. Model. 2009, 49, 1669-1678 10.1021/ci900115y
12. Consonni, V.; Ballabio, D.; Todeschini, R. Evaluation of Model Predictive Ability by External Validation Techniques J. Chemom. 2010, 24, 194-201 10.1002/cem.1290
13. Chirico, N.; Gramatica, P. Real External Predictivity of QSAR Models: How to Evaluate it? Comparison of Different Validation Criteria and Proposal of Using the Concordance Correlation Coefficient J. Chem. Inf. Model. 2011, 51, 2320-2335 10.1021/ci200211n
14. 2m as a Metric For Validation of QSAR Models J. Chemom. 2010, 24, 22-33 10.1002/cem.1268
15. 2m Metrics for Validation of QSPR Models Chemom. Intell. Lab. Syst. 2011, 107, 194-205 10.1016/j.chemolab.2011.03.011
16. Roy, P. P.; Roy, K. On Some Aspects of Variable Selection for Partial Least Squares Regression Models QSAR Comb. Sci. 2008, 27, 302-313 10.1002/qsar.200710043
17. Gramatica, P.; Sangion, A. A Historical Excursus on the Statistical Validation Parameters for QSAR Models: A Clarification Concerning Metrics And Terminology J. Chem. Inf. Model. 2016, 56, 1127-1131 10.1021/acs.jcim.6b00088
18. Lin, L. I. A Concordance Correlation Coefficient to Evaluate Reproducibility Biometrics 1989, 45, 255-268 10.2307/2532051
19. Lin, L. I. Assay Validation Using The Concordance Correlation Coefficient Biometrics 1992, 48, 599-604 10.2307/2532314
20. Golbraikh, A.; Tropsha, A. Predictive QSAR Modeling Based on Diversity Sampling of Experimental Datasets for the Training and Test Set Selection J. Comput.-Aided Mol. Des. 2002, 16, 357-369 10.1023/A:1020869118689
21. Martin, T. M.; Harten, P.; Young, D. M.; Muratov, E. N.; Golbraikh, A.; Zhu, H.; Tropsha, A. Does Rational Selection of Training and Test Sets Improve the Outcome of QSAR Modeling? J. Chem. Inf. Model. 2012, 52, 2570-2578 10.1021/ci300338w
22. 2m″ Metrics for Judging Quality of Quantitative Structure-Activity Relationship Predictions: Emphasis on Scaling of Response Data J. Comput. Chem. 2013, 34, 1071-1082 10.1002/jcc.23231
23. 2m (Rank) Metric Incorporating Rank-Order Predictions as an Additional Tool for Validation of QSAR/QSPR Models Chemom. Intell. Lab. Syst. 2012, 118, 200-210 10.1016/j.chemolab.2012.06.004
24. Atkinson, G.; Nevill, A. Comment on The Use of Concordance Correlation to Assess the Agreement Between Twovariables Biometrics 1997, 53, 775-777
25. Roy, K.; Das, R. N.; Ambure, P.; Aher, R. B. Be Aware of Error Measures. Further Studies on Validation of Predictive QSAR Models Chemom. Intell. Lab. Syst. 2016, 152, 18-33 10.1016/j.chemolab.2016.01.008
26. Sahigara, F.; Mansouri, K.; Ballabio, D.; Mauri, A.; Consonni, V.; Todeschini, R. Comparison of Different Approaches to Define The Applicability Domain of QSAR Models Molecules 2012, 17, 4791-4810 10.3390/molecules17054791
27. Roy, K.; Mitra, I.; Kar, S.; Ojha, P. K.; Das, R. N.; Kabir, H. Comparative Studies on Some Metrics for External Validation of QSPR Models J. Chem. Inf. Model. 2012, 52, 396-408 10.1021/ci200520g
28. Chirico, N.; Gramatica, P. Real External Predictivity of QSAR Models. Part 2. New Intercomparable Thresholds for Different Validation Criteria And the Need for Scatter Plot Inspection J. Chem. Inf. Model. 2012, 52, 2044-2058 10.1021/ci300084j
29. De Lima Ribeiro, F. A.; Ferreira, M. M. C. QSPR Models of Boiling Point, Octanol-Water Partition Coefficient And Retention Time Index of Polycyclic Aromatic Hydrocarbons J. Mol. Struct.: THEOCHEM 2003, 663, 109-126 10.1016/j.theochem.2003.08.107
30. Yao, X. J.; Panaye, A.; Doucet, J. P.; Zhang, R. S.; Chen, H. F.; Liu, M. C.; Hu, Z. D.; Fan, B. T. Comparative Study of QSAR/QSPR Correlations Using Support Vector Machines, Radial Basis Function Neural Networks, And Multiple Linear Regression J. Chem. Inf. Comp. Sci. 2004, 44, 1257-1266 10.1021/ci049965i
31. Grisoni, F.; Cassotti, M.; Todeschini, R. Reshaped Sequential Replacement Algorithm for Variable Selection in QSPR Modelling: Comparison with Other Benchmark Methods J. Chemom. 2014, 28, 249-259 10.1002/cem.2603
32. Cronin, M. T.; Aptula, A. O.; Duffy, J. C.; Netzeva, T. I.; Rowe, P. H.; Valkova, I. V.; Schultz, T. W. Comparative Assessment of Methods to Develop QSARs for the Prediction of the Toxicity of Phenols to Tetrahymena Pyriformis Chemosphere 2002, 49, 1201-1221 10.1016/S0045-6535(02)00508-8
33. Votano, J. R.; Parham, M.; Hall, L. M.; Hall, L. H.; Kier, L. B.; Oloff, S.; Tropsha, A. QSAR Modeling of Human Serum Protein Binding with Several Modeling Techniques Utilizing Structure-Information Representation J. Med. Chem. 2006, 49, 7169-7181 10.1021/jm051245v
34. Tetko, I. V.; Sushko, I.; Pandey, A. K.; Zhu, H.; Tropsha, A.; Papa, E.; Õberg, T.; Todeschini, R.; Fourches, D.; Varnek, A. Critical Assessment of QSAR Models of Environmental Toxicity Against Tetrahymena Pyriformis: Focusing on Applicability Domain and Overfitting by Variable Selection J. Chem. Inf. Model. 2008, 48, 1733-1746 10.1021/ci800151m
35. Todeschini, R.; Consonni, V.; Mauri, A.; Pavan, M. Detecting ″Bad″ Regression Models: Multicriteria Fitness Functions in Regression Analysis Anal. Chim. Acta 2004, 515, 199-208 10.1016/j.aca.2003.12.010
36. Sutter, J. M.; Peterson, T. A.; Jurs, P. C. Prediction of Gas Chromatographic Retention Indices of Alkylbenzene Anal. Chim. Acta 1997, 342, 113-122 10.1016/S0003-2670(96)00578-8