ROC analysis: Applications to the classification of biological sequences and 3D structures

Briefings in Bioinformatics

Volumn 9, Issue 3, 2008, Pages 198-209

  Sonego, Paolo ^a   Kocsor, András ^b   Pongor, Sándor ^a,c  

^a INTERNATIONAL CENTRE FOR GENETIC ENGINEERING AND BIOTECHNOLOGY   (Italy)

^b UNIVERSITY OF SZEGED   (Hungary)

^c INSTITUTE OF EXPERIMENTAL MEDICINE   (Hungary)

Author keywords

Classification; Function prediction; Performance assessment; Protein similarity searching; ROC analysis

Indexed keywords

BIOINFORMATICS; CLASSIFICATION ALGORITHM; COMPARATIVE STUDY; COMPUTER SIMULATION; DATA ANALYSIS SOFTWARE; GENE SEQUENCE; PREDICTION; PROTEIN DATABASE; PROTEIN STRUCTURE; PROTEOMICS; QUANTITATIVE ANALYSIS; RECEIVER OPERATING CHARACTERISTIC; REVIEW; SEQUENCE ANALYSIS; STATISTICAL MODEL; STRUCTURAL GENOMICS;

ALGORITHMS; MODELS, CHEMICAL; MODELS, MOLECULAR; MOLECULAR CONFORMATION; ROC CURVE; SEQUENCE ALIGNMENT; SEQUENCE ANALYSIS; SOFTWARE;

EID: 43149103140     PISSN: 14675463     EISSN: 14774054     Source Type: Journal    
DOI: 10.1093/bib/bbm064     Document Type: Review

References (61)

1. Duda RO, Hart PE, Stork DG. Pattern Classification, 2nd sub edn. New York: Wiley-Interscience, 2000.
2. Baldi P, Brunak S, Chauvin Y, et al. Assessing the accuracy of prediction algorithms for classification: An overview. Bioinformatics 2000;16:412-24.
3. Bajić VB. Comparing the success of different prediction software in sequence analysis: A review. Briefings Bioinformatics 2000;1 214-28.
4. Egan JP. Signal Detection Theory and Roc Analysis. New York: Academic Press, 1975.
5. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. Clin Chem 1993;39:561-77.
6. Lusted LB. Signal detectability and medical decisionmaking. Science 1971;171:1217-9.
7. Provost F, Fawcett T. Robust classification for imprecise environments. Mach Learn 2001;42:203-31.
8. Flach P, Blockeel H, Ferri C, et al. Decision support for data mining: Introduction to ROC analysis and its application. In: Data Mining and Decision Support: Aspects of Integration and Collaboration Dordrecht, The Netherlands: Kluwer Academic Publishers, 2003, 81-90.
9. Fawcett T. An introduction to ROC analysis. Pattern Recogn Lett 2006;27:861-74.
10. ROC Links. http://www.tech.plym.ac.uk/spmc/links/roc/roc_links.html (11 October 2007, date last accessed).
11. Fawcett T. ROC Graphs: Notes and Practical Considerations for Researchers, 2004.
12. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 1988;44:837-45.
13. Metz CE. Basic principles of ROC analysis. Semin nucl med 1978; 8:283-98.
14. McClish DK. Comparing the areas under more than two independent ROC curves. Med Decis Making 1987;7:149-55.
15. Hanley JA. The use of the 'binormal' model for parametric ROC analysis of quantitative diagnostic tests. Med Decis Making 1988;8 197-203.
16. Goddard MJ, Hinberg I. Receiver operator characteristic (ROC) curves and non-normal data: An empirical study. Stat Med 1990;9:325-37.
17. Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction. USA: Oxford University Press, 2004.
18. Obuchowski NA. ROC analysis. Am J Roentgenol 2005;184: 364-72.
19. Flach P, Wu S. Repairing concavities in ROC curves. Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI' 05). Menlo Park, CA, 2005;702-07.
20. Altschul SF, Gish W, Miller W, et al. Basic local alignment search tool. J Mol Biol 1990;215:403-10.
21. Smith TF, Waterman MS. Identification of common molecular subsequences. J Mol Biol 1981;147:195-7.
22. Holm L, Park J. DaliLite workbench for protein structure comparison. Bioinformatics 2000;16:566-7.
23. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143 29-36.
24. Liao L, Noble WS. Combining pairwise sequence similarity and support vector machines for detecting remote protein evolutionary and structural relationships. J Computat Biol 2003;10:857-68.
25. Macskassy SA, Provost F, Rosset S. ROC confidence bands: An empirical evaluation. Proceedings of the 22nd International Conference on Machine Learning, 2005, 537-44.
26. Cortes C, Mohri M. Confidence intervals for the area under the ROC curve. Advances in Neural Information Processing Systems 2004; 17.
27. Bandos AI, Rockette HE, Gur D. Resampling methods for the area under the ROC curve. Proceedings of the ICML 2006 workshop on ROC Analysis in Machine Learning, 2006.
28. Elofsson A, Sonnhammer E. A comparison of sequence and structure protein domain families as a basis for structural genomics. Bioinformatics 1999;15:480-500.
29. Kertész-Farkas A, Dhir S, Sonego, P, et al. Benchmarking protein classification algorithms via supervised cross-validation. J Biochem Biophys Methods 2007 [Epub ahead of print, 31 May 2007].
30. Streiner DL, Cairney J. What's under the ROC? An introduction to receiver operating characteristics curves. Can J Psychiatry 2007;52:121-8.
31. Eweris WJ, Grant G. Statistical Methods in Bioinformatics: An Introdudion. 2nd edn, Corr. 2nd printing edn. New York: Springer, 2005.
32. Armitage P, Berry G, Matthews JNS. Statistical Methods in Medical Research, 4 edn. Malden, MA: Blackwell Publishing Limited, 2001.
33. Westin LK. Receiver operating characteristic (ROC) analysis. Evaluating discriminance effects among ducision support systems. Sweden: Umeå universitet - Department of Computing Science, 2001.
34. Dodd LE, Pepe MS. Partial AUC estimation and regression. Biometrics 2003;59:614-23.
35. Obuchowski NA. Receiver operating characteristic curves and their use in radiology. Radiology 2003;229:3-8.
36. Jiang Y, Metz CE, Nishikawa R.M. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology 1996;201:745-50.
37. Salzberg SL. On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Min Knowl Discov 1997;1: 317-28.
38. Gribskov M, Robinson NL. Use of receiver operating characteristic (ROC) analysis to evaluate sequence matching. Comp chem 1996;20 25-33.
39. Schäffer AA, Aravind L, Madden TL, et al. Improving the accuracy of PSI-BLAST protein database searches with composition-based statistics and other refinements. Nucleic Acids Res 2001;29 2994-3005.
40. Jaakkola T, Diekhans M, Haussler D. Using the Fisher kernel method to detect remote protein homologies. Proceedings of the Seventh International Conference on Intelligent Systems for Molecular Biology 1999, 149-58.
41. Stephan C, Wesseling S, Schink T, et al. Comparison of eight computer programs for receiver-operating characteristic analysis. Clin Chem 2003;49:433-9.
42. Park SH, Goo JM, Jo C. Receiver operating characteristic (ROC) curve: practical review for radiologistis. Korean J Radiol: Official J Korean Radiol Soc 2004;5:11-8.
43. Web-based Calculator for ROC Curves. www.jrocfit.org (30 September 2007, date Last accessed).
44. R: A Language and Environment for Statstical Computing. Vienna, Austria: R Foundation for Statistical Computing, 2007.
45. Sing T, Sander O, Beerenwinkel N, et al. ROCR. http://rocr.bioinf.mpi-sb.mpg.de/ (30 September 2007, date last accessed).
46. Gentleman R, Carey V, Bates D, et al. Bioconductor: Open software development for computational biology and bioinformatics. Genome Biol 2004;5:R80.
47. Sonego P, Pacurar M, Dhir S, et al. A protein classification benchmark collection for machine learning. Nucleic Acids Res 2007;35(Suppl 1):D232-6.
48. Frank E, Hall M, Trigg L, et al. Data mining in bioinformatics using Weka. Bioinformatics 2004;20:2479-81.
49. R Weka. http://cran.r-project.org/src/contrib/Descriptions/RWeka.html (30 September 2007, date last accessed).
50. Gewehr JE, Szugat M, Zimmer R. BioWeka - extending the Weka framework for bioinformatics. Bioinformatics 2007; 23:651-3.
51. The receiver operating characteristic (ROC) curve toolkit for MATLAB®. http://theoval.sys.uea.ac.uk/matlab/default.html#roc (30 September 2007, date last accessed).
52. Wootton JC. Evaluating the effectiveness of sequence analysis algorithms using measures of relevant information. Comp chem 1997;21 191-202.
53. Tax DMJ. One-class classification; concept-learning in the absence of counter-examples. Delft, The Netherlands: Delft University of Technology, 2001.
54. Kaján L, Kertész-Farkas A, Franklin D, et al. Application of a simple likelihood ratio approximant to protein sequence classification. Bioinformatics 2006;22:2865-9.
55. Busa-Fekete R, Kertész-Farkas A, Kocsor A, et al. Balanced ROC analysis (BAROC) protocol for the evaluation of protein similarities. J Biochem Biophys Methods 2007 [Epub ahead of print, 5 July 2007].
56. Noble WS, Xuang R, Leslie C, et al. Identifying remote protein homologs by network propagation. FEBS J 2005; 272:5119-28.
57. Weston J, Kuang R, Leslie C, et al. Protein ranking by semi-supervised network propagation. BMC Bioinformatics 2006; 7 Suppl 1):S10.
58. Sam V, Tai C, Garnier J, et al. ROC and confusion analysis of structure comparison methods identify the main causes of divergence from manual protein classification. BMC Bioinformatics 2006;7:206.
59. Gibrat JF, Madej T, Bryant SH. Surprising similarities in structure comparison. Curr Opin Struct Biol 1996;6:377-85.
60. Kocsor A, Kertész-Farkas A, Kaján L, et al. Application of compression-based distance measures to protein sequence classification: a methodological study. Bioinformatics 2006;22: 407-12.
61. Leslie CS, Eskin E, Cohen A, et al. Mismatch string kernels for discriminative protein classification. Bioinformatics 2004; 20 467-76.