The Role of Balanced Training and Testing Data Sets for Binary Classifiers in Bioinformatics

PLoS ONE

Volumn 8, Issue 7, 2013, Pages

  Wei, Qiong ^a   Dunbrack, Roland L ^a  

^a FOX CHASE CANCER CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AMINO ACID;

AMINO ACID SEQUENCE; AREA UNDER THE CURVE; ARTICLE; BALANCED TRAINING; BIOINFORMATICS; CONTINGENCY TABLE; GENE MUTATION; HUMAN; HUMAN GENOME; MACHINE LEARNING; MISSENSE MUTATION; PHENOTYPE; RECEIVER OPERATING CHARACTERISTIC; TRAINING;

EID: 84879987629     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0067863     Document Type: Article

References (79)

1. Yue P, Melamud E, Moult J, (2006) SNPs3D: candidate gene and SNP selection for association studies. BMC Bioinformatics 7: 166.
2. Ferrer-Costa C, Gelpi JL, Zamakola L, Parraga I, de la Cruz X, et al. (2005) PMUT: a web-based tool for the annotation of pathological mutations on proteins. Bioinformatics 21: 3176-3178.
3. Bao L, Zhou M, Cui Y, (2005) nsSNPAnalyzer: identifying disease-associated nonsynonymous single nucleotide polymorphisms. Nucleic Acids Res 33: W480-482.
4. Bao L, Cui Y, (2005) Prediction of the phenotypic effects of non-synonymous single nucleotide polymorphisms using structural and evolutionary information. Bioinformatics 21: 2185-2190.
5. Bromberg Y, Rost B, (2007) SNAP: predict effect of non-synonymous polymorphisms on function. Nucleic Acids Res 35: 3823-3835.
6. Bromberg Y, Yachdav G, Rost B, (2008) SNAP predicts effect of mutations on protein function. Bioinformatics 24: 2397-2398.
7. Wainreb G, Ashkenazy H, Bromberg Y, Starovolsky-Shitrit A, Haliloglu T, et al. (2010) MuD: an interactive web server for the prediction of non-neutral substitutions using protein structural data. Nucleic Acids Res 38Suppl: W523-528.
8. Ramensky V, Bork P, Sunyaev S, (2002) Human non-synonymous SNPs: server and survey. Nucleic Acids Res 30: 3894-3900.
9. Dosztanyi Z, Csizmok V, Tompa P, Simon I, (2005) The pairwise energy content estimated from amino acid composition discriminates between folded and intrinsically unstructured proteins. J Mol Biol 347: 827-839.
10. Dosztanyi Z, Csizmok V, Tompa P, Simon I, (2005) IUPred: web server for the prediction of intrinsically unstructured regions of proteins based on estimated energy content. Bioinformatics 21: 3433-3434.
11. Garbuzynskiy SO, Lobanov MY, Galzitskaya OV, (2004) To be folded or to be unfolded? Protein Sci 13: 2871-2877.
12. Galzitskaya OV, Garbuzynskiy SO, Lobanov MY, (2006) FoldUnfold: web server for the prediction of disordered regions in protein chain. Bioinformatics 22: 2948-2949.
13. Yang ZR, Thomson R, McNeil P, Esnouf RM, (2005) RONN: the bio-basis function neural network technique applied to the detection of natively disordered regions in proteins. Bioinformatics 21: 3369-3376.
14. Ishida T, Kinoshita K, (2007) PrDOS: prediction of disordered protein regions from amino acid sequence. Nucleic Acids Res 35: W460-464.
15. Ward JJ, McGuffin LJ, Bryson K, Buxton BF, Jones DT, (2004) The DISOPRED server for the prediction of protein disorder. Bioinformatics 20: 2138-2139.
16. Zhang T, Faraggi E, Xue B, Dunker AK, Uversky VN, et al. (2012) SPINE-D: accurate prediction of short and long disordered regions by a single neural-network based method. J Biomol Struct Dyn 29: 799-813.
17. Hirose S, Shimizu K, Noguchi T, (2010) POODLE-I: Disordered Region Prediction by Integrating POODLE Series and Structural Information Predictors Based on a Workflow Approach. In Silico Biol 10: 185-191.
18. Kaur H, Raghava GP, (2004) A neural network method for prediction of beta-turn types in proteins using evolutionary information. Bioinformatics 20: 2751-2758.
19. Asgary M, Jahandideh S, Abdolmaleki P, Kazemnejad A, (2007) Analysis and prediction of beta-turn types using multinomial logistic regression and artificial neural network. Bioinformatics 23: 3125-3130.
20. Cai YD, Liu XJ, Xu XB, Chou KC, (2002) Support vector machines for the classification and prediction of beta-turn types. J Pept Sci 8: 297-301.
21. Sonnhammer EL, von Heijne G, Krogh A, (1998) A hidden Markov model for predicting transmembrane helices in protein sequences. Proc Int Conf Intell Syst Mol Biol 6: 175-182.
22. Worth CL, Preissner R, Blundell TL, (2011) SDM-a server for predicting effects of mutations on protein stability and malfunction. Nucleic Acids Res 39: W215-222.
23. Calabrese R, Capriotti E, Fariselli P, Martelli PL, Casadio R, (2009) Functional annotations improve the predictive score of human disease-related mutations in proteins. Hum Mutat 30: 1237-1244.
24. Masica DL, Sosnay PR, Cutting GR, Karchin R, (2012) Phenotype-optimized sequence ensembles substantially improve prediction of disease-causing mutation in cystic fibrosis. Hum Mutat 33: 1267-1274.
25. Wei Q, Wang L, Wang Q, Kruger WD, Dunbrack RL Jr, (2010) Testing computational prediction of missense mutation phenotypes: functional characterization of 204 mutations of human cystathionine beta synthase. Proteins 78: 2058-2074.
26. Chin L, Hahn WC, Getz G, Meyerson M, (2011) Making sense of cancer genomic data. Genes Dev 25: 534-555.
27. Weiss GM, Provost F (2001) The effect of class distribution on classifier learning: An empirical study. Department of Computer Science.
28. Laurikkala J (2001) Improving identification of difficult small classes by balancing class distribution. 63-66.
29. Estabrooks A, Jo T, Japkowicz N, (2004) A multiple resampling method for learning from imbalanced data sets. Computational Intelligence 20: 18-36.
30. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP, (2002) SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research 16: 321-357.
31. Liu XY, WU J, Zhou ZH (2006) Exploratory under sampling for class imbalanced learning. 965-969.
32. Han H, Wang WY, Mao BH (2005) Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning. 878-887.
33. He H, Bai Y, Garcia EA, Li S (2008) ADASYN: Adaptive synthetic sampling approach for imbalanced learning. 1322-1328.
34. Tomek I, (1976) Two modifications of CNN. IEEE Trans System, Man Cybernetics 6: 769-772.
35. Jo T, Japkowica N, (2004) Class imbalances versus small disjuncts. ACM SIGKDD Explorations Newsletter 6: 40-49.
36. Elkan C (2001) The Foundations of cost-sensitive learning. 973-978.
37. Ting KM, (2002) An instance-weighing method to induce cost-sensitive trees. IEEE Trans Knowledge and Data Eng 14: 659-665.
38. Provost F (2000) Learning with imbalanced data sets 101. AAAI workshop on imbalamced data sets.
39. He H, Garcia EA, (2009) Learning from Imbalaned data. IEEE transactions on Knowledge and Data Engineering 21: 1263-1284.
40. Capriotti E, Calabrese R, Casadio R, (2006) Predicting the insurgence of human genetic diseases associated to single point protein mutations with support vector machines and evolutionary information. Bioinformatics 22: 2729-2734.
41. Capriotti E, Fariselli P, Rossi I, Casadio R, (2008) A three-state prediction of single point mutations on protein stability changes. BMC Bioinformatics 9Suppl 2: S6.
42. Yue P, Li Z, Moult J, (2005) Loss of protein structure stability as a major causative factor in monogenic disease. J Mol Biol 353: 459-473.
43. Ferrer-Costa C, Orozco M, de la Cruz X, (2002) Characterization of disease-associated single amino acid polymorphisms in terms of sequence and structure properties. J Mol Biol 315: 771-786.
44. Ferrer-Costa C, Orozco M, de la Cruz X, (2004) Sequence-based prediction of pathological mutations. Proteins 57: 811-819.
45. Capriotti E, Arbiza L, Casadio R, Dopazo J, Dopazo H, et al. (2008) Use of estimated evolutionary strength at the codon level improves the prediction of disease-related protein mutations in humans. Hum Mutat 29: 198-204.
46. Teng S, Srivastava AK, Wang L, (2010) Sequence feature-based prediction of protein stability changes upon amino acid substitutions. BMC Genomics 11Suppl 2: S5.
47. Dobson RJ, Munroe PB, Caulfield MJ, Saqi MA, (2006) Predicting deleterious nsSNPs: an analysis of sequence and structural attributes. BMC Bioinformatics 7: 217.
48. Mottaz A, David FP, Veuthey AL, Yip YL, (2010) Easy retrieval of single amino-acid polymorphisms and phenotype information using SwissVar. Bioinformatics 26: 851-852.
49. Magrane M, Consortium U, (2011) UniProt Knowledgebase: a hub of integrated protein data. Database (Oxford) 2011: bar009.
50. Care MA, Needham CJ, Bulpitt AJ, Westhead DR, (2007) Deleterious SNP prediction: be mindful of your training data! Bioinformatics. 23: 664-672.
51. Pace HC, Kercher MA, Lu P, Markiewicz P, Miller JH, et al. (1997) Lac repressor genetic map in real space. Trends Biochem Sci 22: 334-339.
52. Alber T, Sun DP, Nye JA, Muchmore DC, Matthews BW, (1987) Temperature-sensitive mutations of bacteriophage T4 lysozyme occur at sites with low mobility and low solvent accessibility in the folded protein. Biochemistry 26: 3754-3758.
53. Altschul SF, Madden TL, Schäffer AA, Zhang J, Zhang Z, et al. (1997) Gapped BLAST and PSI-BLAST: a new generation of database programs. Nucleic Acids Research 25: 3389-3402.
54. Altschul SF, Wootton JC, Gertz EM, Agarwala R, Morgulis A, et al. (2005) Protein database searches using compositionally adjusted substitution matrices. Febs J 272: 5101-5109.
55. Velankar S, McNeil P, Mittard-Runte V, Suarez A, Barrell D, et al. (2005) E-MSD: an integrated data resource for bioinformatics. Nucleic Acids Res 33: D262-265.
56. Edgar RC, (2004) MUSCLE: a multiple sequence alignment method with reduced time and space complexity. BMC Bioinformatics 5: 113.
57. Edgar RC, (2004) MUSCLE: multiple sequence alignment with high accuracy and high throughput. Nucleic Acids Res 32: 1792-1797.
58. Pei J, Grishin NV, (2001) AL2CO: calculation of positional conservation in a protein sequence alignment. Bioinformatics 17: 700-712.
59. Hubbard SJ, Thornton JM (1993) NACCESS. London: Department of Biochemistry and Molecular Biology, University College London.
60. Sokal RR, Rohlf FJ (1995) Biometry: the principles and practice of statistics in biological research. New York: W.H. Freeman. xix, 887 p.
61. Pearson K, (1900) On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine Series 5 50: 157-175.
62. Velez DR, White BC, Motsinger AA, Bush WS, Ritchie MD, et al. (2007) A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction. Genet Epidemiol 31: 306-315.
63. Matthews BW, (1975) Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim Biophys Acta 405: 442-451.
64. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology: 29-36.
65. Walsh I, Martin AJ, Di Domenico T, Tosatto SC, (2012) ESpritz: accurate and fast prediction of protein disorder. Bioinformatics 28: 503-509.
66. Peng K, Radivojac P, Vucetic S, Dunker AK, Obradovic Z, (2006) Length-dependent prediction of protein intrinsic disorder. BMC Bioinformatics 7: 208.
67. Yue P, Moult J, (2006) Identification and analysis of deleterious human SNPs. J Mol Biol 356: 1263-1274.
68. Vapnik VN (1995) The nature of statistical learning theory. New York: Springer-Verlag New York, Inc.
69. Joachims T (1999) Making large-scale support vector machine learning practical. Cambridge: MIT Press.
70. Joachims T (2002) Learning to classify text using support vector machines: Springer.
71. Chang C-C, Lin C-J, (2011) LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2: 21-27: 27.
72. Li S, Xi L, Li J, Wang C, Lei B, et al. (2010) In silico prediction of deleterious single amino acid polymorphisms from amino acid sequence. J Comput Chem.
73. Adzhubei IA, Schmidt S, Peshkin L, Ramensky VE, Gerasimova A, et al. (2010) A method and server for predicting damaging missense mutations. Nature Methods 7: 248-249.
74. Monastyrskyy B, Fidelis K, Moult J, Tramontano A, Kryshtafovych A Evaluation of disorder predictions in CASP9. Proteins 79Suppl 10: 107-118.
75. Ward JJ, Sodhi JS, McGuffin LJ, Buxton BF, Jones DT, (2004) Prediction and functional analysis of native disorder in proteins from the three kingdoms of life. J Mol Biol 337: 635-645.
76. Deng X, Eickholt J, Cheng J, (2009) PreDisorder: ab initio sequence-based prediction of protein disordered regions. BMC Bioinformatics 10: 436.
77. Zhou Z-H. Cost-sensitive learning; 2011; Berlin. Springer-Verlag. 17-18.
78. Cui G, Wong ML, Wan X, (2012) Cost-Sensitive Learning via Priority Samling to Improve the Return on Markering and CRM Investment. Journal of Management Information Systems 29: 341-374.
79. Thai-Nghe N, Gantner Z, Schmidt-Thieme L (2010) Cost-sensitive learning methods for imbalanced data. The 2010 International Joint Conference on Neural Network. 1-8.