Validation of computational methods in genomics

Current Genomics

Volumn 8, Issue 1, 2007, Pages 1-19

  Dougherty, Edward R ^a,b,c   Hua, Jianping ^b   Bittner, Michael L ^b  

^a Texas AandM University   (United States)

^b TRANSLATIONAL GENOMICS RESEARCH INSTITUTE   (United States)

^c UNIVERSITY OF TEXAS MD ANDERSON CANCER CENTER   (United States)

Author keywords

Classifier models; Clustering; Confidence bounds; Linear classifier separating; Sequential forward selection (SFS)

Indexed keywords

BETA1 INTEGRIN; CD36 ANTIGEN; CD5 ANTIGEN; RNA;

ACCURACY; ANALYTICAL ERROR; B CELL LYMPHOMA; BAYES THEOREM; BIOINFORMATICS; BOOTSTRAPPING; BREAST TUMOR; CLASSIFICATION ALGORITHM; CLASSIFIER; DISCRIMINANT ANALYSIS; DNA MICROARRAY; GENE EXPRESSION; GENOMICS; HIGH THROUGHPUT SCREENING; HUMAN; LARGE CELL LYMPHOMA; LEARNING ALGORITHM; MACHINE LEARNING; MATHEMATICAL MODEL; PERFORMANCE; PREDICTION; REVIEW; SAMPLE SIZE; VALIDATION PROCESS;

EID: 33947684240     PISSN: 13892029     EISSN: None     Source Type: Journal    
DOI: 10.2174/138920207780076956     Document Type: Review

References (59)

1. Garrod, A.E. The incidence of alkaptonuria: a study in chemical individuality. Lancet 1902, 2: 1616-1620.
2. Garrod, A.E. Inborn errors of metabolism. London: H. Frowde, Hodder & Stoughton, 1923.
3. Lejeune, J., Gautier M., Turpin R. Study of somatic chromosomes from 9 mongoloid children. C. R. Hebd. Seances Acad. Sci. 1959, 248: 1721-1722.
4. Nowell, P.C., Hungerford D.A. A minute chromosome in human chronic granulocytic leukemia. Science 1960, 132: 1497.
5. Goyette, M.C., Cho, K., Fasching, C.L., Levy, D.B., Kinzler, K.W., Paraskeva, C., Vogelstein B., Stanbridge, E.J. Progression of colorectal cancer is associated with multiple tumor suppressor gene defects but inhibition of tumorigenicity is accomplished by correction of any single defect via chromosome transfer. Mol. Cell. Biol. 1992, 12: 1387-1395.
6. Perou, C.M., Sorlie, T., Eisen, M.B., van de Rijn, M., Jeffrey, S.S., Rees, C.A., Pollack, J.R., Ross, D.T., Johnsen, H., Akslen, L.A., Fluge, O., Pergamenschikov, A., Williams, C., Zhu, S.X., Lonning, P.E., Borresen-Dale, A.L., Brown P.O., Botstein, D. Molecular portraits of human breast tumours. Nature 2000, 406: 747-752.
7. van 't Veer, L. J., Dai, H., van de Vijver, M.J., He, Y.D., Hart, A.A., Bernards, R., Friend, S.H. Expression profiling predicts outcome in breast cancer. Breast Cancer Res. 2003, 5: 57-58.
8. Gidalevitz, T., Ben-Zvi, A., Ho, K.H., Brignull, H.R., Morimoto, R.I. Progressive disruption of cellular protein folding in models of polyglutamine diseases. Science 2006, 311: 1471-1474.
9. Kaneta, Y., Kagami, Y., Tsunoda, T., Ohno, R., Nakamura, Y., Katagiri, T. Genome-wide analysis of gene-expression profiles in chronic myeloid leukemia cells using a cDNA microarray. Int. J. Oncol. 2003, 23: 681-691.
10. Mao, R., Wang, X., Spitznagel, E. L., Jr., Frelin, L.P., Ting, J.C., Ding, H., Kim, J.W., Ruczinski, I., Downey, T.J., Pevsner, J. Primary and secondary transcriptional effects in the developing human Down syndrome brain and heart. Genome Biol. 2005, 6: R107.
11. Alizadeh, A.A., Eisen, M.B., Davis, R.E., Ma, C., Lossos, I.S., Rosenwald, A., Boldrick, J.C., Sabet, H., Tran, T., Yu, X., Powell, J.I., Yang, L., Marti, G.E., Moore, T., Hudson, J., Jr., Lu, L., Lewis, D.B., Tibshirani, R., Sherlock, G., Chan, W.C., Greiner, T.C., Weisenburger, D.D., Armitage, J.O., Warnke, R., Levy, R., Wilson, W., Grever, M.R., Byrd, J.C., Botstein, D., Brown, P.O., Staudt, L.M. Distinct types of diffuse large B-cell lymphoma identified by gene expression profiling. Nature 2000, 403: 503-511.
12. Khan, J., Wei, J.S., Ringner, M., Saal, L.H., Ladanyi, M., Westermann, F., Berthold, F., Schwab, M., Antonescu, C.R., Peterson, C., Meltzer, P.S. Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks. Nat. Med. 2001, 7: 673-679.
13. Hedenfalk, I., Duggan, D., Chen, Y., Radmacher, M., Bittner, M., Simon, R., Meltzer, P., Gusterson, B., Esteller, M., Kallioniemi, O.P., Wilfond, B., Borg, A., Trent, J., Raffeld, M., Yakhini, Z., Ben-Dor, A., Dougherty, E., Kononen, J., Bubendorf, L., Fehrle, W., Pittaluga, S., Gruvberger, S., Loman, N., Johannsson, O., Olsson, H., Sauter, G. Gene-expression profiles in hereditary breast cancer. N. Engl. J. Med. 2001, 344: 539-548.
14. Monti, S., Savage, K.J., Kutok, J.L., Feuerhake, F., Kurtin, P., Mihm, M., Wu, B., Pasqualucci, L., Neuberg, D., Aguiar, R.C., DalCin, P., Ladd, C., Pinkus, G.S., Salles, G., Harris, N.L., Dalla-Favera, R., Habermann, T. M., Aster, J.C., Golub, T.R., Shipp, M.A. Molecular profiling of diffuse large B-cell lymphoma identifies robust subtypes including one characterized by host inflammatory response. Blood 2005, 105: 1851-1861.
15. Gruvberger, S., Ringner, M., Chen, Y., Panavally, S., Saal, L.H., Borg, A., Ferno, M., Peterson, C., Meltzer, P.S. Estrogen receptor status in breast cancer is associated with remarkably distinct gene expression patterns. Cancer Res. 2001, 61: 5979-5984.
16. Allander, S.V., Nupponen, N.N., Ringner, M., Hostetter, G., Maher, G.W., Goldberger, N., Chen, Y., Carpten, J., Elkahloun, A.G., Meltzer, P.S. Gastrointestinal stromal tumors with KIT mutations exhibit a remarkably homogeneous gene expression profile. Cancer Res. 2001, 61: 8624-8628.
17. Dave, S. S., Wright, G., Tan, B., Rosenwald, A., Gascoyne, R.D., Chan, W.C., Fisher, R.I., Braziel, R.M., Rimsza, L.M., Grogan, T.M., Miller, T.P., LeBlanc, M., Greiner, T.C., Weisenburger, D.D., Lynch, J.C., Vose, J., Armitage, J.O., Smeland, E.B., Kvaloy, S., Holte, H., Delabie, J., Connors, J.M., Lansdorp, P.M., Ouyang, Q., Lister, T.A., Davies, A.J., Norton, A.J., Muller-Hermelink, H.K., Ott, G., Campo, E., Montserrat, E., Wilson, W.H., Jaffe, E.S., Simon, R., Yang, L., Powell, J., Zhao, H., Goldschmidt, N., Chiorazzi, M., Staudt, L.M. Prediction of survival in follicular lymphoma based on molecular features of tumor-infiltrating immune cells. N. Engl. J. Med. 2004, 351: 2159-2169.
18. Dougherty, E.R. Small Sample Issues for Microarray-Based Classification. Comparative and Functional Genomics 2001, 2: 28-34.
19. Vapnik, V.N., Chervonenkis, A. On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities. Theory of Probability and its Applications 1971, 16: 264-280.
20. Braga-Neto, U.M., Dougherty, E.R. Is Cross-Validation Valid for Small-Sample Microarray Classification. Bioinformatics 2004, 20: 374-380.
21. Molinaro, A.M., Simon, R., Pfeiffer, R.M. Prediction Error Estimation: A Comparison of Resampling Methods. Bioinformatics 2005, 21: 3301-3307.
22. Sima, C., Attoor, S., Braga-Neto, U.M., Lowey, J., Suh, E., Dougherty, E.R. Impact of Error Estimation on Feature-Selection Algorithms. Pattern Recognition 2005, 38: 2472-2482.
23. Sima, C., Dougherty, E.R. What Should Be Expected from Feature Selection in Small-Sample Settings. Bioinformatics 2006, 22: 2430-2436.
24. Zhou, X., Mao, K.Z. The Ties Problem Resulting from Counting-Based Error Estimators and Its Impact on Gene Selection Algorithms. Bioinformatics 2006, 22:2507-2515.
25. Dougherty, E.R., Braga-Neto, U., Epistemology of Computational Biology: Mathematical Models and Experimental Prediction as the Basis of Their Validity. Biol. Syst. 2006, 14: 65-90.
26. Kauffman, S.A. Metabolic stability and epigenesis in randomly constructed genetic nets. Theoretical Biol. 1969, 22: 437-467.
27. Kauffman, S.A. Homeostasis and differentiation in random genetic control networks. Nature 1969, 224: 177-178.
28. Mehta, T., Murat, T., Allison, D.B. Towards sound epistemological foundations of statistical methods for high-dimensional biology. Nat. Genet. 2004, 36: 943-947.
29. van de Vijver, M.J., He, Y.D., van't Veer, L.J., Dai, H., Hart, A.A., Voskuil, D.W., Schreiber, G.J., Peterse, J.L., Roberts, C., Marton, M.J., Parrish, M., Atsma, D., Witteveen, A., Glas, A., Delahaye, L., van der Velde, T., Bartelink, H., Rodenhuis, S., Rutgers, E.T., Friend, S.H., Bernards, R. Gene Expression Signature as a Predictor of Survival in Breast Cancer. N. Engl. J. Med. 2002, 347: 1999-2009.
30. van't Veer, L.J., Dai, H., van de Vijver, M.J., He, Y.D., Hart, A.A.M., Mao, M., Peterse, H.L., van der Kooy, K., Marton, M.J., Witteveen, A.T., Schreiber, G.J., Kerkhoven, R.M., Roberts, C., Linsley, P.S., Bernards, R., Friend, S.H. Gene Expression Profiling Predicts Clinical Outcome of Breast Cancer. Nature 2002, 415: 530-536.
31. Braga-Neto, U.M., Fads and Fallcies in the Name of Small-Sample Microarray Classification. IEEE Signal Processing Magazine 2007.
32. -. Diffuse Large B-cell Lymphoma and Mantle Cell Lymphoma. Cancer Res. 2003, 63: 60-66.
33. Devroye, L., Gyorfi, L., Lugosi, G. A Probabilistic Theory of Pattern Recognition. New York: Springer-Verlag, 1996.
34. Braga-Neto, U.M., Dougherty, E.R. Exact Performance of Error Estimators for Discrete Classifiers. Pattern Recognition 2005, 38: 1799-1814.
35. Efron, B. Bootstrap Methods: Another Look at the Jacknife. Annals of Statistics 1979, 7: 1-26.
36. Efron, B. The Jacknife, The Bootstrap, and Other Resampling Plans. SIAM NSF-CBMS, monograph #38, 1983.
37. Sima, C., Dougherty, E.R. Optimal Convex Error Estimators for Classification. Pattern Recognition 2006, 39: 1763-1780.
38. Braga-Neto, U.M., Dougherty, E.R. Bolstered Error Estimation. Pattern Recognition 2004, 37: 1267-1281.
39. Xu, Q., Hua, J., Braga-Neto, U.M. Xiong, Z., Suh, E., Dougherty, E.R. Confidence Intervals for the True Classification Error Conditioned on the Estimated Error. Technology in Cancer Research and Treatment 2006, 5: 579-590.
40. Devroye, L. Wagner, T. Distribution-free Inequalities for the Deleted and Hold-out Error Estimates. IEEE Transactions on Information Theory 1979, 25: 202-207.
41. Sima, C., Braga-Neto, U.M., Dougherty, E.R. Superior Feature-Set Ranking For Small Samples Using Bolstered Error Estimation, Bioinformatics 2005, 21: 1046-1054.
42. Braga-Neto, U.M., Hashimoto, R., Dougherty, E.R., Nguyen, D.V., Carroll, R. J. Is Cross-validation Better than Resubstitution for Ranking Genes. Bioinformatics 2004, 20: 253-258.
43. Hughes, G.F. On the Mean Accuracy of Statistical Pattern Recognizers. IEEE Transactions on Information Theory 1968, 14: 55-63.
44. Jain, A.K., Chandrasekaran, B. Dimensionality and Sample Size Considerations in Pattern Recognition Practice. In Krishnaiah, P.R., Kanal, L.N. (ed)., Classification, Pattern Recognition and Reduction of Dimensionality, vol. 2, Handbook of Statistics. Amsterdam: North-Holland, 1982, 835-856.
45. Hua, J., Xiong, Z., Dougherty, E. R. Determination of the Optimal Number of Features for Quadratic Discriminant Analysis Via the Normal Approximation to the Discriminant Distribution. Pattern Recognition 2005, 38: 403-421.
46. Hua, J., Xiong, Z., Lowey, J., Suh, E., Dougherty, E.R. Optimal Number of Features as a Function of Sample Size for Various Classification Rules. Bioinformatics 2005, 21: 1509-1515.
47. Cover, T., van Campenhout, J. On the Possible Orderings in the Measurement Selection Problem. IEEE Transactions on Systems, Man, and Cybernetics 1977, 7: 657-661.
48. Pudil, P., Novovicova, J., Kittler, J. Floating Search Methods in Feature Selection. Pattern Recognition Lett. 1994, 15: 1119-1125.
49. Jain, A.K., Zongker, D. Feature Selection: Evaluation, Application, and Small Sample Performance. IEEE Transactions on Pattern Analysis and Machine Intelligence 1997, 19: 153-158.
50. Kudo, M., Sklansky, J. Comparison of Algorithms that Select Features for Pattern Classifiers. Pattern Recognition 2000, 33: 25-41.
51. Choudhary, A., Brun, M., Hua, J., Lowey, J., Suh, E., Dougherty, E.R. Genetic Test Bed for Feature Selection. Bioinformatics 2006, 22: 837-842.
52. Ben-Dor, A., Shamir, R., Yakhini, Z. Clustering Gene Expression Patterns. Computational Biology 1999, 6: 281-297.
53. Jain, A.K., Murty, M.N., Flynn, P.J. Data Clustering: A Review. ACM Computer Surveys 1999, 31: 264-323.
54. Dougherty, E.R., Brun, M. A Probabilistic Theory of Clustering. Pattern Recognition 2004, 37: 917-925.
55. Dougherty, E.R., Barrera, J., Brun, M., Kim, S., Cesar, R.M., Chen, Y., Bittner, M.L., Trent, J.M. Inference From Clustering With Application to Gene-Expression Microarrays. Comput. Biol. 2002, 9: 105-126.
56. Brun, M., Sima, C., Hua, J., Lowey, J., Carroll, B., Suh, E., Dougherty, E.R. Model-Based Evaluation of Clustering Validation Measures. Pattern Recognition 2007, 40: 807-824.
57. Dunn, J.C. A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Well-Separated Clusters. Cybernetics 1973, 3: 32-57.
58. Lange, T., Braun, M., Roth, V., Buhmann, J.M. Stability-based Model Selection. Advances in Neural Information Processing Systems, 2002.
59. Roth, V., Braun, M., Lange, T., Buhmann, J.M. Stability-based Model Order Selection in Clustering with Application to Gene Expression Data. In: Artificial Neural Networks - ICANN, Berlin: Springer, 2002: 607-612.