Joint sampling distribution between actual and estimated classification errors for linear discriminant analysis

IEEE Transactions on Information Theory

Volumn 56, Issue 2, 2010, Pages 784-804

  Zollanvari, Amin ^a   Braga Neto, Ulisses M ^a   Dougherty, Edward R ^a,b  

^a Department of Electrical and Computer Engineering   (United States)

^b TRANSLATIONAL GENOMICS RESEARCH INSTITUTE   (United States)

Author keywords

Classification; Cross validation; Error estimation; Leave one out; Linear discriminant analysis; Resubstitution; Sampling distribution

Indexed keywords

CROSS VALIDATION; LEAVE-ONE-OUT; LINEAR DISCRIMINANT ANALYSIS; RESUBSTITUTION; SAMPLING DISTRIBUTION;

DIAGNOSIS; DISCRIMINANT ANALYSIS; ERRORS; ESTIMATION; IMAGE RETRIEVAL; MOLECULAR BIOLOGY; PROTEINS;

ASYMPTOTIC ANALYSIS;

EID: 77649305686     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2009.2037034     Document Type: Article

References (49)

1. C. H. S. Michiels and S. Koscielny, "Prediction of cancer outcome with microarrays: A multiple random validation strategy," The Lancet, vol. 365, pp. 488-492, 2005.
2. A. Dupuy and R. Simon, "Critical review of published microarray studies for cancer outcome and guildelines on statistical analysis and reporting," J. Nat. Cancer Inst., vol. 99, p. 147, 2007.
3. U. Braga-Neto and E. Dougherty, "Is cross-validation valid for microarray classification?," Bioinformatics, vol. 20, no. 3, pp. 374-380, 2004.
4. U. Braga-Neto, "Fads and fallacies in the name of small-sample microarray classification," IEEE Signal Processing Mag., vol. 24, no. 1, pp. 91-99, 2007.
5. E. Dougherty, J. Hua, and M. Bittner, "Validation of computational methods in genomics," Curr. Genom., vol. 8, no. 1, pp. 1-19, 2007.
6. U. Braga-Neto, R. Hashimoto, E. Dougherty, D. Nguyen, and R. Carroll, "Is cross-validation better than resubstitution for ranking genes?," Bioinformatics, vol. 20, no. 2, pp. 253-258, 2004.
7. C. Smith, "Some examples of discrimination," Ann. Eugen., vol. 18, pp. 272-282, 1947.
8. P. Lachenbruch and M. Mickey, "Estimation of error rates in discriminant analysis," Technometrics, vol. 10, pp. 1-11, 1968.
9. T. Cover, "Learning in pattern recognition," in Methodologies of Pattern Recognition, S. Watanabe, Ed. New York: Academic, 1969, pp. 111-132.
10. G. Toussaint and R. Donaldson, "Algorithms for recognizing contour-traced hand-printed characters," IEEE Trans. Comput., vol. 19, pp. 541-546, 1970.
11. M. Stone, "Cross-validatory choice and assessment of statistical predictions," J. Roy. Statist. Soc. Series B (Methodol.), vol. 36, pp. 111-147, 1974.
12. T. Nagahata et al., "Expression profiling to predict postoperative prognosis for estrogen receptor-negative breast cancers by analysis of 25,344 genes on a cdna microarray," Cancer Sci., vol. 95, no. 3, pp. 218-225, 2004.
13. S. Chiaretti et al., "Gene expression profile of adult t-cell acute lymphocytic leukemia identifies distinct subsets of patients with different response to therapy and survival," Blood, vol. 103, no. 7, pp. 2771-2778, 2004.
14. F. De Smet, N. Pochet, K. Engelen, T. Van Gorp, P. Van Hummelen, K. Marchal, F. Amant, D. Timmerman, B. De Moor, and I. Vergote, "Predicting the clinical behavior of ovarian cancer from gene expression profiles," Int. J. Gynecol. Cancer, vol. 16, no. Suppl 1, pp. 147-151, 2006.
15. H. Somura, N. Iizuka, T. Tamesa, K. Sakamoto, T. Hamaguchi, R. Tsunedomi, H. Yamada-Okabe, M. Sawamura, M. Eramoto, T. Miyamoto, Y. Hamamoto, and M. Oka, "A three-gene predictor for early intrahepatic recurrence of hepatocellular carcinoma after curative hepatectomy," Oncol. Rep., vol. 19, no. 2, pp. 489-495, 2008.
16. M. Shirahata, K. Iwao-Koizumi, S. Saito, N. Ueno, M. Oda, N. Hashimoto, J. Takahashi, and K. Kato, "Gene expression-based molecular diagnostic system for malignant gliomas is superior to histological diagnosis," Clin. Cancer Res., vol. 13, no. 24, pp. 7341-7356, 2007.
17. C. Rimkus, J. Friederichs, A. Boulesteix, J. Theisen, J. Mages, K. Becker, H. Nekarda, R. Rosenberg, K. Janssen, and J. Siewert, "Microarray-based prediction of tumor response to neoadjuvant radiochemotherapy of patients with locally advanced rectal cancer," Clin. Gastroenterol. Hepatol., vol. 6, no. 1, pp. 53-61, 2008.
18. O. Gevaert, F. De Smet, T. Van Gorp, N. Pochet, K. Engelen, F. Amant, B. De Moor, D. Timmerman, and I. Vergote, "Expression profiling to predict the clinical behaviour of ovarian cancer fails independent evaluation," BMC Cancer, vol. 8, p. 18, 2008.
19. L. Devroye, L. Gyorfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition. New York: Springer, 1996.
20. J. Kittler and P. DeVijver, "Statistical properties of error estimators in performance assessment of recognition systems," IEEE Trans. Pattern Anal. Machine Intell., vol. 4, pp. 215-220, 1982.
21. D. Hand, "Recent advances in error rate estimation," Pattern Recogn. Lett., vol. 4, pp. 335-346, 1986.
22. R. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population," Biometrika, vol. 10, pp. 507-521, 1915.
23. R. Fisher, "On the probable error of a coefficient of correlation deduced from a small sample," Metron, vol. 1, pp. 3-32, 1921.
24. R. Fisher, "The general sampling distribution of the multiple correlation coefficient," Proc. Roy. Soc., Ser. A., vol. 121, pp. 654-673, 1928.
25. D. Hand, "Classifier technology and the illusion of progress," Statistic. Sci., vol. 21, pp. 1-14, 2006.
26. S. Attoor and E. Dougherty, "Classifier performance as a function of distributional complexity," Pattern Recogn., vol. 37, no. 8, pp. 1629-1640, 2004.
27. R. Fisher, "The use of multiple measurements in taxonomic problems," Ann. Eugen., vol. 7, pp. 179-188, 1936.
28. A. Wald, "On a statistical problem arising in the classification of an individual into one of two groups," Ann. Math. Statist., vol. 15, pp. 145-162, 1944.
29. T. Anderson, "Classification by multivariate analysis," Psychometrika, vol. 16, pp. 31-50, 1951.
30. S. John, "Errors in discrimination," Ann. Math. Statist., vol. 32, pp. 1125-1144, 1961.
31. R. Sitgreaves, "Some results on the distribution of the w-classification," in Studies in Item Analysis and Prediction, H. Solomon, Ed. Stanford, CA: Stanford University Press, 1961, pp. 241-251.
32. 2 and anderson's classification statistic w in terms of simple statistics," in Studies in Item Analysis and Prediction, H. Solomon, Ed. Stanford, CA: Stanford University Press, 1961, pp. 285-292.
33. A. Bowker and R. Sitgreaves, "An asymptotic expansion for the distribution function of the w-classification statistic," in Studies in Item Analysis and Prediction, H. Solomon, Ed. Stanford, CA: Stanford University Press, 1961, pp. 292-310.
34. H. Harter, "On the distribution of wald's classification statistics," Ann. Math. Statist., vol. 22, pp. 58-67, 1951.
35. R. Sitgreaves, "On the distribution of two random matrices used in classification procedures," Ann. Math. Statist., vol. 23, pp. 263-270, 1951.
36. D. Teichroew and R. Sitgreaves, "Computation of an empirical sampling distribution for the w-classification statistic," in Studies in Item Analysis and Prediction, H. Solomon, Ed. Stanford, CA: Stanford University Press, 1961, pp. 285-292.
37. M. Okamoto, "An asymptotic expansion for the distribution of the linear discriminant function," Ann. Math. Statist., vol. 39, pp. 1358-1359, 1968.
38. D. Kabe, "Some results on the distribution of two random matrices used in classification procedures," Ann. Math. Statist., vol. 34, pp. 181-185, 1963.
39. M. Hills, "Allocation rules and their error rates," J. Roy. Statistic. Soc. Series B (Methodol.), vol. 28, no. 1, pp. 1-31, 1966.
40. M. Moran, "On the expectation of errors of allocation associated with a linear discriminant function," Biometrika, vol. 62, no. 1, pp. 141-148, 1975.
41. G. McLachlan, "The bias of the apparent error in discriminant analysis," Biometrika, vol. 63, no. 2, pp. 239-244, 1976.
42. S. Raudys, "Comparison of the estimates of the probability of misclassification," in Proc. 4th Int. Conf. Pattern Recogn., Kyoto, Japan, 1978, pp. 280-282.
43. D. Foley, "Considerations of sample and feature size," IEEE Trans. Inf. Theory, vol. IT-18, pp. 618-626, 1972.
44. A. Jain and W. Waller, "On the optimal number of features in the classification of multivariate Gaussian data," Pattern Recogn., vol. 10, pp. 365-374, 1978.
45. A. Zollanvari, U. Braga-Neto, and E. R. Dougherty, "On the sampling distribution of resubstitution and leave-one-out error estimators for linear classifiers," Pattern Recognit., vol. 42, no. 11, pp. 2705-2723, Nov. 2009.
46. A. Genz and F. Bretz, "Methods for the computation of multivariate t-probabilities," J. Stat. Comp. Simul., vol. 11, pp. 950-971, 2002.
47. M. van de Vijver, Y. He, L. van't Veer, H. Dai, A. Hart, D. Voskuil, G. Schreiber, J. Peterse, C. Roberts, M. Marton, M. Parrish, D. Astma, A. Witteveen, A. Glas, L. Delahaye, T. van der Velde, H. Bartelink, S. Rodenhuis, E. Rutgers, S. Friend, and R. Bernards, "A gene-expression signature as a predictor of survival in breast cancer," The New England J. Med., vol. 347, pp. 1999-2009, Dec. 2002.
48. B. Hanczar, J. Hua, and E. Dougherty, "Decorrelation of the true and estimated classifier errors in high-dimensional settings," EURASIP J. Bioinform. Syst. Biol., vol. 2007, p. 12, 2007, Article ID 38473.
49. W. Anderson, An Introduction to Multivariate Statistical Analysis, 2nd ed. New York: Wiley, 1984.