Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 13, Issue 3, 1991, Pages 252-264

  Raudys, Sarunas J ^a   Jain, Anil K ^b  

^a Institute of Mathematics and Cybernetics Lithuanian of Academy Sciences ^*  (Russian Federation)

^b MICHIGAN STATE UNIVERSITY   (United States)

Author keywords

Classification error; classifier design; curse of dimensionality; feature selection; statistical pattern recognition; test samples; training samples

Indexed keywords

IMAGE PROCESSING - IMAGE ANALYSIS; MATHEMATICAL STATISTICS; PROBABILITY;

CLASSIFICATION ERROR; CLASSIFIER DESIGN; FEATURE SELECTION; MULTINOMIAL CLASSIFIER; STATISTICAL PATTERN RECOGNITION; TRAINING SAMPLES;

PATTERN RECOGNITION;

EID: 0026120032     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/34.75512     Document Type: Article

References (58)

1. R.A. Abusev and Y.P. Lumelskij, “Unbiased estimators and classification problems for multivariate normal populations”, Theor. Prob. and Appl., vol. 25, pp. 381–389, 1980 (in Russian).
2. S.A. Aivazian, V. M. Buchstaber, I.S. Yenyukov, and L. D. Meshalkin, “Applied statistics: Classification and reduction of dimensionality”, Finansy i Statistika (Reference Edition), Moscow, 1989 (in Russian).
3. B. G. Batchelor and D. J. Hand, “Pattern recognition competition”, in Proc. 3rd Int. Conf. Pattern Recognition, Coronado, 1976, pp. 315–321.
4. M. Ben-Bassat, “Use of distance measures, information measures and error bounds in feature evaluation”, in Handbook of Statistics, vol. 2, P. R. Krishnaiah and L. N. Kanal, Eds. Amsterdam, The Netherlands: North-Holland, 1982, pp. 773–791.
5. L. Breiman, J. Friedman, R.A. Olsen, and C.J. Stone, Classification and Regression Trees. Belmont, CA: Wadsworth, 1984.
6. Y. D. Broffitt, “Nonparametric Classification”, in Handbook of Statistics, vol. 2, P. R. Krishnaiah and L. N. Kanal, Eds. Amsterdam, The Netherlands: North-Holland, 1982, pp. 139–168.
7. B. Chandrasekaran and A.K. Jain, “On balancing decision functions”, J. Cybern. Inform. Sci., vol. 2, pp. 12–15, 1979.
8. L. Devroye and T. J. Wagner, “Nearest neighbor methods in discrimination”, in Handbook of Statistics, vol. 2, P. R. Krishnaiah and L. N. Kanal, Eds. Amsterdam, The Netherlands: North-Holland, 1982, pp. 193–198.
9. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973.
10. B. Efron, “The efficiency of logistic regression compared to normal discriminant analysis”, J. Amer. Statist. Assoc., vol. 70, pp. 892–898, 1975.
11. I. S. Enukov, “A choice of a set of measurements with maximal discriminating power in the case of limited learning sample size”, in Multivariate Statistical Analysis in Social-Economic Research. Moscow, USSR: Nauka, 1974, pp. 394–397 (in Russian).
12. D. M. Foley, “Considerations of sample and feature size”, IEEE Trans. Inform. Theory, vol. IT-18, pp. 618–626, 1972.
13. K. Fukunaga, “Statistical pattern recognition”, in Handbook of Pattern Recognition and Image Processing, T. Y. Young and K. S. Fu, Eds. New York: Academic, 1986, pp. 3–32.
14. K. Fukunaga and L. D. Hostetler, “Optimization of K-nearest-neighbor density estimates”, IEEE Trans. Inform. Theory, vol. IT-19, pp. 320–326, 1973.
15. S. Geiser, “Posterior odds for multivariate normal classifications”, J. Roy. Statist. Soc. B, vol. 21, no. 1, pp. 69 -76, 1964.
16. N. Glick, “Additive estimators for probabilities of correct classification”, Pattern Recog., vol. 10, no. 3, pp. 211–222, 1978.
17. M. Goldstein and W.R. Dillon, Discrete Discriminant Analysis, New York: Wiley, 1978.
18. V. Grabauskas, Inst. Math. Cybern., Acad. Sci., Lithuania, personal communication, 1983.
19. D. Griškevičius and Š. Raudys, “On the expected probability of the classification error of the classifier for discrete variables”, in Statistical Problems of Control, issue 38, Š. Raudys, Ed. Vilnius, USSR: Inst. Math. Cybern. Press, 1979, pp. 95–112 (in Russian).
20. D.J. Hand, “Recent advances in error rate estimation”, Pattern Recog. Lett., vol. 5, pp. 335–346, 1986.
21. A.K. Jain, R.C. Dubes, and C.C. Chen, “Bootstarp techniques for error estimation”, IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-9, no. 9, pp. 628–636, 1987.
22. A. K. Jain and B. Chandrasekaran, “Dimensionality and sample size considerations in pattern recognition practice”, in Handbook of Statistics, vol. 2, P. R. Krishnaiah and L. N. Kanal, Eds. Amsterdam, The Netherlands, North-Holland, 1982, pp. 835–855.
23. A. K. Jain and M. D. Ramaswami, “Classifier design with Parzen windows”, in Pattern Recognition and Artificial Intelligence, E. S. Gelsema and L. N. Kanal, Eds. Amsterdam, The Netherlands: Elsevier, 1988, pp. 211–228.
24. A. K. Jain and W. G. Waller, “On the optimal number of features in the classification of multivariate Gaussian data”, Pattern Recog., vol. 10, pp. 365–374, 1978.
25. L. Kanal, “Patterns in pattern recognition 1968-1974”, Trans. Inform. Theory, vol. IT-20, pp. 697–722, 1974.
26. L. Kanal and B. Chandrasekaran, “On dimensionality and sample size in statistical pattern classification”, Pattern Recog., vol. 3, pp. 238–255, 1971.
27. D. G. Keehn, “A note on learning for Gaussian properties”, IEEE Trans. Inform. Theory, vol. IT-11, no. 1, pp. 126–131, 1965.
28. J. Kittler, “Feature selection and extraction”, in Handbook of Pattern Recognition and Image Processing, T. Y. Young and K. S. Fu, Eds. New York: Academic, 1986, pp. 60–83.
29. P.A. Lachenbruch and R. M. Mickey, “Estimation of error rates in discriminant analysis”, Technometrics, vol. 10, no. 1, pp. 1–11, 1968.
30. P. A. Lachenbruch, C. Sneeringer, and L. T. Revo, “Robustness the linear and quadratic discriminant functions to certain types of non-normality”, Commun. Statist., vol. 1, no. 1, pp. 39–56, 1972.
31. G. S. Lbov, “Logical functions in the problems of empirical prediction”, in Handbook of Statistics, vol. 2, P. R. Krishnaiah and L. N. Kanal, Eds. Amsterdam, The Netherlands: North-Holland, 1982, pp. 479–491.
32. T. Lissack and K. S. Fu, “Error estimation in pattern via L-distance between posterior density functions”, IEEE Trans. Inform. Theory, vol. IT-22, pp. 34–45, 1976.
33. G.J. McLachlan, “The bias of the apparent error rate in discriminant analysis”, Biometrika, vol. 63, pp. 239 - 244, 1976.
34. G.J. McLachlan, “Assessing the performance of an allocation rule”, Comput. Math. Applicat., vol. 12A, pp. 261–272, 1976.
35. G.J. McLachlan, “The efficiency of Efron's ‘bootstrap’ to error estimation in discriminant analysis”, J. Stat. Comput. Simulation, vol. 11, pp. 273–279, 1980.
36. G.J. McLachlan, “Error rate advances”, in Advances in Multivariate Statistical Analysis, A. K. Gupta, Ed. Dordrect, The Netherlands: Reidel, 1987, pp. 233–252.
37. L. Miroshnichenko, “Comparison of algorithms for selecting the best feature set in pattern recognition”, in Statistical Problems of Control, issue 93. Vilnius, USSR: Inst. Math. Cybern. Press, 1990, pp. 78–91 (in Russian).
38. T. Y. O'Neill, “The error rate a classification procedure with application to logistic regression discrimination”, J. Amer. Statist. Assoc., vol. 75, pp. 154–160, 1980.
39. K.W. Pettis, T.A. Bailey, A.K. Jain, and R.C. Dubes, “An intrinsic dimensionality estimator from near-neighbor information”, IEEE Trans. Pattern Anal. Machine Intell, vol. PAMI-1, no. 1, pp. 25–37, 1979.
40. V. Pikelis, “Analysis of learning speed of three linear classifiers”, Ph.D. Dissertation, Inst. Phys. Math., Vilnius, pp. 1–136, 1974 (in Russian).
41. Š. Raudys, “On the problems of sample size in pattern recognition”, in Proc. 2nd All-Union Conf. Statistical Methods in Control Theory, Moscow, USSR: Nauka, 1970, pp. 64–67 (in Russian).
42. Š. Raudys, V. Pikelis, and K. Juškevičius, “Experimental comparison of thirteen classification algorithms”, in Statistical Problems of Control, issue 11, Vilnius, USSR: Inst. Phys. Math. Press, 1975, pp. 35–80 (in Russian).
43. Š. Raudys, “Comparison of the estimates of the probability of misclassification”, in Proc. 4th Int. Conf. Pattern Recognition, Kyoto, Japan, Nov. 1978, pp. 280–282.
44. Š. Raudys, “Determination of optimal dimensionality in pattern classification”, Pattern Recog., vol. 11, pp. 263 -270, 1979.
45. Š. Raudys and V. Pikelis, “On dimensionality, sample size, classification error, and complexity of classification algorithm in pattern recognition”, IEEE Trans. Pattern Anal. Machine Intell, vol. PAMI-2, no. 3, pp. 242–252, 1980.
46. Š. Raudys, “The influence of sample size on classification performance”, in Statistical Problems of Control, issue 66. Vilnius, USSR, Inst. Math. Cybern. Press, 1984, pp. 9–42 (in Russian).
47. Š. Raudys, and V. Vaitukaitis, “Methods to estimate the probability of misclassification”, in Statistical Problems of Control, issue 66. Vilnius, USSR: Inst. Math. Cybern. Press, 1984, pp. 43–65 (in Russian).
48. Š. Raudys, “On the accuracy of a bootstrap estimate of the classification error”, in Proc. 9th Int. Conf Pattern Recognition, Rome, Italy, Nov. 1988, pp. 1230–1232.
49. Š. Raudys, V. Pikelis, and D. Stasaitis, “The effects of the number of initial and final features, the dependence between the features and the type of a classification rule on the accuracy of feature selection”, Pattern Recog. Artificial Intell., 1990, submitted for publication.
50. J. W. Sayre, “The distribution of actual error rates in linear discriminant analysis”, J. Amer. Statist. Assoc., vol. 75, pp. 201–205, 1980.
51. I. K. Sethi and G. P. R. Sarvarayudu, “Hierarchical classifier design using mutual information”, IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-4, pp. 441–445, 1982.
52. M. Siotani, “Large sample approximations and asymptotic expansions of classification statistics”, in Handbook of Statistics, vol. 2, P. R. Krishnaiah and L. N. Kanal, Eds. Amsterdam, The Netherlands: North-Holland, 1982, pp. 61–100.
53. M. Skurikhina, “Effect of the kernel form on the quality of nonparametric Parzen window classifier”, in Statistical Problems of Control, issue 93. Vilnius, USSR: Inst. Math. Cybern. Press, 1990 (in Russian).
54. G.T. Toussaint, “Bibliography on estimation of misclassification”, IEEE Trans. Inform. Theory, vol. 20, pp. 472–479, 1974.
55. N. Vanichsetakul, “Tree structured classification via recursive discriminant analysis”, Ph.D. Dissertation, Univ. Wisconsin, 1986.
56. V. N. Vapnik, Recovery of Dependencies from Empirical Data. New York: Springer-Verlag, 1982.
57. C.T. Wolverton and T.J. Wagner, “Asymptotically optimal discriminant functions for pattern classification”, IEEE Trans. Inform. Theory, vol. IT-15, no. 2, pp. 258–265, 1969.
58. D. Zvirenaite, “Criteria for selecting the informative features in pattern recognition”, in Statistical Problems of Control, issue 74. Vilnius, USSR: Inst. Math. Cybern. Press, 1986, pp. 76–103 (in Russian).