Finding the Number of Clusters in a Dataset: An Information-Theoretic Approach

Journal of the American Statistical Association

Volumn 98, Issue 463, 2003, Pages 750-763

  Sugar, Catherine A ^a   James, Gareth M ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Cluster analysis; Distortion; Information theory; K Means clustering; Mixture models

Indexed keywords

EID: 0242679438     PISSN: 01621459     EISSN: None     Source Type: Journal    
DOI: 10.1198/016214503000000666     Document Type: Article

References (38)

1. Akaike, H. (1973), "Information Theory and an Extension of the Maximum Likelihood Principle," in Second International Symposium on Information Theory, eds. B. N. Petrov and F. Czáki, Budapest: Akademiai Kiadó, pp. 267-281.
2. Berger, J. O., and Bernardo, J. M. (1989), "Estimating a Product of Means: Bayesian Analysis With Reference Priors," Journal of the American Statistical Association, 84, 200-207.
3. Berger, T. (1971), Rate Distortion Theory: A Mathematical Basis for Data Compression, Englewood Cliffs, NJ: Prentice-Hall.
4. Bernardo, J. M. (1979), "Expected Information as Expected Utility," The Annals of Statistics, 7, 686-690.
5. Biernacki, C., Celeux, G., and Govaert, G. (2000), "Assessing a Mixture Model for Clustering With the Integrated Completed Likelihood," IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 719-725.
6. Blahut, R. E. (1987), Principles and Practice of Information Theory, Reading, MA: Addison-Wesley.
7. Calinski, R. B., and Harabasz, J. (1974), "A Denrite Method for Cluster Analysis," Communications in Statistics, 3, 1-27.
8. Cover, T. M., and Thomas, J. A. (1991), Elements of Information Theory (2nd ed.), New York: Wiley.
9. Csiszar, I., and Korner, J. (1981). Information Theory: Coding Theorems for Discrete Memoryless Systems (2nd ed.), New York: Academic Press.
10. Efron, B., and Tibshirani, R. (1993), An Introduction to the Bootstrap, London: Chapman & Hall.
11. Fisher, R. A. (1936), "The Use of Multiple Measurements in Taxonomic Problems," Annals of Eugenics, 7, 179-188.
12. Frayley, C., and Raftery, A. (1998), "How Many Clusters? Which Clustering Method? Answers via Model-Based Cluster Analysis," Technical Report 329, University of Washington, Dept. of Statistics.
13. Frigui, H., and Krishnapuram, R. (1999), "A Robust Competitive Clustering Algorithm With Applications in Computer Vision," IEEE Transactions on Pattern Analysis and Machine Intelligence, 21, 450-465.
14. Gallager, R. G. (1968), Information Theory and Reliable Communication, New York: Wiley.
15. Gersho, A., and Gray, R. (1992), Vector Quantization and Signal Compression, Boston: Kluwer Academic Publishers.
16. Gokcay, E., and Principe, J. C. (2002), "Information Theoretic Clustering," IEEE Transactions on Pattern Analysis and Machine Intelligence, 24, 158-171.
17. Hardy, A. (1996), "On the Number of Clusters," Computational Statistics and Data Analysis, 23, 83-96.
18. Hartigan, J. A. (1975), Clustering Algorithms, New York: Wiley.
19. James, G. M., and Sugar, C. A. (2003), "Clustering for Sparsely Sampled Functional Data," Journal of the American Statistical Association, 98, 397-408.
20. Joe, H. (1989), "Relative Entropy Measures of Multivariate Dependence," Journal of the American Statistical Association, 84, 157-164.
21. Kass, R. E., and Raftery, A. E. (1995), "Bayes Factors," Journal of the American Statistical Association, 90, 773-795.
22. Kaufman, L., and Rousseeuw, P. (1990), Finding Groups in Data: An Introduction to Cluster Analysis, New York: Wiley.
23. Krzanowski, W. J., and Lai, Y. T. (1985), "A Criterion for Determining the Number of Clusters in a Data Set," Biometrics, 44, 23-34.
24. Kullback, S., and Leibler, R. A. (1951), "On Information and Sufficiency," The Annals of Mathematical Statistics, 22, 79-86.
25. Lindley, D. V. (1956), "On a Measure of the Information Provided by an Experiment," The Annals of Mathematical Statistics, 27, 986-1005.
26. McEliece, R. J. (1977), The Theory of Information and Coding: A Mathematical Framework for Communication, Reading, MA: Addison-Wesley.
27. Milligan, G. W., and Cooper, M. C. (1985), "An Examination of Procedures for Determining the Number of Clusters in a Data Set," Psychometrika, 50, 159-179.
28. Na, S., and Neuhoff, D. (1995), "Bennett's Integral for Vector Quantizers," IEEE Transactions on Information Theory, 41, 886-900.
29. Quinlan, R. (1993), "Combining Instance-Based and Model-Based Learning", in Proceedings of the Tenth International Conference of Machine Learning, University of Massachusetts, Amherst, pp. 236-243.
30. Roberts, S. J., Husmeier, D., Rezek, I., and Penny, W. (1998), "Bayesian Approaches to Gaussian Mixture Modeling," IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 1133-1142.
31. Shannon, C. E. (1948), "A Mathematical Theory of Communication," Bell System Technical Journal, 27, 379-423.
32. _ (1959), "Coding Theorems for a Discrete Source With a Fidelity Criterion," IRE National Convention Record, 7, 142-163.
33. Sooft, E. S. (1994), "Capturing the Intangible Concept of Information," Journal of the American Statistical Association, 89, 1243-1254.
34. Sugar, C. A. (1999), "An Application of Cluster Analysis to Health Services Research: Empirically Defined Health States for Depression From the sf-12," Technical Report, Stanford University, Dept. of Statistics.
35. Tibshirani, R., Walther, G., and Hastie, T. (2001), "Estimating the Number of Clusters in a Data Set via the Gap Statistic," Journal of the Royal Statistical Society, Ser. B, 63, 411-423.
36. Wolberg, W. H., and Mangasarian, O. L. (1990), "Multisurface Method of Pattern Separation for Medical Diagnosis Applied to Breast Cytology," Proceedings of the National Academy of Sciences, USA, 87, 9193-9196.
37. Yuan, A., and Clarke, B. (1999a), "An Information Criterion for Likelihood Selection," IEEE Transactions on Information Theory, 45, 562-571.
38. _ (1999b), "A Minimally Informative Likelihood for Decision Analysis: Illustration and Robustness," Canadian Journal of Statistics, 27, 649-666.