Outlier detection and data cleaning in multivariate non-normal samples: The PAELLA algorithm

Data Mining and Knowledge Discovery

Volumn 9, Issue 2, 2004, Pages 171-187

  Limas, Manuel Castejón ^a   Meré, Joaquín B Ordieres ^b   Ascacibar, Francisco J Martinez De Pison ^b   González, Eliseo P Vergara ^b  

^a UNIVERSITY OF LEÓN   (Spain)

^b UNIVERSITY OF LA RIOJA   (Spain)

Author keywords

Cluster analysis; Data cleaning; EM algorithm; Mixture model; Multivariate; Non normal; Outlier

Indexed keywords

ALGORITHMS; CALCULATIONS; COMPUTATIONAL METHODS; DATABASE SYSTEMS; LINEAR CONTROL SYSTEMS; MANUFACTURE; OPTIMIZATION; PERTURBATION TECHNIQUES; PROCESS CONTROL; REGRESSION ANALYSIS; SENSORS;

CLUSTER ANALYSIS; DATA CLEANING; EM ALGORITHMS; MIXTURE MODELS; NON-NORMAL; OUTLIER;

DATA MINING;

EID: 3543070082     PISSN: 13845810     EISSN: None     Source Type: Journal    
DOI: 10.1023/B:DAMI.0000031630.50685.7c     Document Type: Article

References (33)

1. Ihaka, R. and Gentleman, R. 1996. R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics, 5(3):299-314.
2. Banfield, J. and Raftery, A. 1993. Model-based Gaussian and non-Gaussian clustering. Biometrics, 49:803-821.
3. Billor, N., Hadi, A.S., and Velleman, P.F. 2000. BACON: Blocked adaptive computationally-efficient outlier nominators. Computational Statistics and Analysis, 34:279-298.
4. Bilmes, J. 1998. A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models.
5. Bradley, P., Fayyad, U., and Reina, C. 1999. Scaling EM (expectation-maximization) clustering to large databases. Technical Report MSR-TR-98-35., Microsoft Research, Seattle.
6. Campbell, N.A. 1990. Robust procedures in multivariate analysis I: Robust covariance estimation. Applied Statistics, 29:231-237.
7. Castejón Limas, M., Ordieres Meré, J.B., de Cos Juez, F.J., and Martínez de Pisón Ascacibar, F.J. 2001. Control de Calidad. Metodología para el Análisis Previo a la Modelización de Datos en Procesos Industriales. Fundamentos Teóricos y Aplicaciones Prácticas con R. Logroño: Servicio de Publicaciones de la Universidad de La Rioja.
8. Coleman, D., Dong, X., Hardin, J., and Rocke ad David L. Woodruff, D.M. 1999. Some computational issues in cluster analysis with no a priori metric. Computational Statistics and Data Analysis, 31:1-11.
9. Cuevas, A., Febrero, M., and Fraiman, R. 1996. Estimating the number of clusters. The Canadian Journal of Statistics, 28(2):367-382.
10. Cuevas, A., Febrero, M., and Fraiman, R. 2001. Cluster analysis: A further approach based in density estimation. Computational Statistics and Data Analalysis, 36(4):441-459.
11. de Ammorin, S., Barthelemy, J.-P., and Ribeiro, C. 1992. Clustering and clique partitioning: Simulated annealing and tabu search approaches. J. Classification, 9:17-41.
12. De Veaux, R. and Kreiger, A. 1990. Robust estimation of anormal mixture. Statistics & Probability Letters, 10:1-7.
13. Dempster, A., Laird, N., and Rubin, D. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, 39(1).
14. Fraley, C. and Raftery, A.E. 1998. How many clusters? Which clustering method? Answers via model-based cluster analysis. The Computer Journal 41(8);578-588.
15. Fraley, C. and Raftery, A.E. 1999. MCLUST: Software for model-based cluster analysis. Journal of Classification, 16:297-306.
16. Friedman, J. and Stuetzle, W. 1981. Projection pursuit regression. Journal of the American Statistical Association, 76(376):817-823.
17. Gallegos, M.T. 2000. A robust method for clustering analysis. Technical Report MIP-0013, Fakultät für Mathematik und Informatik, Universität Passau.
18. Hardy, A. 1996. On the number of clusters. Computational Statistics & Data Analysis, 23:83-96.
19. Hartigan, J. 1975. Clustering Algorithms. New York: Wiley.
20. Hawkins, D. 1980. Identifications of Outliers. New York: Chapman and Hall.
21. Markatou, M. 1998. Mixture models, robustness and the weighted likelihood methodology. Technical Report 1998-9, Department of Statistics, Stanford University.
22. McLachlan, G.J. 1988. On the choice of starting values for the EM algorithm in fitting mixture models. The Statistician, 37:417-425.
23. McLachlan, G.J. and Krishnan, T. 1997. The EM Algorithm and Extensions, Probability and Mathematical Statistics: Applied Probability and Statistics Section. New York: John Wiley & Sons.
24. McLachlan, G.J. and Peel, D.J. 2000a. On computational aspects of clustering via mixtures of normal and t-components. In Proceedings of the American Statistical Association (Bayesian Statistical Section); Indianapolis.
25. McLachlan, G.J. and Peel, D.J. 2000b. Robust cluster analysis via mixtures of multivariate t-distributions. Lectures Notes in Computer Science, 1451:658-666.
26. Muller, D. and Sawitzki, G. 1991. Using excess mass estimates to investigate the modality of a distribution. The Frontiers of Statistical Scientific Theory & Industrial Applications, 26:355-382.
27. Rocke, D. and Woodruff, D. 1996. Identification of outliers in multivariate data. J. Amer. Statist. Assoc., 91:1047-1061.
28. Rocke, D. and Woodruff, D. 1997. Robust estimation of multivariate location and shape. Journal of Statistical Planning and Inference, 57:245-255.
29. Rousseeuw, P.J. and Leroy, A. 1987. Robust Regression and Outlier DetectionDiagnostic Regression Analysis. New York: John Wiley and Sons.
30. Srivastava, M.S. and von Rosen, D. 1998. Outliers in multivariate regression models. Journal of Multivariate Analysis, 65:195-208.
31. Stanford, D. and Raftery, A.E. 1997. Principal curve clustering with noise. Technical Report 317, Department of Statistics. University of Washington.
32. Thiesson, B., Meek, C., and Heckerman, D. 2000. Accelerating EM for large databases. Technical Report MSR-TR-99-31., Microsoft Research, Seattle.
33. Wang, X.Z. 1999. Data mining and Knowledge Discovery for Process Monitoring and Control. London: Springer-Verlag.