Active learning and subspace clustering for anomaly detection

Intelligent Data Analysis

Volumn 15, Issue 2, 2011, Pages 151-171

  Pichara, Karim ^a   Soto, Alvaro ^a  

^a Edificio San Agustín ^*  (Chile)

Author keywords

active learning; Anomaly detection; Bayessian network; Dirichlet distribution; probabilistic model

Indexed keywords

ACTIVE LEARNING; ANOMALY DETECTION; BAYESSIAN NETWORK; DIRICHLET DISTRIBUTIONS; PROBABILISTIC MODELS;

CLUSTER ANALYSIS; CLUSTERING ALGORITHMS; INSURANCE; SEMANTICS;

PROBABILITY DISTRIBUTIONS;

EID: 79953301764     PISSN: 1088467X     EISSN: 15714128     Source Type: Journal    
DOI: 10.3233/IDA-2010-0461     Document Type: Article

References (49)

1. N. Abe, B. Zadrozny and J. Langford, Outlier detection by active learning, In KDD '06: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, New York, NY, USA, 2006, pp. 504-509, ACM. (Pubitemid 44535546)
2. D. Agarwal, An empirical bayes approach to detect anomalies in dynamic multidimensional arrays, in: Proceedings of the 5th IEEE International Conference on Data Mining. IEEE Computer Society, 2005, pp. 26-33. (Pubitemid 47385673)
3. R. Agrawal, J. Gehrke, D. Gunopulos and P. Raghavan, Automatic subspace clustering of high dimensional data for data mining applications, in: Proceedings ACMSIGMOD International Conference Management of Data, 1998, pp. 94-105. (Pubitemid 128655960)
4. A. Arning, R. Agrawal and P. Raghavan, A linear method for deviation detection in large databases. in: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1996, pp. 164-169.
5. A. Bianco, M. Ben, E. Martinez and V. Yohai, Outlier detection in regression models with arima errors using robust estimates, Journal of Forecasting 20(8) (2001), 565-579. (Pubitemid 34036017)
6. C.M. Bishop, Novelty detection and neural network validation, in: Proceedings of IEE Conference on Vision, Image and Signal Processing, 1994, pp. 217-222.
7. D. Blackwell and J. MacQueen, Ferguson distribution via polya urn schemes, The Annals of Statistics 1(2) (1973), 353-355.
8. M. Breunig, H. Kriegel, R. Ng and J. Sander, Lof: Identifying densitybased local outliers, in: Proceedings of 2000 ACM SIGMOD International Conference on Management of Data, 2000, pp. 93-104.
9. A. Cansado and A. Soto, Unsupervised anomaly detection in large databases using bayesian networks, Applied Artificial Intelligence 22(4) (2008), 309-330. (Pubitemid 351612907)
10. N. Cebron and M. Berthold, Active learning for object classification: from exploration to exploitation, Data Mining and Knowledge Discovery 18(2) (2008), 283-299.
11. D. Chen, X. Shao, B. Hu and Q. Su, Simultaneous wavelength selection and outlier detection in multivariate regression of nearinfrared spectra, Analytical Sciences 21(2) (2005), 161-167.
12. S. Djorgovski, A. Mahabal, R. Brunner, R. Gal, S. Castro, R. de Carvalho and S. Odewahn, Searches for rare and new types of objects, in: Virtual Observatories of the Future, ASP Conference Proceedings, (Vol. 225), 2001, pp. 52-63.
13. E. Eskin, Anomaly detection over noisy data using learned probability distributions, in: Proceedings of the Seventeenth International Conference on Machine Learning, 2000, pp. 255-262.
14. M. Ester, H. Kriegel and X. Xu, A density based algorithm for discovering clusters in large spatial databases with noise, in: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, 1996, pp. 226-231.
15. T. Fawcett and F. Provost, in: Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1999, p. 5362.
16. T. Ferguson, A bayesian analysis of some nonparametric problems, The Annals of Statistics 1(2) (1973), 209-230.
17. N. Friedman, I.Nachman and D. Peér, Learning Bayesian network structure frommassive datasets: The Sparse Candidate algorithm, in: Proceedings of 15th Conference on Uncertainty in Artificial Intelligence, 1999, pp. 206-215.
18. Grubb and E. Frank, Procedures for detecting outlying observations in samples, Technometrics 11(1) (1969), 1-21.
19. J. He and J. Carbonell, Nearestneighborbased active learning for rare category detection, in: Advances in Neural Information Processing Systems, (vol. 20), 2007.
20. V. Hodge and J. Austin, A survey of outlier detection methodologies, Artificial Intelligence Review 22(2) (2004), 85-126.
21. Z. Huang, Extensions to the kmeans algorithm for clustering large data sets with categorical values, Data Mining and Knowledge Discovery 2 (1998), 283-304. (Pubitemid 128695480)
22. L. Hubert and P. Arabie, Comparing partitions, Journal of classification 2(1) (1985), 193-218.
23. W. Jin, A. Tung and J. Han, Mining topn local outliers in large databases, in: Proceedings of the seventh ACM SIGKDD International Conference on Knowledge discovery and data mining, 2001, pp. 293-298.
24. G. John, Robust decision trees: Removing outliers from databases, in: Proceedings of the first International Conference on Knowledge Discovery and Data Mining, 1995, pp. 174-179.
25. E. Knorr and R. Ng, Algorithms for mining distancebased outliers in large datasets, in: Proceedings of the VLDB Conference, New York, USA, 1998, pp. 392-403.
26. Y. Kou, C. Lu, S. Sirwongwattana and Y. Huang, Survey of fraud detection techniques, in: Proceedings of the IEEE International Conference on Networking, Sensing and Control, 2004, pp. 749-754.
27. T. Lane and C. Brodley, Applications of machine learning to anomaly detection, in: Applications of Artificial Intelligence in Engineering, R. Adey, G. Rzevski and T. Teti, eds, Southampton, UK: Comput. Mech. Publications., 1997, p. 11314.
28. D. Lewis and W. Gale. A sequential algorithm for training text classifiers, in: Proceedings of 17th International Conference ACM SIGIR, 1994, pp. 3-12.
29. D. Mery and D. Filbert, Automated flaw detection in aluminum castings based on the tracking of potential defects in a radioscopic image sequence, IEEE Transactions on Robotics and Automation 18(6) (2002), 890-901.
30. A. Nairac, N. Towsend, R. Carr, S. King, P. Cowley and L. Tarassenko, A system for the analisys of jet system vibration data, Integrated ComputerAided Engineering 6(1) (1999), 53-65.
31. R. Neapolitan, Learning Bayesian Networks, Prentice Hall, 2004.
32. S. Papadimitriou, H. Kitagawa, P. Gibbons and C. Faloutsos, Loci: Fast outlier detection using the local correlation integral. Technical Report IRPTR0209, Intel Research Laboratory, Pittsburgh, PA, 2002.
33. J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann, 1988.
34. D. Pelleg and A.Moore, Active learning for anomaly and rarecategory detection, in: Proceedings of the 18th Conference on Advances in Neural Information Processing Systems, NIPS, 2004.
35. C. Phua, D. Alahakoon and V. Lee, Minority report in fraud detection: Classification of skewed data, SIGKDD Explor Newsl 6(1) (2004), 50-59.
36. S. Ramaswamy, R. Rastogi and K. Shim, Efficient algorithms for mining outliers from large data sets, in: Proceedings of the ACM SIGMOD Conference on Management of Data, Dallas, TX, 2000, pp. 427-438.
37. N.Roy and A.McCallum, Toward optimal active learning through sampling estimation of error reduction, in: Proceedings of 18th International Conference on Machine Learning, ICML, 2001, pp. 441-448.
38. T. Seidl, E. Müller, I. Assent and U. Steinhausen, Outlier detection and ranking based on subspace clustering, in: Uncertainty Management in Information Systems, 2009.
39. S. Seung, M. Opper and H. Sompolinski, Query by committee, in: Proceedings of 5th Annual ACM Workshop on Computational Learning Theory, 1992, pp. 287-294. (Pubitemid 23615454)
40. Ch. Son, S. Cho and J. Yoo, Volume traffic anomaly detection using hierarchical clustering, in: Management Enabling the Future Internet for Changing Business and New Computing Services, (vol. 5787), Springer Berlin, Heidelberg, 2009, pp. 291-300.
41. A. Soto, A. Cansado and F. Zavala, Detection of rare objects inmassive astronomical datasets using innovative knowledge discovery technology, in: Astronomical Data Analysis Software and Systems XIV ASP Conference Series, (Vol. 347), 2005, pp. 66-71.
42. A. Soto, F. Zavala and A. Araneda, An accelerated algorithm for density estimation in large databases using Gaussian mixtures, Cybernetics and Systems 38(2) (2007), 123-139. (Pubitemid 46346880)
43. C. Surace and K. Worden, A novelty detection method to diagnose damage in structures: An application to an offshore platform, in: Proceedings of Eighth International Conference of Offshore and Polar Engineering, 1998, pp. 64-70.
44. S. Tong and D. Koller, Active learning for parameter estimation in bayesian networks, in: Proceedings of the 13th Conference on Advances in Neural Information Processing Systems, NIPS, 2001, pp. 647-653.
45. Arindam Banerjee Varun Chandola and Vipin Kumar, Anomaly detection: A survey, ACM Computing Surveys 41(3), 2009.
46. H. Yang, F. Xie and Y. Lu, Clustering and classification based anomaly detection, in: Fuzzy Systems and Knowledge Discovery, (vol. 4223), 2006, pp. 1082-1091. (Pubitemid 44670824)
47. N.Ye andQ.Chen,An anomaly detection technique based on a chisquare statistic for detecting intrusions into information systems, Quality and Reliability Engineering International 17 (2001), 105-112. (Pubitemid 32443869)
48. C. Zhang and T. Chen, An active learning framework for contentbased information retrieval, IEEE Transactions on Multimedia 4 (2002), 260-268.
49. T. Zhang, R. Ramakrishnan and M. Livny, Birch: An efficient data clustering method for very large databases, in: Proceedings of the ACM SIGMOD International Conference on Management of Data, 1996, pp. 103-114. (Pubitemid 126440760)