Iterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs

Pattern Recognition

Volumn 43, Issue 8, 2010, Pages 2732-2752

  Khreich, Wael ^a   Granger, Eric ^a   Miri, Ali ^b   Sabourin, Robert ^a  

^a ÉCOLE DE TECHNOLOGIE SUPÉRIEURE   (Canada)

^b UNIVERSITY OF OTTAWA   (Canada)

Author keywords

Anomaly detection; Combination of classifiers; Computer and network security; Hidden Markov models; Limited and imbalanced data; Receiver operating characteristics

Indexed keywords

ANOMALY DETECTION; APPLICATION DOMAINS; BOOLEAN COMBINATIONS; COMBINATION OF CLASSIFIERS; COMPUTER AND NETWORK SECURITY; HOST-BASED INTRUSION DETECTION; IMBALANCED DATA; MULTIPLE CLASSIFIERS; OPERATING SYSTEM KERNEL; REAL-WORLD; REAL-WORLD APPLICATION; ROC CURVES; SYSTEM CALLS; TIME COMPLEXITY; TRAINING DATA; TWO-CLASS CLASSIFIER;

BOOLEAN FUNCTIONS; CLASSIFIERS; COMPUTER NETWORKS; COMPUTER OPERATING SYSTEMS; COMPUTER SIMULATION; INTRUSION DETECTION; NETWORK SECURITY; OBJECT RECOGNITION;

HIDDEN MARKOV MODELS;

EID: 77951257705     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2010.03.006     Document Type: Article

References (56)

1. V. Chandola, A. Banerjee, V. Kumar, Anomaly detection for discrete sequences: a survey, Technical Report TR 09-015, University of Minnesota, Department of Computer Science and Engineering, 2009
2. S. Forrest, S.A. Hofmeyr, A. Somayaji, T.A. Longstaff, A sense of self for Unix processes, in: Proceedings of the 1996 IEEE Symposium on Research in Security and Privacy, 1996, pp. 120-128
3. C. Warrender, S. Forrest, B. Pearlmutter, Detecting intrusions using system calls: alternative data models, in: Proceedings of the IEEE Computer Society Symposium on Research in Security and Privacy, Oakland, CA, USA, 1999, pp. 133-45
4. Rabiner L. A tutorial on Hidden Markov Models and selected applications in speech recognition. Proceedings of the IEEE 77 2 (1989) 257-286
5. B. Gao, H.-Y. Ma, Y.-H. Yang, HMMs (Hidden Markov Models) based on anomaly intrusion detection method, in: Proceedings of 2002 International Conference on Machine Learning and Cybernetics, vol. 1, 2002, pp. 381-385
6. X. Hoang, J. Hu, An efficient Hidden Markov Model training scheme for anomaly intrusion detection of server applications based on system calls, in: IEEE International Conference on Networks, ICON, vol. 2, Singapore, 2004, pp. 470-474
7. M.J. Beal, Z. Ghahramani, C.E. Rasmussen, The infinite Hidden Markov Model, in: Advances in Neural Information Processing Systems (NIPS) 2001, vol. 14, MIT Press, Cambridge, MA, 2002, pp. 577-585
8. J.V. Gael, Y. Saatci, Y.W. Teh, Z. Ghahramani, Beam sampling for the infinite Hidden Markov Model, in: Proceedings of the 25th International Conference on Machine Learning, ACM, Helsinki, Finland, 2008, pp. 1088-1095
9. W. Khreich, E. Granger, R. Sabourin, A. Miri, Combining Hidden Markov Models for anomaly detection, in: International Conference on Communications (ICC), Dresden, Germany, 2009
10. Scott M.J.J., Niranjan M., and Prager R.W. Realisable classifiers: improving operating performance on variable cost problems. In: Lewis P.H., and Nixon M.S. (Eds). Proceedings of the Ninth British Machine Vision Conference vol. 1 (1998), University of Southampton, UK 304-315
11. F. Provost, T. Fawcett, Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions, in: Proceedings of the Third International Conference on Knowledge Discovery and Data Mining, AAAI Press, Menlo Park, CA, 1997, pp. 43-48
12. T. Fawcett, ROC graphs: Notes and practical considerations for researchers, Technical Report HPL-2003-4, HP Laboratories, Palo Alto, CA, USA, 2004
13. M. Barreno, A. Cardenas, D. Tygar, Optimal ROC for a combination of classifiers, in: Advances in Neural Information Processing Systems (NIPS), vol. 20, 2008
14. Tao Q., and Veldhuis R. Threshold-optimized decision-level fusion and its application to biometrics. Pattern Recognition 41 5 (2008) 852-867
15. W.B. Langdon, B.F. Buxton, Evolving receiver operating characteristics for data fusion, in: EuroGP '01: Proceedings of the 4th European Conference on Genetic Programming, Springer-Verlag, London, UK, 2001, pp. 87-96
16. S. Haker, W.M. Wells, S.K. Warfield, I.-F. Talos, J.G. Bhagwat, D. Goldberg-Zimring, A. Mian, L. Ohno-Machado, K.H. Zou, Combining classifiers using their receiver operating characteristics and maximum likelihood estimation, in: Medical Image Computing and Computer Assisted Intervention (MICCAI), vol. 3749, 2005, pp. 506-514
17. J. Hill, M. Oxley, K. Bauer, Receiver operating characteristic curves and fusion of multiple classifiers, in: Proceedings of the 6th International Conference on Information Fusion, vol. 2, 2003, pp. 815-822
18. M. Oxley, S. Thorsen, C. Schubert, A Boolean Algebra of receiver operating characteristic curves, in: 10th International Conference on Information Fusion, 2007, pp. 1-8
19. P.A. Flach, S. Wu, Repairing concavities in ROC curves, in: Proceedings of the 19th International Joint Conference on Artificial Intelligence, IJCAI, 2005, pp. 702-707
20. Hanley J., and McNeil B. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143 1 (1982) 29-36
21. Walter S.D. The partial area under the summary ROC curve. Statistics in Medicine 24 13 (2005) 2025-2040
22. Baum L.E., Petrie G.S., and Weiss N. A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. The Annals of Mathematical Statistics 41 1 (1970) 164-171
23. Ephraim Y., and Merhav N. Hidden Markov processes. IEEE Transactions on Information Theory 48 6 (2002) 1518-1569
24. Hanley J.A. The robustness of the "binormal" assumptions used in fitting ROC curves. Medical Decision Making 8 3 (1988) 197-203
25. Metz C. Basic principles of ROC analysis. Seminars in Nuclear Medicine 8 (1978) 283-298
26. Provost F.J., and Fawcett T. Robust classification for imprecise environments. Machine Learning 42 3 (2001) 203-231
27. J. Daugman, Biometric decision landscapes, Technical Report UCAM-CL-TR-482, University of Cambridge, UK, 2000
28. Black M.A., and Craig B.A. Estimating disease prevalence in the absence of a gold standard. Statistics in Medicine 21 18 (2002) 2653-2669
29. Venkataramani K., and Kumar B. Role of statistical dependence between classifier scores in determining the best decision fusion rule for improved biometric verification. Multimedia Content Representation, Classification and Security 4105 (2006) 489-496
30. Neyman J., and Pearson E.S. On the problem of the most efficient tests of statistical hypotheses. Royal Society of London Philosophical Transactions Series A 231 (1933) 289-337
31. Shen C. On the principles of believe the positive and believe the negative for diagnosis using two continuous tests. Journal of Data Science 6 (2008) 189-205
32. Thomopoulos S., Viswanathan R., and Bougoulias D. Optimal distributed decision fusion. IEEE Transactions on Aerospace and Electronic Systems 25 5 (1989) 761-765
33. Varshney P.K. Distributed Detection and Data Fusion (1997), Springer-Verlag, New York
34. Pepe M.S., and Thompson M.L. Combining diagnostic test results to increase accuracy. Biostatistics 1 2 (2000) 123-140
35. L.I. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms Wiley, Hoboken, NJ, 2004
36. Tulyakov S., Jaeger S., Govindaraju V., and Doermann D. Review of classifier combination methods. In: Simone Marinai H.F. (Ed). Studies in Computational Intelligence: Machine Learning in Document Analysis and Recognition (2008), Springer 361-386
37. Breiman L. Bagging predictors. Machine Learning 24 2 (1996) 123-140
38. Y. Freund, R.E. Schapire, Experiments with a new boosting algorithm, in: ICML 96, 1996, pp. 148-156
39. Ho T.K. The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 20 8 (1998) 832-844
40. Kuncheva L.I., and Whitaker C.J. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning 51 2 (2003) 181-207
41. Brown G., Wyatt J., Harris R., and Yao X. Diversity creation methods: a survey and categorisation. Journal of Information Fusion 6 1 (2005) 5-20
42. R. Banfield, L. Hall, K. Bowyer, W. Kegelmeyer, A new ensemble diversity measure applied to thinning ensembles, in: Multiple Classifier Systems, vol. 2709, 2003, pp. 306-316
43. Ruta D., and Gabrys B. Classifier selection for majority voting. Information Fusion 6 1 (2005) 63-81
44. Kittler J. Combining classifiers: a theoretical framework. Pattern Analysis & Applications 1 1 (1998) 18-27
45. Wolpert D.H. Stacked generalization. Neural Networks 5 (1992) 241-259
46. F. Roli, G. Fumera, J. Kittler, Fixed and trained combiners for fusion of imbalanced pattern classifiers, in: Proceedings of the Fifth International Conference on Information Fusion, vol. 1, 2002, pp. 278-284
47. Ho T.K., Hull J., and Srihari S. Decision combination in multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 16 1 (1994) 66-75
48. M. Van Erp, L. Schomaker, Variants of the borda count method for combining ranked classifier hypotheses, in: Seventh International Workshop on Frontiers in Handwriting Recognition, Amsterdam, 2000
49. Ruta D., and Gabrys B. A theoretical analysis of the limits of majority voting errors for multiple classifier systems. Pattern Analysis & Applications 5 4 (2002) 333-350
50. Å. Raudys, F. Roli, The behavior knowledge space fusion method: analysis of generalization error and strategies for performance improvement, in: Multiple Classifier Systems, vol. 2709, 2003, pp. 55-64
51. D. MacKay, Ensemble learning for hidden Markov models, Technical Report, Cavendish Laboratory, Cambridge, UK, 1997
52. Tan K., and Maxion R. Determining the operational limits of an anomaly-based intrusion detector. IEEE Journal on Selected Areas in Communications 21 1 (2003) 96-110
53. Bradley A.P. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition 30 7 (1997) 1145-1159
54. Huang J., and Ling C. Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering 17 3 (2005) 299-310
55. Zhang D.D., Zhou X.-H., Freeman Jr. D.H., and Freeman J.L. A non-parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets. Statistics in Medicine 21 5 (2002) 701-715
56. E.M. Dos Santos, R. Sabourin, P. Maupin, Pareto analysis for the selection of classifier ensembles, in: Genetic and Evolutionary Computation Conference (GECCO), Atlanta, GA, USA, 2008, pp. 681-688