Online linear and quadratic discriminant analysis with adaptive forgetting for streaming classification

Statistical Analysis and Data Mining

Volumn 5, Issue 2, 2012, Pages 139-166

  Anagnostopoulos, Christoforos ^a,c   Tasoulis, Dimitris K ^b   Adams, Niall M ^c   Pavlidis, Nicos G ^d   Hand, David J ^c  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b Winton Capital Management Ltd   (United Kingdom)

^c IMPERIAL COLLEGE LONDON   (United Kingdom)

^d LANCASTER UNIVERSITY MANAGEMENT SCHOOL   (United Kingdom)

Author keywords

Forgetting factor; Linear discriminant analysis; Online; Streaming data; Time varying classification

Indexed keywords

ADAPTIVE FILTERS; CONSUMER BEHAVIOR; DISCRIMINANT ANALYSIS; FACTOR ANALYSIS; TECHNOLOGY TRANSFER; TUNING;

ADAPTIVITY; CONSUMER CREDITS; DATA TECHNOLOGIES; FORGETTING FACTORS; LINEAR DISCRIMINANT ANALYZE; ONLINE; QUADRATIC DISCRIMINANT ANALYSIS; REAL-TIME INFORMATION; STREAMING DATA; TIME-VARYING CLASSIFICATIONS;

ADAPTIVE FILTERING;

EID: 84858324284     PISSN: 19321872     EISSN: 19321864     Source Type: Journal    
DOI: 10.1002/sam.10151     Document Type: Article

References (79)

1. D. J. Hand and D. J. Weston, Statistical techniques for fraud detection, prevention and assessment, In Mining Massive Data Sets for Security, Fogelman-Soulie, D. Perrotta, J. Pisorski, R. Steinberger eds. Amsterdam, IOS Press, 2008, 257-270.
2. G. Hébrail, Practical data mining in a large utility company, Questiio 25(3) (2001), 509-520.
3. J. Gama, P. Medas, G. Castillo, and P. Rodrigues, Learning with drift detection, SBIA Brazilian Symposium on Artificial Intelligence, 2004, 286-295.
4. D. J. Hand, Classifier technology and the illusion of progress (with discussion), Stat Sci 21(1) (2006), 1-34.
5. K. D. Copsey, Automatic target recognition using both measurements from identity sensors and motion information from tracking sensors, In Automatic Target Recognition, Proceedings of SPIE, USA, Vol. 5807, 2005, 273-283.
6. S. J. Delaney, P. Cunniungham, and A. Tsymbal, A comparison of ensemble and case-base maintenance techniques for handling concept drift in spam filtering, G. Sutcliffe and R. Goebel, eds. 19th International Conference on Artificial Intelligence FLAIRS'2006, 2006, 340-345.
7. F. Fdez-Riverola, E. L. Iglesias, F. Diaz, J. R. Mendez, and J. M. Corchado, SpamHunting: an instance-based reasoning system for spam labelling and filtering, Decis Support Syst 43 (2007), 722-726.
8. B. Crabtree and S. J. Soltysiak, Identifying and tracking changing interests, Int J Dig Lib 2(1) (1998), 38-53.
9. M. Black and R. Hickey, Learning classification rules for telecom customer call data under concept drift, Soft Comput 8(2) (2004), 102-108.
10. K. Copsey and A. Webb, Classifier design for population and sensor drift, In Structural, Syntactic and Statistical Pattern Recognition, LNCS 3138, Berlin, Springer-Verlag, 2004, 744-752.
11. L. Cohen, G. Avrahami-Bakish, M. Last, A. Kandel, and O. Kipersztok, Real-time data mining of non-stationary data streams from sensor networks, Informat Fusion 3(9) (2008), 344-353.
12. M. G. Kelly, D. J. Hand, and N. M. Adams, The impact of changing populations on classifier performance, In KDD '99: Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, 1999, 367-371.
13. D. J. Hand, C. Whitrow, N. M. Adams, P. Juszczak, and D. J. Weston, Performance criteria for plastic card fraud detection tools, J Oper Res Soc 59 (2008), 956-962.
14. L. I. Kuncheva and I. Zliobaite, On the window size for classification in changing environments, Intell Data Anal 13(6) (2009), 861-872.
15. C. C. Aggarwal, J. Han, J. Wang, and P. S. Yu, On demand classification of data streams, In KDD '04: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data mining, New York, 2004, 503-508.
16. J. Wu, X. S. Hua, and B. Zhang, Tracking concept drift with Gaussian mixture models, In Proceedings of SPIE, USA, Vol.5960, 2005, 1562-1570.
17. A. Lucas, Updating scorecards: removing the mystique, In Credit Scoring and Credit Control, L. Thomas, J. Crook and D. Edelman, eds. Oxford, Clarendon, 1992, 180-197.
18. Y. Law and C. Zaniolo, An adaptive nearest neighbour classification algorithm for data streams, LNCS 3721(3721) (2005), 108-120.
19. G. Hulten, L. Spencer, and P. Domingos, Mining time-changing data streams, In KDD '01: Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, 2001, 97-106.
20. L. I. Kuncheva, Classifier ensembles for changing environments, In 5th International Workshop on Multiple Classifier Systems, Cagliari, Italy, Vol. 3077, F. Roli, J. Kittler, and T. Windeatt, eds. LNCS, Berlin, Springer Verlag, 2004, 1-15.
21. J. Z. Kolter and M. A. Maloof, Dynamic weighted majority: an ensemble method for drifting concepts, J Mach Learn Res 8 (2007), 2755-2790.
22. H. Wang, W. Fan, P. S. Yu, and J. Han, Mining concept-drifting data streams using ensemble classifiers, In KDD'03: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, 2003, 226-235.
23. W. N. Street and Y. Kim, A streaming ensemble algorithm (SEA) for large-scale classification, In KDD'01: Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, 2001, 377-382.
24. A. R. Webb, Statistical Pattern Recognition, (2nd ed), West Sussex, England, Wiley, 2002.
25. K. Hiraoka, S. Yoshizawa, K. Hidai, M. Hamahira, H. Mizoguchi, and T. Mishima, Convergence analysis of online linear discriminant analysis, In Proceedings of IJCNN 2000, Italy, Como, 2000.
26. T. K. Kim, S. F. Wong, B. Stenger, J. Kittler, and R. Cipolla, Incremental linear discriminant analysis using sufficient spanning set approximations, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, USA, 2007, 1-8.
27. S. Pang, S. Ozawa, and N. Kasabov, Incremental linear discriminant analysis for classification of data streams, IEEE Trans Syst Man Cybern B: Cybern 35(5) (2005), 905-914.
28. L. I. Kuncheva, and C. O. Plumpton, Adaptive learning rate for online linear discriminant classifiers, SSPR and SPR 2008, LNCS 5342 (2008), 510-519.
29. N. G. Pavlidis, D. K. Tasoulis, N. M. Adams, and D. J. Hand, λ-perceptron: an adaptive classifier for data streams, Pattern Recogn 44(1) (2011), 78-96.
30. C. Anagnostopoulos, D. K. Tasoulis, N. M. Adams, and D. J. Hand, Temporally adaptive estimation of logistic classifiers on data streams, ADAC 3(3) (2009), 243-261.
31. L. Carbonara and A. Borrowman, A comparison of batch and incremental supervised learning algorithms, Principl Data Mining Knowl Discov, LNCS 1510 (1998), 264-272.
32. G. Fung and O. L. Mangasarian, Incremental support vector machine classification, Proceedings of the Second SIAM International Conference on Data Mining, Arlington, Virginia, 2002, 247-260.
33. APACS, Press release, Published Online, March 2007,
34. G. Widmer and M. Kubat, Learning in the presence of concept drift and hidden contexts, Mach Learn 23(1) (1996), 69-101.
35. B. D. Ripley, Pattern Recognition and Neural Networks, Cambridge, Cambridge University Press, 1996.
36. R. Alaiz-Rodrigez and N. Japkowicz, Assessing the impact of changing environments on classifier performance, Advances in Artificial Intelligence, Lecture Notes in Computer Science, Vol. 5032/2008, Berlin / Heidelberg, Springer, 2008, 13-24.
37. J. W. Krzanowski and D. J. Hand, ROC Curves for Continuous Data, Taylor & Francis Group, Boca Raton, Chapman & Hall CRC Press, 2009.
38. M. West and J. Harrison, Bayesian Forecasting and Dynamic Models, New York, Springer-Verlag, 1997, Corrected Second Printing, 1999.
39. W. D. Penny and S. J. Roberts, Dynamic logistic regression, In Proceedings of the International Joint Conference on Neural Networks, Washington, DC, 1999, 3.
40. J. Whittaker, C. Whitehead, and M. Somers, A dynamic scorecard for monitoring baseline performance with application to tracking a mortgage portfolio, J Oper Res Soc 58(11) (2008), 911-921.
41. S. Haykin, Adaptive Filter Theory, Upper Saddle River, NJ, USA, Prentice-Hall, Inc., 1996.
42. L. Guo, L. Ljung, and P. Priouret, Performance analysis of the forgetting factor rls algorithm, Int J Adapt Contr Signal Process 7 (1993), 525-537.
43. L. Ljung and S. Gunnarsson, Adaptation and tracking in system identification? a survey, Automatica (J IFAC) 26(1) (1990), 7-21.
44. Z. Zhiyong, W. Shifu, F. Chongzhi, and K. Jingli, Unified framework of adaptive forgetting identification algorithms for time-varying systems, Proceedings of the IEEE Tencon, 1993, 424-427.
45. L. Ljung, System Identification: Theory for the User, Upper Saddle River, NJ, USA, Prentice-Hall, Inc., 1986.
46. M. E. Salgado, G. C. Goodwin, and R. H. Middleton, Modified least squares algorithm incorporating exponential resetting and forgetting, Int J Contr 47(2) (1988), 477-491.
47. M. Niedzwiecki, Identification of Time-Varying Processes, New York, John Wiley & Sons, 2000.
48. P. Andersson, Adaptive forgetting in recursive identification through multiple models, Int J Contr 42 (1985), 1175-1193.
49. M. Bodson, An adaptive algorithm with information-dependent data forgetting, In Proceedings of the American Control Conference, Seattle, 1995, 5.
50. Y. S. Cho, S. B. Kim, and E. J. Powers, Time-varying spectral estimation using ar models with variable forgetting factors, IEEE Trans Signal Process 39(6) (1991), 1422-1426.
51. T. R. Fortescue, L. S. Kershenbaum, and B. E. Ydstie, Implementation of self-tuning regulators with variable forgetting factors, Automatica 17(6) (1981), 831-835.
52. D. J. Park, B. E. Jun, and J. H. Kim, Fast tracking RLS algorithm using novel variable forgetting factor with unity zone, Electron Lett 27(23) (1991), 2150-2151.
53. S. Saelid and B. Foss, Adaptive controllers with a variable forgetting factor, In Proceedings of the 22nd IEEE Conference on Decision and Control, San Antonio, TX, 1983.
54. I. Koychev, Gradual forgetting for adaptation to concept drift, In Proceedings of ECAI 2000 Workshop on Current Issues in Spatio-Temporal Reasoning, Berlin, Germany, 2000, 101-106.
55. A. Benveniste, P. Priouret, and M. Métivier, Adaptive Algorithms and Stochastic Approximations, New York, Inc., USA, Springer-Verlag, 1990.
56. R. S. Sutton, Gain adaptation beats least squares? In Proceedings of the Seventh Yale Workshop on Adaptive and Learning Systems, Yale University, New Haven, Connecticut, USA, 1992.
57. H. J. Kushner and J. Yang, Analysis of adaptive step size SA algorithms for parameter tracking, In Proceedings of the 33rd IEEE Conference on Decision and Control, Lake Buena Vista, FL, 1994, 1.
58. K. Crammer, M. Dredze, and F. Pereira, Exact convex confidence-weighted learning, Advan Neural Inform Process Syst 21 (2009), 345-352.
59. A. O'Hagan, J. Forster, eds, Kendall's Advanced Theory of Statistics, Vol. 2B, Bayesian Inference, London, Arnold Publishers, 2004.
60. M. Opper, A bayesian approach to online learning, Online Learning in Neural Networks, Cambridge, Cambridge University Press, 1998, 363-378.
61. A. P. Dawid, Properties of diagnostic data distributions, Biometrics 32(3) (1976), 647-658.
62. D. K. Tasoulis, N. M. Adams, and D. J. Hand, Selective fusion of delayed measurements in filtering, In IEEE International Workshop on Machine Learning for Signal Processing, (IEEE MLSP), Thessaloniki, Greece, 2007, 336-341.
63. S. Dasgupta and D. Hsu, Online estimation with the multivariate Gaussian distribution, In Proceedings of the 20th Annual Conference On Learning Theory (COLT'07), Berlin, Spinger-Verlag, 2007.
64. X. Wang, C. van Eeden, and J. V. Zidek, Asymptotic properties of maximum weighted likelihood estimators, J Stat Plan Infer 119(1) (2004), 37-54.
65. C. Anagnostopoulos, A Statistical Framework for Streaming Data Analysis, Ph.D. Thesis, Imperial College London, 2010.
66. D. A. Harville, Matrix Algebra from a Statistician's Perspective, New York, Springer Science, 2008.
67. H. J. Kushner and J. Yang, Analysis of adaptive step size SA algorithms for parameter tracking, IEEE Trans Automat Contr 40(8) (1995), 1403-1410.
68. M. Riedmiller and H. Braun, A direct adaptive method for faster backpropagation learning: the RPROP algorithm, IEEE International Conference on Neural Networks, 1993, 586-591.
69. T. Hastie, R. Tibshirani, and J. H. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, New York, Springer Science, 2001.
70. Q. McNemar, Note on the sampling error of the difference between correlated proportions or percentages, Psychometrika 12(2) (1947), 153-157.
71. E. R. DeLong, D. M. Delong, and D. L. Clarke-Pearson, Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach, Biometrics 44 (1988), 837-845.
72. F. Rosenblatt, The perceptron: a probabilistic model for information storage and organization in the brain, Psychol Rev 65(6) (1958), 386-408.
73. C. Anagnostopoulos, D. Tasoulis, D. J. Hand, and N. Adams, Online optimisation for variable selection in data streams, In Proceedings of the 18th European Conference on Artificial Intelligence, ECAI 2008, Patras, Greece, 2008, 132-136.
74. M. Baena-Garcia, J. del Campo-Avila, R. Fidalgo, A. Bifet, R. Gavalda, and R. Morales-Bueno, Early drift detection method, In ECML PKDD 2006 Workshop on Knowledge Discovery from Data Streams, Berlin, Germany, 2006, 77-86.
75. A. Narasimhamurthy and L. I. Kuncheva, A framework for generating data to simulate changing environments, Proceedings of the 25th IASTED: artificial intelligence and applications, Innsbruck, Austria, ACTA Press, 2007, 384-389.
76. R. J. Bolton and D. J. Hand, Statistical fraud detection: a review, Stat Sci 17(3) (2002), 235-254.
77. D. K. Tasoulis, D. J. Weston, N. M. Adams, and D. J. Hand, Mining information from plastic card transaction streams, P. Brito, ed, In 18th International Conference on Computational Statistics, Porto, Portugal, 2008, 315-322.
78. L. Wasserman, All of Statistics: a Concise Course in Statistical Inference, (2nd ed), New York, USA, Springer Verlag, 2004.
79. J. R. Munkres, Analysis of Manifolds, Addison-Wesley, Reading, MA, 1991.