On evaluating stream learning algorithms

Machine Learning

Volumn 90, Issue 3, 2013, Pages 317-346

  Gama, João ^a   Sebastião, Raquel ^a   Rodrigues, Pedro Pereira ^b  

^a UNIVERSITY OF PORTO   (Portugal)

^b University of Porto   (Portugal)

Author keywords

Concept drift; Data streams; Evaluation design; Prequential analysis

Indexed keywords

BAYES ERROR; BEST PRACTICES; CHANGE DETECTION; CHANGE DETECTION ALGORITHMS; CONCEPT DRIFTS; CONTINUOUS PROCESS; DATA STREAM; DECISION MODELS; ERROR ESTIMATORS; EVALUATION DESIGN; FADING FACTORS; FORGETTING MECHANISMS; HYPOTHESIS TESTING; MEMORYLESS; PERFORMANCE ASSESSMENT; PREQUENTIAL ANALYSIS; RESOURCE AWARE; SLIDING WINDOW; STREAMING APPLICATIONS;

DECODING; ESTIMATION; EXPERIMENTS; MODELS; SIGNAL DETECTION;

LEARNING ALGORITHMS;

EID: 84874677467     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-012-5320-9     Document Type: Article

References (47)

1. Asuncion, A., & Newman, D. (2007). UCI machine learning repository. http://www.ics.uci.edu/~mlearn/MLRepository.html.
2. Babcock, B., Datar, M., Motwani, R., & O'Callaghan, L. (2003). Maintaining variance and k-medians over data stream windows. In T. Milo (Ed.), Proceedings of the 22nd symposium on principles of database systems, San Diego, USA (pp. 234-243). New York: ACM.
3. Bach, S. H., & Maloof, M. A. (2008). Paired learners for concept drift. In ICDM (pp. 23-32). Los Alamitos: IEEE Comput. Soc.
4. Basseville, M., & Nikiforov, I. (1993). Detection of abrupt changes: theory and applications. New York: Prentice Hall
5. Bifet, A., & Gavaldà, R. (2007). Learning from time-changing data with adaptive windowing. In Proceedings SIAM international conference on data mining, Minneapolis, USA (pp. 443-448). Philadelphia: SIAM.
6. Bifet, A., Holmes, G., Kirkby, R., & Pfahringer, B. (2010a). MOA: massive online analysis. Journal of Machine Learning Research, 11, 1601-1604.
7. Bifet, A., Holmes, G., Pfahringer, B., & Frank, E. (2010b). Fast perceptron decision tree learning from evolving data streams. In Advances in knowledge discovery and data mining, 14th Pacific-Asia conference (pp. 299-310).
8. Bishop, C. (1995). Neural networks for pattern recognition. London: Oxford University Press.
9. Chi, Y., Wang, H., Yu, P. S., & Muntz, R. R. (2006). Catch the moment: maintaining closed frequent itemsets over a data stream sliding window. Knowledge and Information Systems, 10(3), 265-294.
10. Cormode, G., Muthukrishnan, S., & Zhuang, W. (2007). Conquering the divide: continuous clustering of distributed data streams. In ICDE: proceedings of the international conference on data engineering, Istanbul, Turkey (pp. 1036-1045).
11. Datar, M., Gionis, A., Indyk, P., & Motwani, R. (2002). Maintaining stream statistics over sliding windows. SIAM Journal on Computing, 31(6), 1794-1813.
12. Dawid, A. P. (1984). Statistical theory: the prequential approach. Journal of the Royal Statistical Society. Series A, 147, 278-292.
13. Demsar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1-30.
14. Dietterich, T. (1996). Approximate statistical tests for comparing supervised classification learning algorithms. Corvallis, technical report nr. 97.331, Oregon State University.
15. Domingos, P., & Hulten, G. (2000). Mining high-speed data streams. In I. Parsa, R. Ramakrishnan, & S. Stolfo (Eds.), Proceedings of the ACM sixth international conference on knowledge discovery and data mining, Boston, USA (pp. 71-80). New York: ACM.
16. Duda, R., & Hart, P. (1973). Pattern classification and scene analysis. New York: Willey.
17. Ferrer-Troyano, F., Aguilar-Ruiz, J. S., & Riquelme, J. C. (2004). Discovering decision rules from numerical data streams. In Proceedings of the ACM symposium on applied computing, Nicosia, Cyprus (pp. 649-653). New York: ACM Press.
18. Gama, J., & Kosina, P. (2011). Learning decision rules from data streams. In Proceedings of the 22nd international joint conference on artificial intelligence, IJCAI (pp. 1255-1260).
19. Gama, J., Medas, P., Castillo, G., & Rodrigues, P. (2004). Learning with drift detection. In A. L. C. Bazzan & S. Labidi (Eds.), Lecture notes in computer science: Vol. 3171. Advances in artificial intelligence - SBIA 2004, São Luis, Brasil (pp. 286-295). Berlin: Springer.
20. Gama, J., Rocha, R., & Medas, P. (2003). Accurate decision trees for mining high-speed data streams. In Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, Washington, DC, USA (pp. 523-528). New York: ACM.
21. Gama, J., Sebastião, R., & Rodrigues, P. P. (2009). Issues in evaluation of stream learning algorithms. In Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, Paris, France (pp. 329-338). New York: ACM.
22. Ghosh, B., & Sen, P. (1991). Handbook of sequential analysis. New York: Dekker.
23. Giannella, C., Han, J., Pei, J., Yan, X., & Yu, P. (2003). Mining frequent patterns in data streams at multiple time granularities. In H. Kargupta, A. Joshi, K. Sivakumar, & Y. Yesha (Eds.), Next generation data mining. Menlo Park/Cambridge: AAAI Press/MIT Press.
24. Hartl, C., Baskiotis, N., Gelly, S., & Sebag, M. (2007). Change point detection and meta-bandits for online learning in dynamic environments. In Conférence Francophone sur l'apprentissage automatique, Cepadues (pp. 237-250).
25. Herbster, M., & Warmuth, M. (1998). Tracking the best expert. Machine Learning, 32(2), 151-178.
26. Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301), 13-30.
27. Hulten, G., & Domingos, P. (2001). Catching up with the data: research issues in mining data streams. In Proc. of workshop on research issues in data mining and knowledge discovery, Santa Barbara, USA.
28. Hulten, G., & Domingos, P. (2003). VFML - a toolkit for mining high-speed time-changing data streams. Technical report, University of Washington. http://www.cs.washington.edu/dm/vfml/
29. Hulten, G., Spencer, L., & Domingos, P. (2001). Mining time-changing data streams. In Proceedings of the 7th ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco, California (pp. 97-106). New York: ACM.
30. Japkowicz, N. & Shah, M. (Eds.) (2011). Evaluating learning algorithms: a classification perspective. Cambridge: Cambridge University Press.
31. Katakis, I., Tsoumakas, G., & Vlahavas, I. (2010). Tracking recurring contexts using ensemble classifiers: an application to email filtering. Knowledge and Information Systems, 22, 371-391.
32. Kearns, M., & Vazirani, U. (1994). An introduction to computational learning theory. Cambridge: MIT Press.
33. Kifer, D., Ben-David, S., & Gehrke, J. (2004). Detecting change in data streams. In Proceedings of the international conference on very large data bases, Toronto, Canada (pp. 180-191). San Mateo: Morgan Kaufmann.
34. Kirkby, R. (2008). Improving Hoeffding trees. Ph.D. thesis, University of Waikato, New Zealand.
35. Klinkenberg, R. (2004). Learning drifting concepts: example selection vs. example weighting. Intelligent Data Analysis, 8(3), 281-300.
36. Kolter, J. Z., & Maloof, M. A. (2007). Dynamic weighted majority: an ensemble method for drifting concepts. Journal of Machine Learning Research, 8, 2755-2790.
37. Koychev, I. (2000). Gradual forgetting for adaptation to concept drift. In Proceedings of ECAI workshop current issues in spatio-temporal reasoning, Berlin, Germany (pp. 101-106). Leipzig: ECAI Press.
38. Kuh, A., Petsche, T., & Rivest, R. (1990). Learning time-varying concepts. In Proceedings advances in neural information processing (pp. 183-189). San Mateo: Morgan Kaufmann.
39. Li, P., Wu, X., & Hu, X. (2010). Mining recurring concept drifts with limited labeled streaming data. Journal of Machine Learning Research - Proceedings Track, 13, 241-252.
40. Liang, C., Zhang, Y., & Song, Q. (2010). Decision tree for dynamic and uncertain data streams. Journal of Machine Learning Research - Proceedings Track, 13, 209-224.
41. Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006). Yale: rapid prototyping for complex data mining tasks. In ACM SIGKDD int. conf. on knowledge discovery and data mining (pp. 935-940). New York: ACM Press.
42. Mitchell, T. (1997). Machine learning. New York: McGraw-Hill
43. Mouss, H., Mouss, D., Mouss, N., & Sefouhi, L. (2004). Test of Page-Hinkley, an approach for fault detection in an agro-alimentary production system. In Proceedings of the Asian control conference (Vol. 2, pp. 815-818).
44. Page, E. S. (1954). Continuous inspection schemes. Biometrika, 41(1/2), 100-115.
45. Rodrigues, P. P., Gama, J., & Pedroso, J. P. (2008). Hierarchical clustering of time series data streams. IEEE Transactions on Knowledge and Data Engineering, 20(5), 615-627.
46. Street, W. N., & Kim, Y. (2001). A streaming ensemble algorithm SEA for large-scale classification. In Proceedings 7th ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco, California (pp. 377-382). New York: ACM Press.
47. Widmer, G., & Kubat, M. (1996). Learning in the presence of concept drift and hidden contexts. Machine Learning, 23, 69-101.