Learn++ .NC: Combining ensemble of classifiers with dynamically weighted consult-and-vote for efficient incremental learning of new classes

IEEE Transactions on Neural Networks

Volumn 20, Issue 1, 2009, Pages 152-168

  Muhlbaier, Michael D ^a,b   Topalis, Apostolos ^a,c   Polikar, Robi ^a  

^a Rowan University   (United States)

^b Spaghetti Engineering ^*  (United States)

^c LOCKHEED MARTIN   (United States)

Author keywords

Consult and vote majority voting; Incremental learning; Multiple classifier systems

Indexed keywords

CLASSIFIERS; EDUCATION; LEARNING ALGORITHMS;

CONSULT-AND-VOTE MAJORITY VOTING; INCREMENTAL LEARNING; INDIVIDUAL CLASSIFIERS; LEARN++; MULTIPLE-CLASSIFIER SYSTEMS; NEW ALGORITHM; NEW CLASS;

LEARNING SYSTEMS;

VOLATILE ORGANIC COMPOUND;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; CHEMISTRY; COMPARATIVE STUDY; COMPUTER SIMULATION; FACTUAL DATABASE; IMAGE PROCESSING; REPRODUCIBILITY; TIME;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; DATABASES, FACTUAL; IMAGE PROCESSING, COMPUTER-ASSISTED; PATTERN RECOGNITION, AUTOMATED; REPRODUCIBILITY OF RESULTS; TIME FACTORS; VOLATILE ORGANIC COMPOUNDS;

EID: 58649083899     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2008.2008326     Document Type: Article

References (72)

1. E. M. Gold, "Language identification in the limit," Inf. Control vol. 10, no. 5, pp. 447-474, May 1967.
2. K. Jantke, "Types of incremental learning," in Proc. AAAI Symp. Training Issues Incremental Learn., 1993, vol. SS-93-06, pp. 26-32.
3. A. Sharma, "A note on batch and incremental learnability," J. Comput. Syst. Sci., vol. 56, no. 3, pp. 272-276, Jun. 1998.
4. S. Lange and G. Grieser, "On the power of incremental learning," Theor. Comput. Sci., vol. 288, no. 2, pp. 277-307, Oct. 2002.
5. Z. H. Zhou and Z. Q. Chen, "Hybrid decision tree," Knowl.-Based Syst., vol. 15, no. 8, pp. 515-528, Nov. 2002.
6. C. Giraud-Carrier, "A note on the utility of incremental learning," Artif. Intell. Commun., vol. 13, no. 4, pp. 215-223, 2000.
7. S. Lange and S. Zilles, "Formal models of incremental learning and their analysis," in Proc. Int. Joint Conf. Neural Netw., 2003, vol. 4, pp. 2691-2696.
8. L. M. Fu, "Incremental knowledge acquisition in supervised learning networks," IEEE Trans. Syst. Man Cybern. A, Syst. Humans, vol. 26, no. 6, pp. 801-809, Nov. 1996.
9. S. Grossberg, "Nonlinear neural networks: Principles, mechanisms, and architectures," Neural Netw., vol. 1, no. 1, pp. 17-61, 1988.
10. F. H. Hamker, "Life-long learning cell structures - Continuously learning without catastrophic interference," Neural Netw., vol. 14, no. 4-5, pp. 551-573, May 2001.
11. M. A. Maloof and R. S. Michalski, "Incremental learning with partial instance memory," Artif. Intell., vol. 154, no. 1-2, pp. 95-126, Apr. 2004.
12. T. Artieres, S. Marukatat, and P. Gallinari, "Online handwritten shape recognition using segmental hidden Markov models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 2, pp. 205-217, Feb. 2007.
13. R. Polikar, L. Udpa, S. Udpa, and V. Honavar, "An incremental learning algorithm with confidence estimation for automated identification of NDE signals," IEEE Trans. Ultrason. Ferroelectr. Freq. Control, vol. 51, no. 8, pp. 990-1001, Aug. 2004.
14. J. Macek, "Incremental learning of ensemble classifiers on ECG data," in Proc. IEEE Symp. Comput. Based Med. Syst., 2005, pp. 315-320.
15. R. Polikar, L. Udpa, S. S. Udpa, and V. Honavar, "Learn++: An incremental learning algorithm for supervised neural networks," IEEE Trans. Syst. Man Cybern. C, Appl. Rev., vol. 31, no. 4, pp. 497-508, Jul. 2001.
16. P. Winston, "Learning structural descriptions from examples," in The Psychology of Computer Vision, P. Winston, Ed. New York: Mc Graw Hill, 1975, pp. 157-209.
17. N. Littlestone, "Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm," Mach. Learn., vol. 2, no. 4, pp. 285-318, Apr. 1988.
18. R. Servedio, "PAC analogues of perceptron and Winnow via boosting the margin," Mach. Learn., vol. 47, no. 2-3, pp. 133-151, May 2002.
19. S. Nieto Sanchez, E. Triantaphyllou, J. Chen, and T. W. Liao, "An incremental learning algorithm for constructing Boolean functions from positive and negative examples," Comput. Operat. Res., vol. 29, no. 12, pp. 1677-1700, Oct. 2002.
20. M. K. Warmuth, D. P. Helmbold, and S. Panizza, "Direct and indirect algorithms for online learning of disjunctions," Theor. Comput. Sci. vol. 284, no. 1, pp. 109-142, 2002.
21. J. L. Sancho, W. E. Pierson, B. Ulug, A. R. Figueiras-Vidal, and S. C. Ahalt, "Class separability estimation and incremental learning using boundary methods," Neurocomputing, vol. 35, no. 1-4, pp. 3-26, Nov. 2000.
22. A. Shilton, M. Palaniswami, D. Ralph, and C. T. Ah, "Incremental training of support vector machines," IEEE Trans. Neural Netw., vol. 16, no. 1, pp. 114-131, Jan. 2005.
23. S. Ferrari and M. Jensenius, "A constrained optimization approach to preserving prior knowledge during incremental training," IEEE Trans. Neural Netw., vol. 19, no. 6, pp. 996-1009, Jun. 2008.
24. S. Ozawa, S. Pang, and N. Kasabov, "Incremental learning of chunk data for online pattern classification Systems," IEEE Trans. Neural Netw. vol. 19, no. 6, pp. 1061-1074, Jun. 2008.
25. K. Kasabov, "On-line learning, reasoning, rule extraction and aggregation in locally optimized evolving fuzzy neural networks," Neurocomputing, vol. 41, no. 1-4, pp. 25-45, Oct. 2001.
26. G. G. Yen and P. Meesad, "Constructing a fuzzy rule-based systems using the ILFN network and genetic algorithm," Int. J. Neural Syst., vol. 11, no. 5, pp. 427-443, 2001.
27. L. M. Fu, H. Hui-Huang, and J. C. Principe, "Incremental backpropagation learning networks," IEEE Trans. Neural Netw., vol. 7, no. 3, pp. 757-761, May 1996.
28. K. Yamauchi, N. Yamaguchi, and N. Ishii, "Incremental learning methods with retrieving of interfered patterns," IEEE Trans. Neural Netw. vol. 10, no. 6, pp. 1351-1365, Nov. 1999.
29. D. Obradovic, "On-line training of recurrent neural networks with continuous topology adaptation," IEEE Trans. Neural Netw., vol. 7, no. 1, pp. 222-228, Jan. 1996.
30. T. Hoya, "On the capability of accommodating new classes within probabilistic neural networks," IEEE Trans. Neural Netw., vol. 14, no. 2, pp. 450-453, Mar. 2003.
31. P. Utgoff, N. C. Berkman, and J. A. Clause, "Decision tree induction based on efficient tree restructuring," Mach. Learn., vol. 29, no. 1, pp. 5-44, 1997.
32. J. C. Schlimmer and R. H. Granger, "Incremental learning from noisy data," Mach. Learn., vol. 1, no. 3, pp. 317-354, Sep. 1986.
33. G. A. Carpenter, S. Grossberg, N. Markuzon, J. H. Reynolds, and D. B. Rosen, "Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps," IEEE Trans. Neural Netw., vol. 3, no. 5, pp. 698-713, Sep. 1992.
34. J. R. Williamson, "Gaussian ARTMAP: A neural network for fast incremental learning of noisy multidimensional maps," Neural Netw. vol. 9, no. 5, pp. 881-897, Jul. 1996.
35. C. P. Lim and R. F. Harrison, "An incremental adaptive network for on-line supervised learning and probability estimation," Neural Netw., vol. 10, no. 5, pp. 925-939, Jul. 1997.
36. M. C. Su, J. Lee, and K. L. Hsieh, "A new ARTMAP-based neural network for incremental learning," Neurocomputing, vol. 69, no. 16-18, pp. 2284-2300, 2006.
37. D. Heinke and F. H. Hamker, "Comparing neural networks: A benchmark on growing neural gas, growing cell structures, and fuzzy ARTMAP," IEEE Trans. Neural Netw., vol. 9, no. 6, pp. 1279-1291, Nov. 1998.
38. L. K. Hansen and P. Salamon, "Neural network ensembles," IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 10, pp. 993-1001, Oct. 1990.
39. R. E. Schapire, "The strength of weak learnability," Mach. Learn. vol. 5, no. 2, pp. 197-227, Jun. 1990.
40. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, "Adaptive mixtures of local experts," Neural Comput., vol. 3, no. 1, pp. 79-87, 1991.
41. D. H. Wolpert, "Stacked generalization," Neural Netw., vol. 5, no. 2, pp. 241-259, 1992.
42. T. K. Ho, J. J. Hull, and S. N. Srihari, "Decision combination in multiple classifier systems," IEEE Trans. Pattern Anal. Mach. Intell., vol. 16, no. 1, pp. 66-75, Jan. 1994.
43. J. Kittler, M. Hatef, R. P. W. Duin, and J. Mates, "On combining classifiers," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 3, pp. 226-239, Mar. 1998.
44. L. I. Kuncheva, Combining Pattern Classifiers, Methods and Algorithms New York: Wiley Interscience, 2005.
45. K. Woods, W. P. J. Kegelmeyer, and K. Bowyer, "Combination of multiple classifiers using local accuracy estimates," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 4, pp. 405-410, Apr. 1997.
46. E. Alpaydin and M. I. Jordan, "Local linear perceptrons for classification," IEEE Trans. Neural Netw., vol. 7, no. 3, pp. 788-792, May 1996.
47. Z. H. Zhou, J. Wu, and W. Tang, "Ensembling neural networks: Many could be better than all," Artif. Intell., vol. 137, no. 1-2, pp. 239-263, May 2002.
48. L. Breiman, "Bagging predictors," Mach. Learn., vol. 24, no. 2, pp. 123-140, 1996.
49. Y. Freund and R. E. Schapire, "Decision-theoretic generalization of on-line learning and an application to boosting," J. Comput. Syst. Sci., vol. 55, no. 1, pp. 119-139, 1997.
50. L. Breiman, "Arcing classifiers," Ann. Statist., vol. 26, no. 3, pp. 801-849, 1998.
51. G. Fumera and F. Roli, "A theoretical and experimental analysis of linear combiners for multiple classifier systems," IEEE Tran. Pattern Anal. Mach. Intell., vol. 27, no. 6, pp. 942-956, Jun. 2005.
52. G. Rogova, "Combining the results of several neural network classifiers," Neural Netw., vol. 7, no. 5, pp. 777-781, 1994.
53. L. I. Kuncheva, J. C. Bezdek, and R. P. W. Duin, "Decision templates for multiple classifier fusion: An experimental comparison," Pattern Recognit., vol. 34, no. 2, pp. 299-314, Feb. 2001.
54. L. Todorovski and L. Dzeroski, "Combining classifiers with meta decision trees," Mach. Learn., vol. 50, no. 223, p. 249, 2003.
55. R. Polikar, "Ensemble based systems in decision making," IEEE Circuits Syst. Mag., vol. 6, no. 3, pp. 21-45, 2006.
56. R. Polikar, "Bootstrap - Inspired techniques in computation intelligence," IEEE Signal Process. Mag., vol. 24, no. 4, pp. 59-72, Jul. 2007.
57. A. Fern and R. Givan, "Online ensemble learning: An empirical study," Mach. Learn., vol. 53, no. 1-2, pp. 71-109, 2003.
58. N. C. Oza, "Online bagging and boosting," in Proc. IEEE Int. Conf. Syst. Man Cybern., Waikoloa, HI, 2005, vol. 3, pp. 2340-2345.
59. D. Chakraborty and N. R. Pal, "A novel training scheme for multi-layered perceptrons to realize proper generalization and incremental learning," IEEE Trans. Neural Netw., vol. 14, no. 1, pp. 1-14, Jan. 2003.
60. W. N. Street and Y. Kim, "A streaming ensemble algorithm (SEA) for large-scale classification," in Proc. 7th ACM SIGKDD Int. Conf. Knowl. Disc. Data Mining, 2001, pp. 377-382.
61. J. Z. Kolter and M. A. Maloof, "Dynamic weighted majority: An ensemble method for drifting concepts," J. Mach. Learn. Res., vol. 8, pp. 2755-2790, 2007.
62. R. Polikar, J. Byorick, S. Krause, A. Marino, and M. Moreton, "Learn++: A classifier independent incremental learning algorithm for supervised neural networks," in Proc. IEEE Int. Joint Conf. Neural Netw., 2002, vol. 2, pp. 1742-1747.
63. M. Muhlbaier, A. Topalis, and R. Polikar, "Ensemble confidence estimates posterior probability," in Proc. 6th Int. Workshop Multiple Classifier Syst., N. C. Oza, R. Polikar, J. Kittler, and F. Roli, Eds., 2005, vol. 3541, pp. 326-335.
64. M. Muhlbaier, A. Topalis, and R. Polikar, "Incremental learning from unbalanced data," in Proc. IEEE Int. Joint Conf. Neural Netw., 2004, vol. 2, pp. 1057-1062.
65. M. Muhlbaier, A. Topalis, and R. Polikar, "Learn++.MT: A new approach to incremental learning," in Lecture Notes in Computer Science, F. Roli, J. Kittler, and T. Windeatt, Eds. Berlin, Germany: Springer-Verlag, 2004, vol. 3077, pp. 52-61.
66. P. J. Boland, "Majority system and the condorcet jury theorem," Statistician, vol. 38, no. 3, pp. 181-189, 1989.
67. A. Gangardiwala and R. Polikar, "Dynamically weighted majority voting for incremental learning and comparison of three boosting based approaches," in Proc. IEEE Int. Joint Conf. Neural Netw., 2005, vol. 2, pp. 1131-1136.
68. ++NC," Jul. 18, 2008 [Online]. Available: http:// www.engineering.rowan.edu/~polikar/Learn++
69. R. Polikar, R. Shinar, L. Udpa, and M. D. Porter, "Artificial intelligence methods for selection of an optimized sensor array for identification of volatile organic compounds," Sens. Actuators B, Chem., vol. 80, no. 3, pp. 243-254, 2001.
70. A. Asuncion and D. J. Newman, UCI Repository of Machine Learning Database, Irvine, CA, Jul. 18, 2008 [Online]. Available: http:// www.ics.uci.edu/~mlearn/MLReporitory.html
71. M. Muldbaier and R. Polikar, "An ensemble approach for incremental learning in nonstationery environments," in Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2007, vol. 4472, pp. 490-500.
72. M. Karnick, M. Ahiskali, M. Muhlbaier, and R. Polikar, "Learning concept drift in nonstationary environments using an ensemble of classifiers based approach," in Proc. World Congr. Comput. Intell./Int. Joint Conf. Neural Netw., Hong Kong, 2008, pp. 3455-3462.