Pareto-based multiobjective machine learning: An overview and case studies

IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews

Volumn 38, Issue 3, 2008, Pages 397-415

  Jin, Yaochu ^a   Sendhoff, Berhard ^a  

^a HONDA RESEARCH INSTITUTE EUROPE GMBH   (Germany)

Author keywords

Ensemble; Evolutionary multiobjective optimization; Generalization; Machine learning; Multiobjective learning; Multiobjective optimization; Neural networks; Pareto optimization

Indexed keywords

COST FUNCTIONS; EVOLUTIONARY ALGORITHMS; FEATURE EXTRACTION; LEARNING ALGORITHMS; MULTIOBJECTIVE OPTIMIZATION; NEURAL NETWORKS; PARETO PRINCIPLE;

PARETO-OPTIMAL SOLUTIONS; POPULATION-BASED STOCHASTIC SEARCH METHODS;

LEARNING SYSTEMS;

EID: 43449135303     PISSN: 10946977     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCC.2008.919172     Document Type: Review

References (94)

1. E. Alpaydin, Introduction to Machine Learning. Cambridge, MA: MIT Press, 2004.
2. T. M. Mitchell, Machine Learning. New York: McGraw-Hill, 1997.
3. Z. Zoltan, Z. Kalmar, and C. Szepesvari, "Multi-criteria reinforcement learning," in Proc. Int. Conf. Mach. Learn., 1998, pp. 197-205.
4. Y. Jin, B. Sendhoff, and E. Körner, "Evolutionary multi-objective optimization for simultaneous generation of signal-type and symbol-type representations," in Proc. 3rd Int. Conf. Evol. Multi-Criterion Optim. Lecture Notes in Computer Science, vol. 3410. New York: Springer-Verlag, 2005, pp. 752-766.
5. F. Girosi, M. Jones, and T. Poggio, "Regularization theory and neural network architectures," Neural Comput, vol. 7, pp. 219-269, 1995.
6. Y. Jin, "Fuzzy modeling of high-dimensional systems: Complexity reduction and interpretability improvement:" IEEE Trans. Fuzzy Syst. vol. 8, no. 2, pp. 212-221, 2000.
7. 3 fuzzy rule systems from data using evolution strategies," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 29, no. 6. pp. 829-845, Dec. 1999.
8. Y. Liu and X. Yao, "Ensemble learning via negative correlation," Neural Netw., vol. 12, pp. 1399-1404, 1999.
9. C. Cortes and V. Vapink, "Support vector networks," Mach. Learn. vol. 20, pp. 273-297, 1995.
10. B. Olhausen and D. Field, "Sparse coding with an overcomplete basis set: A strategy employed by V1?," Vis. Res., vol. 37, pp. 3311-3325, 1997.
11. T. Fawcett, "ROC graphs: Notes and practical considerations for data mining researchers," HP Labs., Palo Alto, CA, Tech. Rep. HPL-2003-4, 2003.
12. J. Banfield and A. Raftery, "Model-based Gaussian and non-Gaussian clustering," Biometrics, vol. 49, pp. 803-821, 1993.
13. M. H. C. Law, A. P. Topchy, and A. K. Jain, "Multiobjective data clustering," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Piscataway, NJ: IEEE Press, 2004, vol. 2, pp. 424-430.
14. I. Das and J. Dennis. "A closer look at drawbacks of minimizing weighted sum of objectives for Pareto set generation in multicriteria optimization problems," Struct. Optim., vol. 14, no. 1, pp. 63-69, 1997.
15. Y. Jin, M. Olhofer, and B. Sendhoff, "Dynamic weighted aggregation for evolutionary multi-objective optimization: Why does it work and how?," in Proc. Genetic Evol. Comput. Conf. San Mateo, CA: Morgan Kaufmann, 2001, pp. 1042-1049.
16. H. Abbass, "Speeding up back-propagation using multi-objective evolutionary algorithms," Neural Comput., vol. 15, no. 11, pp. 2705-2726, 2003.
17. R. de A. Teixeira, A. Braga, R. Takahashi, and R. Saldanha, "Improving generalization of MLP with multi-objective optimization," Neurocomputing, vol. 35, pp. 189-194, 2000.
18. Y. Jin, Ed., Multi-Objective Machine Learning. New York: Springer-Verlag, 2006.
19. G. Liu and V. Kadirkamanathan, "Learning with multi-objective criteria," in Proc. Inst. Electr. Eng. Conf. Artif. Neural Netw., 1995, pp. 53-58.
20. Y. Matsuyama, "Harmonic competition: A self-organizing multiple criteria optimization," IEEE Trans. Neural Netw., vol. 7, no. 3, pp. 652-668, May 1996.
21. K. Kottathra and Y. Attikiouzel, "A novel multicriteria optimization algorithm for the structure determination of multilayer feedforward neural networks," J. Netw. Comput. Appl., vol. 19, pp. 135-147, 1996.
22. K. Deb, Multi-Objective Optimization Using Evolutionary Algorithms Chichester, U.K.: Wiley, 2001.
23. T. Furukawa, "Parameter identification with weightless regularization," Int. J. Numer. Methods Eng., vol. 52, pp. 219-238, 2001.
24. H. Abbass, "A memetic Pareto approach to artificial neural networks," in Proc. 14th Aust. Joint Conf. Artif. Intell., 2001, pp. 1-12.
25. Y. Jin, T. Okabe, and B. Sendhoff, "Evolutionary multi-objective approach to constructing neural network ensembles for regression," in Applications of Evolutionary Multi-Objective Optimization, C. Coello Coello, Ed. Singapore: World Scientific, 2004, pp. 653-672.
26. Y. Jin, T. Okabe, and B. Sendhoff, "Neural network regularization and ensembling using multi-objective evolutionary algorithms," in Proc. Congr. Evol. Comput. Piscataway, NJ: IEEE Press, Jun. 2004, pp. 1-8.
27. J. Fieldsend and S. Singh, "Pareto multi-objective non-linear regression modelling to aid CAPM analogous forecasting," in Proc. Int. Joint Conf. Neural Netw., 2002, vol. 1, pp. 388-393.
28. J. Fieldsend and S. Singh, "Pareto evolutionary neural networks," IEEE Trans. Neural. Netw., vol. 16, no. 2, pp. 338-354, Mar. 2005.
29. N. Garcia-Pedrajas, C. Hervas-Martinez, and J. Munoz-Perez, "Multi-objective cooperative coevolution of artificial neural networks," Neural Netw., vol. 15, no. 10, pp. 1255-1274, 2002.
30. G. G. Yen and H. Lu, "Hierarchical rank density genetic algorithm for radial-basis function neural network design," in Proc. Congr. Evol. Comput., 2002, vol. 1, pp. 25-30.
31. T. Hatanaka, N. Kondo, and K. Uosaki, "Multi-objective structure selection for radial basis function networks based on genetic algorithm," in Proc. Congr. Evol. Comput., 2003, pp. 1095-1100.
32. J. Bi, "Multi-objective programming in SVMs," in Proc. 20th Int. Conf. Mach. Learn., Washington, DC, 2003, pp. 35-42.
33. C. Igel, "Multi-objective model selection for support vector machines," in Evolution Multi-Criterion Optimization Lecture Notes in Computer Science, vol. 3410. New York: Springer-Verlag, 2005, pp. 534-546.
34. H. Nakayama and T. Asada, "Support vector machines formulated as multi-objective linear programming," in Proc. ICOTA, 2001, pp. 1171-1178.
35. D. Kim, "Structural risk minimization on decision trees using an evolutionary multiobjective optimization," in Proc. Eur. Conf. Genetic Program. Lecture Notes in Computer Science, vol. 3003. New York: Springer-Verlag, 2004, pp. 338-348.
36. E. Bernado-Manssilla and J. Garrell-Guii, "MOLeCS: Using multi-objective evolutionary algorithms for learning," in Proc. EMO 2001 Lecture Notes in Computer Science, vol. 1993. New York: Springer-Verlag, pp. 696-710.
37. S. Wiegand, C. Igel, and U. Handmann, "Evolutionary multiobjective optimization of neural networks fore face detection," Int. J. Comput. Intell., vol. 4, no. 3, pp. 237-253, 2005.
38. Y. Zhang and P. I. Rockett, "Evolving optimal feature extraction using multi-objective genetic programming: A methodology and preliminary study on edge detection," in Proc. Genetic Evol. Comput. Conf., 2005, pp. 795-802.
39. J. Teo and H. Abbass, "Automatic generation of controllers for embodied legged organisms: A Pareto evolutionary multi-objective approach," Evol. Comput., vol. 12, no. 3, pp. 355-394, 2004.
40. M. Luque, O. Cordon, and E. Herrera-Viedma, "A multi-objective genetic algorithm for learning linguistic persistent quries in text retrieval environments," in Multi-Objective Machine Learning, Y. Jin, Ed. New York: Springer-Verlag, 2006, ch. 25, pp. 585-600.
41. Y. Jin, Advanced Fuzzy Systems Design and Applications. Heidelberg, Germany: Physica Verlag, 2003.
42. H. Ishibuchi, I. Murata, and I. Türksen, "Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern recognition," Fuzzy Sets Syst., vol. 89, pp. 135-150, 1997.
43. A. Gomez-Skarleta, F. Jimenez, and J. Ibanez, "Pareto-optimality in fuzzy modeling," in Proc. 6th Eur. Congr. Int. Tech. Soft Comput. 1997, pp. 694-700.
44. T. Suzuki, T. Furuhashi, S. Matsushima, and H. Tsutsui, "Efficient fuzzy modeling under multiple criteria by using genetic algorithms," in Proc. IEEE Int. Conf. Syst., Man, Cybern., 1999, vol. 5, pp. 314-319.
45. H. Ishibuchi, T. Nakashima, and T. Murata, "Three-objective genetics-based machine learning for linguistic rule extraction," Inf. Sci., vol. 136, no. 1-4, pp. 109-133, 2001.
46. K. Tachibana and T. Furuhashi, "A structure identification method of submodels for hierarchical fuzzy modeling using the multiple objective genetic algorithm," Int. J. Intell. Syst., vol. 17, pp. 495-513, 2001.
47. F. Jimenez, G. Sanchez, A. F. Gomez-Skarmeta, H. Roubos, and R. Babuska, "Fuzzy modeling with multi-objective neuro-evolutionary algorithms," in Proc. IEEE Int. Conf. Syst., Man, Cybern, 2002, vol. 3, pp. 253-258.
48. H. Wang, S. Kwong, Y. Jin, W. Wei, and K. Man, "Agent-based evolutionary approach to interpretable rule-based knowledge extraction," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 35, no. 2, pp. 143-155, May 2005.
49. H. Wang, S. Kwong, Y. Jin, W. Wei, and K. Man, "Multi-objective hierarchical genetic algorithm for interpretable fuzzy rule absed knowledge extraction," Fuzzy Sets Syst., vol. 149, no. 1. pp. 149-186, 2005.
50. U. Markowska-Kaczmar and P. Wnuk-Liplnski, "Rule extraction from neural networks with Pareto optimization," in Artificial Intelligence and Soft Computing. Berlin, Germany: Springer-Verlag, 2004, pp. 450-455.
51. L. Hansen and P. Salammon, "Neural network ensembles," IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 10, pp. 993-1101, Oct. 1990.
52. A. Chandra and X. Yao, "DIVACE: Diverse and accurate ensemble learning algorithm," in Proc. 5th Int. Conf. Intell. Data Eng. Autom. Learn. 2004, pp. 619-625.
53. D. Opitz and J. Shavlik, "Generating accurate and diverse members of a neural network ensemble," in Advances in Neural Information Processing Systems 8. Cambridge, MA: MIT Press, 1996, pp. 535-541.
54. G. Brown, J. Wyatt, R. Harris, and X. Yao, "Diversity creation methods: A survey and categorisation," J. Inf. Fusion, vol. 6, no. 1, pp. 5-20, 2005.
55. B. Rosen, "Ensemble learning using decorrelated neural networks," Connection Sci., vol. 88, pp. 373-384, 1996.
56. A. Chandra and X. Yao, "Evolving hybrid ensembles of learning machines for better generalisation," Neurocomputing, vol. 69, pp. 686-700, 2006.
57. H. Abbass, "Pareto neuro-evolution: Constructing ensemble of neural networks using multi-objective optimization," in Proc. Congr. Evol. Comput, Dec. 2003, pp. 2074-2080.
58. N. Garcia-Pedrajas, C. Hervas-Martinez, and J. Munoz-Perez, "COVNET: A cooperative coevolutionary model for evolving artificial neural networks," IEEE Trans. Neural Netw., vol. 14, no. 3, pp. 575-596, May 2003.
59. L. Kuncheva and C. Whitaker, "Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy," Mach. Learn., vol. 51, no. 2, pp. 181-207, 2003.
60. T. Hatanaka, N. Kondo, and K. Uosaki, "Multiobjective structure selection for RBF networks and its application to nonlinear system design," in Multi-Objective Machine Learning, Y. Jin, Ed. New York: Springer-Verlag, 2006, ch. 21, pp. 491-505.
61. H. Ishibuchi and T. Yamamoto, "Evolutionary multi-objective optimization for generating an ensemble of fuzzy rule-based classifiers," in Proc. Genetic Evol. Comput. Conf. Lecture Notes in Computer Science, vol. 2723, 2003, pp. 1077-1088.
62. M. Kupinski and M. Anastasio, "Multiobjective genetic optimization of diagnostic classifiers with implementations for generating receiver operating characteristic curves," IEEE Trans. Med. Imag., vol. 18, no. 8, pp. 675-685, Aug. 1999.
63. J. Horn and N. Nafpliotis, "Multiobjective optimization using the niched Pareto genetic algorithms," IlliGAL, Univ. Illinois at Urbana-Champaign, Urbana-Champaign, Tech. Rep. 93005, 1993.
64. L. Gräning, Y. Jin, and B. Sendhoff, "Generalization improvement in multi-objective learning," in Proc. Int. Joint Conf. Neural Netw. 2006, pp. 9893-9900.
65. R. Everson and J. Fieldsend, "Multi-class ROC analysis from a multi-objective optimisation perspective," Pattern Recognit. Lett. vol. 27, pp. 918-927, 2006.
66. S. Park, D. Nam, and C. H. Park, "Design of a neural controller using multi-objective optimization for nonminimum phase systems," in Proc. IEEE Int. Conf. Fuzzy Sets Syst., 1999, vol. I, pp. 533-537.
67. O. Cordon, F. Herrera, M. del-Jesus, and P. Villar, "A multi-objective genetic algorithm for feature selection and granularity learning in fuzzy-rule based classification systems," in Proc. Joint 9th IFSA World Congr. 20th NAFIPS Int. Conf., 2001, vol. 3, pp. 1253-1258.
68. C. Emmanouilidis, A. Hunter, J. MacIntyre, and C. Cox, "Selecting features in neurofuzzy modelling by multiobjective genetic algorithms," in Proc. Int. Joint Conf. Neural Netw., 1999, pp. 749-754.
69. L. Oliveira, R. Sabourin, F. Bortolozz, and C.Suen, "Feature seelction for ensembles: A hierarchical multi-objective genetic algorithm approach," in Proc. 7th Int. Conf. Anal. Recognit., 2003, pp. 676-680.
70. Y. Kim, W. Street, and F. Menczer, "Evolutionary model selection in unsupervised learning," Intell. Data Anal., vol. 6. pp. 531 556, 2002.
71. J. Handl and J. Knowles, "Exploiting the tradeoff - The benefits of multiple objectives in data clustering," in Evolutionary Multi-Criterion Optimization Lecture Notes in Computer Science, vol. 3410. New York: Springer-Verlag, 2005, pp. 547-560.
72. R. Tibshirani, G. Walther, and T. Hastie, "Estimating the number of clusters in a dataset via the gap statistics," J. R. Stat. Soc. B, vol. 63, pp. 411-423, 2001
73. X. Yao, "Evolving artificial neural networks," Proc. IEEE, vol. 87, no. 9, pp. 1423-1447, Sep. 1999.
74. C. Igel and M. Husken, "Empirical evaluation of the improved Rprop learning algorithm," Neurocomputing, vol. 55, no. C, pp. 105-123, 2003.
75. M. Riedmiller and H. Braun, "A direct adaptive method for faster back-propgation learning: The RPROP algorithm," in Proc. IEEE Int. Conf. Neural Netw., 1993, vol. 1, pp. 586-591.
76. K. Deb, S. Agrawal, A. Pratap, and T. Meyarivan, "A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II," in Proc. Parallel Problem Solving Nat., 2000, vol. VI, pp. 849-858.
77. L. Prechelt, "PROBENI - A set of neural network benchmark problems and benchmarking rules," Fakultät Inf., Univ. Karlsruhe, Karlsruhe, Germany, Tech. Rep., 1994.
78. S. Thrun, "Extracting rules from artificial neural networks with distributed representation," in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 1994, pp. 505-512.
79. S. Kamruzzaman and M. Islami, "Extraction of symbolic rules from artificial neural networks," Trans. Eng., Comput Technol., vol. 10, pp. 271-277, 2005.
80. R. Setiono, "Generating concise and accurate classification rules for breast cancer dignosis," Artif. Intell. Med., vol. 18, pp. 205-219, 2000.
81. K. Burnham and D. Anderson, Model Selection and Multimodel Inference New York: Springer-Verlag, 2002.
82. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. New York: Springer-Verlag, 2001.
83. Y. Yin and B. Sendhoff, "Alleviating catastrophic forgetting via multi-objective learning," in Proc. Int. Joint Conf. Neural Netw. 2006, pp. 6367-6374.
84. S. Amari, "A universal theorem on learning curves," Neural Netw. vol. 6, no-2, pp. 161-166, 1993.
85. T. Fine and S. Mukherjee, "Paramater convergence and learning curves for neural networks," Neural Comput., vol. 11, pp. 747-769, 1999.
86. S. Louis and G. Rawlins, "Pareto optimality, GA-easiness and deception," in Proc. Int. Conf. Genetic Algorithms, 1993, pp. 118-123.
87. J. Knowles, R. Watson, and D. Come, "Reducing local optima in single-objective problems by multi-objectivization," in Proc. 1st Int. Conf. Evol. Multi-Criterion Optim. Lecture Notes in Computer Science, vol. 1993. New York: Springer-Verlag, 2001, pp. 269-283.
88. V. de Angulo and C. Torras, "On-line learning with minimal degradation in feedforward networks," IEEE Trans. Neural Netw., vol. 6, no. 3, pp. 657-668, May 1995.
89. S. Wan and L. Banta, "Parameter incremental learning algorithm for neural. networks," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1424-1438, Nov. 2006.
90. M. Frean and A. Robins, "Catastrophic forgetting in simple networks: An analysis of the pseudo-rehearsal solution," Netw.: Comput. Neural Syst., vol. 10, pp. 227-236, 1999.
91. R. Adams. L. Calcraft, and N. Davey, "Connectivity in real and evolved associative memories," in Proc. Brain inspired Cognit. Syst., 2006, pp. 153-159.
92. O. Sporns and G. Tononi, "Classes of network connectivity and dynamics," Complexity, vol. 7, pp. 28-38, 2002.
93. Y. Yin, R. Wen, and B. Sendhoff, "Evolutionary multi-objective optimization of spiking neural networks," in Proc. Int. Conf. Artif. Neura Netw. (ICANN) Part I, Lecture Notes in Computer Science, vol. 4668, New York: Springer-Verlag, 2007, pp. 370-379.
94. W. Maass and C. Bishop, Pulsed Neural Networks. Cambridge, MA: MIT Press, 1999.