Novel maximum-margin training algorithms for supervised neural networks

IEEE Transactions on Neural Networks

Volumn 21, Issue 6, 2010, Pages 972-984

  Ludwig, Oswaldo ^a   Nunes, Urbano ^a  

^a UNIVERSITY OF COIMBRA   (Portugal)

Author keywords

Information theory; Maximal margin (MM) principle; Multilayer perceptron; Pattern recognition; Supervised learning

Indexed keywords

ADAPTIVE LEARNING RATES; BENCHMARK DATA; BINARY CLASSIFIERS; CONSTRAINED OPTIMIZATION PROBLEMS; DATA SETS; ERROR RATE; FISHER DISCRIMINANT ANALYSIS; GRADIENT DESCENT; HIDDEN LAYERS; MAXIMAL-MARGIN (MM) PRINCIPLE; MULTI LAYER PERCEPTRON; NEURAL MODELS; OBJECTIVE FUNCTIONS; REAL-WORLD PROBLEM; ROC CURVES; SINGLE PROCESS; SPACE COMPLEXITY; STATISTICAL DISTRIBUTION; SUPERVISED NEURAL NETWORKS; SUPPORT VECTOR; TIME AND SPACE; TIME COMPLEXITY; TRAINING ALGORITHMS; TRAINING FRAMEWORK; TRAINING METHODS;

ADAPTIVE ALGORITHMS; BACKPROPAGATION; CLASSIFIERS; CONSTRAINED OPTIMIZATION; DISCRIMINANT ANALYSIS; ELECTRIC LOADS; FISHER INFORMATION MATRIX; GEOMETRY; INFORMATION THEORY; LEARNING ALGORITHMS; MULTILAYERS; PATTERN RECOGNITION; SUPPORT VECTOR MACHINES; VECTOR SPACES;

NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; FEEDBACK SYSTEM; HUMAN; INFORMATION SCIENCE; LEARNING; METHODOLOGY; PHYSIOLOGY; RECEIVER OPERATING CHARACTERISTIC;

ALGORITHMS; COMPUTER SIMULATION; FEEDBACK; HUMANS; INFORMATION THEORY; LEARNING; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED; ROC CURVE;

EID: 77953123103     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2010.2046423     Document Type: Article

References (27)

1. H. Demuth and M. Beale, Neural Network Toolbox User's Guide: For Use with MATLAB, 4.0 ed. Natick, MA: The MathWorks Inc., 2000.
2. U. Franke and S. Heinrich, "Fast obstacle detection for urban traffic situations," IEEE Trans. Intell. Transp. Syst., vol.3, no.3, pp. 173-181, Sep. 2002.
3. I. Guyon, S. Gunn, M. Nikravesh, and L. Zadeh, Feature Extraction, Foundations and Applications. Heidelberg, Germany: Springer-Verlag, 2006.
4. K. Hornik, M. Stinchcombe, and H. White, "Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks," Neural Netw., vol.3, no.5, pp. 551-560, 1990.
5. R. Ilin, R. Kozma, and P. J. Werbos, "Beyond feedforward models trained by backpropagation: A practical training tool for a more efficient universal approximator," IEEE Trans. Neural Netw., vol.19, no.6, pp. 929-937, Jun. 2008.
6. A. K. D. Jayadeva and S. Chandra, "Binary classification by SVM based tree type neural network," in Proc. Int. Conf. Neural Netw., Honolulu, HI, May 2002, vol.3, pp. 2773-2778.
7. C. T. Kim and J. J. Lee, "Training two-layered feedforward networks with variable projection method," IEEE Trans. Neural Netw., vol.19, no.2, pp. 371-375, Feb. 2008.
8. C. Liu, "Gabor-based kernel PCA with fractional power polynomial models for face recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol.26, no.5, pp. 572-581, May 2004.
9. O. Ludwig, P. C. Gonzalez, and A. C. Lima, "Optimization of ANN applied to non-linear system identification," in Proc. 25th IASTED Int. Conf. Model. Identif. Control, Lanzarote, Feb. 2006, pp. 402-407.
10. O. Ludwig, U. Nunes, R. Araujo, L. Schnitman, and H. A. Lepikson, "Applications of information theory, genetic algorithms, and neural models to predict oil flow," Commun. Nonlinear Sci. Numer. Simul., vol.14, no.7, pp. 2870-2885, 2009.
11. R. M. Neal and J. Zhang, "High dimensional classification with Bayesian neural networks and Dirichlet diffusion trees," in Studies in Fuzziness and Soft Computing. Berlin, Germany: Springer-Verlag, 2006, vol.207, pp. 265-296.
12. T. Nishikawa and S. Abe, "Maximizing margins of multilayer neural networks," in Proc. 9th Int. Conf. Neural Inf. Process., Singapore, Nov. 2002, vol.1, pp. 322-326.
13. G. Ou and Y. L. Murphey, "Multi-class pattern classification using neural networks," Pattern Recognit., vol.40, no.1, pp. 4-18, Jan. 2007.
14. H. Peng, F. Long, and C. Ding, "Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy, " IEEE Trans Pattern Anal. Mach. Intell., vol.27, no.8, pp. 1226-1238, Aug. 2005.
15. J. C. Platt, "Sequential minimal optimizer: A fast algorithm for training support vector machines," Microsoft Research, Tech. Rep. MSR-TR-98-114, 1998.
16. G. Ratsch, T. Onoda, and K. Müller, "Soft margins for AdaBoost," in Machine Learning. Norwell, MA: Kluwer, 2001, vol.42, pp. 287-320.
17. A. Ruiz and P. E. Lopez-de-Teruel, "Nonlinear kernel-based statistical pattern analysis," IEEE Trans. Neural Netw., vol.12, no.1, pp. 16-32, Jan. 2001.
18. R. Sexton, R. Dorsey, and J. Johnson, "Optimization of neural networks: A comparative analysis of the genetic algorithm and simulated annealing," Eur. J. Oper. Res., vol.114, no.3, pp. 589-601, May 1999.
19. I. Steinwart, "Sparseness of support vector machines," J. Mach. Learn. Res., vol.4, no.12, pp. 1071-1105, 2003.
20. N. K. Treadgold and T. D. Gedeon, "Exploring constructive cascade networks," IEEE Trans. Neural Netw., vol.10, no.6, pp. 1335-1350, Nov. 1999.
21. I. W. Tsang, J. T. Kwok, and P. M. Cheung, "Core vector machines: Fast SVM training on very large data sets," J. Mach. Learn. Res., vol.6, pp. 363-392, Apr. 2005.
22. V. Vapnik, The Nature of Statistical Learning Theory. London, U.K.: Springer-Verlag, 1995.
23. V. Vapnik and A. Chervonenkis, "On the uniform convergence of relative frequencies of events to their probabilities," Theory Probab. Appl., vol.16, no.2, pp. 264-280, 1971.
24. J. Yang, A. F. Frangi, J. U. Yang, D. Zhang, and Z. Jin, "KPCA plus LDA: A complete kernel Fisher discriminant framework for feature extraction and recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol.27, no.2, pp. 230-244, Feb. 2005.
25. J. Yaochu, T. Okabe, and B. Sendhoff, "Neural network regularization and ensembling using multi-objective evolutionary algorithms," in Proc. Congr. Evol. Comput., Portland, OR, Jun. 2004, DOI: 10.1109/ CEC.2004.1330830 .
26. S. Young and T. Downs, "CARVE: A constructive algorithm for real-valued examples," IEEE Trans. Neural Netw., vol.9, no.6, pp. 1180-1190, Nov. 1998.
27. K. Zhang, I. W. Tsang, and J. T. Kwok, "Maximum margin clustering made practical," IEEE Trans. Neural Netw., vol.20, no.4, pp. 583-596, Apr. 2009.