The Q-norm complexity measure and the minimum gradient method: A novel approach to the machine learning structural risk minimization problem

IEEE Transactions on Neural Networks

Volumn 19, Issue 8, 2008, Pages 1415-1430

  Vieira, Douglas A G ^a   Takahashi, Ricardo H C ^a   Palade, Vasile ^b   Vasconcelos, J A ^a   Caminhas, W M ^a  

^a FEDERAL UNIVERSITY OF MINAS GERAIS   (Brazil)

^b UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Complexity measure; Multiobjective training algorithms; Neural networks; Parallel layer perceptron (PLP); Regularization methods; Structural risk minimization (SRM)

Indexed keywords

ARTIFICIAL INTELLIGENCE; CHLORINE COMPOUNDS; EDUCATION; GRADIENT METHODS; LEARNING SYSTEMS; NEURAL NETWORKS; RISK ASSESSMENT; ROBOT LEARNING;

BI-OBJECTIVE OPTIMIZATION; COMPLEXITY MEASURE; COMPLEXITY MEASURES; LAYER PERCEPTRON; MACHINE LEARNING PROBLEM; MACHINE-LEARNING; MULTIOBJECTIVE TRAINING ALGORITHMS; NETWORK TRAINING; PARALLEL LAYER PERCEPTRON (PLP); QUASI-CONVEX FUNCTIONS; REGULARIZATION METHODS; STRUCTURAL RISK MINIMIZATION; STRUCTURAL RISK MINIMIZATION (SRM); TRAINING ERRORS;

OPTIMIZATION;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; METHODOLOGY; THEORETICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED;

EID: 49649091536     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2008.2000442     Document Type: Article

References (68)

1. V. N. Vapnik, Statistical Learning Theory. New York: Wiley, 1998.
2. C. Cortes and V. Vapnik, "Support vector networks," Mach. Learn. vol. 20, pp. 273-279, 1995.
3. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines (and Other Kernel-Based Learning Methods). Cambridge, U.K.: Cambridge Univ. Press, 2000.
4. V. N. Vapnik, The Nature of Statistical Learning Theory (Statistics for Engineering and Information Science), 2nd ed. New York: Springer-Verlag, Sep. 2001.
5. J. Shawe-Taylor and P. L. Bartlett, "Structural risk minimization over data-dependent hierarchies," IEEE Trans. Inf. Theory, vol. 44, no. 5, pp. 1926-1940, Sep. 1998.
6. G. E. Hinton, "Connections learning procedures," Artif. Intell., vol. 40, pp. 185-234, 1989.
7. A. N. Tikhonov, "On solving ill-posed problem and the method of regularization," Doklady Akademii Nauk USSR, vol. 153, pp. 501-504, 1963.
8. A. N. Tikhonov and V. Y. Arsenin, Solution of Ill-Posed Problems. Washington, DC: W. H. Winston, 1977.
9. S. Haykin, Neural Networks: A Comprehensive Fundation, 2nd ed. Englewood Cliffs, NJ: Prentice-Hall, 1999.
10. R. A. Teixiera, A. P. Braga, R. H. C. Takahashi, and R. R. Saldanha, "Improving generalization of MLPs with multi-objective optimization," Neurocomputing, vol. 35, no. 1-4, pp. 189-194, 2000.
11. M. A. Costa, A. P. Braga, B. R. Menezes, R. A. Teixiera, and G. G. Parma, "Training neural networks with a multi-objective sliding mode control algorithm," Neurocomputing, vol. 51, pp. 467-473, 2003.
12. W. M. Caminhas, D. A. G. Vieira, and J. A. Vasconcelos, "Parallel layer perceptron," Neurocomputing, vol. 55, no. 3-4, pp. 771-778, Oct. 2003.
13. D. A. G. Vieira, W. M. Caminhas, and J. A. Vasconcelos, "Controlling the parallel layer perceptron complexity using a multiobjective learning algorithm," Neural Comput. Appl., vol. 16, no. 4-5, May 2006, DOI: 10.1007/s00521-006-0052-z.
14. F. Girosi, M. Jones, and T. Poggio, "Regularization theory and neural networks architectures," Neural Comput., vol. 7, pp. 219-269, 1995.
15. B. Schölkopf, S. Mika, C. J. C. Burges, P. Knirsch, K.-R. Müller, G. Rätsch, and A. J. Smola, "Input space vs. feature space in Kernel-based methods," IEEE Trans. Neural Netw., vol. 10, no. 5, pp. 1000-1017, Sep. 1999.
16. M. J. Kearns and R. E. Schapire, "Efficient distribution-free learning of probabilistic concepts (abstract)," in Proc. 3rd Annu. Workshop Comput. Learn. Theory, 1990, pp. 382-391.
17. P. L. Bartlett, "The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network," IEEE Trans. Inf. Theory, vol. 44, no. 2, pp. 525-536, Mar. 1998.
18. V. N. Vapnik, "Principles of structural risk minimization for learning theory," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1992, vol. 4, pp. 831-838.
19. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning Internal Representations by Error Propagation, ser. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: Bradford Books (MIT Press), 1986, vol. 1.
20. G. Cybenko, "Approximation by superpositions of a Sigmoid function," Math. Control Signals Syst., vol. 2, pp. 303-314, 1989.
21. K. Funahashi, "On the approximate realization of continuous mappings by neural networks," Neural Netw. Signals Syst., vol. 2, pp. 183-192, 1989.
22. S. E. Fahlman, "Fast-learning variations on back-propagation: An empirical study," in Proc. Connectionist Models Summer School, D. Touretzky, G. Hinton, and T. Senjnowski, Eds., San Mateo, CA, 1988, pp. 38-51.
23. M. T. Hangan and M. B. Menjah, "Training feedforward network with the Marquardt algorithm," IEEE Trans. Neural Netw., vol. 5, no. 6, pp. 989-993, Nov. 1994.
24. P. L. Bartlett, "For valid generalization the size of the weights is more important than the size of the network," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1997, vol. 9, pp. 134-141.
25. V. V. Vasin, "Relationship of several variational methods for approximate solutions of ill-posed problems," Math Notes, vol. 7, pp. 161-166, 1970.
26. D. S. Broomhead and D. Lowe, "Multivariable functional interpolation and adaptive networks," Complex Syst., vol. 2, pp. 321-355, 1989.
27. J. Park and I. W. Sandberg, "Universal approximation using radial-basis-function networks," Neural Comput., vol. 3, pp. 246-257, 1991.
28. A. R. Barron, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 930-945, May 1993.
29. P. Andras, "The equivalence of support vector machine and regularization neural networks," Neural Process. Lett., vol. 15, no. 2, pp. 97-104, 2002.
30. D. S. Yeung, W. W. Y. Ng, D.Wang, E. C. C. Tsang, and X.-Z. Wang, "Localized generalization error model and its application to architecture selection for radial basis function neural network," IEEE Trans. Neural Netw., vol. 18, no. 5, pp. 1294-1305, Sep. 2007.
31. J. E. Moody and T. S. Rögnvaldsson, "Smoothing regularizers for projective basis function networks," in Advances in Neural Information Processing Systems, M. C. Mozer, M. I. Jordan, and T. Petsche, Eds. Cambridge, MA: MIT Press, 1997, vol. 9, pp. 585-605.
32. H. Drucker and Y. LeCun, "Improving generalization performance using double back-propagation," IEEE Trans. Neural Netw., vol. 3, no. 6, pp. 991-997, Nov. 1992.
33. C. M. Bishop, "Curvature driven smoothing: A learning algorithm for feedforward networks," IEEE Trans. Neural Netw., vol. 4, no. 5, pp. 882-884, Sep. 1993.
34. D. A. G. Vieira, W. M. Caminhas, and J. A. Vasconcelos, "Extracting sensitivity information of electromagnetic devices models from a modified ANFIS topology," IEEE Trans. Magn., vol. 40, no. 2, pp. 1180-1183, Mar. 2004.
35. Matlab Toolboxes. The Mathworks, Natick, MA [Online]. Available: www.mathworks.com
36. K. Pelckmans, J. A. K. Suykens, and B. De Moor, "Convex approach to validation-based learning of the regularization constant," IEEE Trans. Neural Netw., vol. 18, no. 3, pp. 917-920, May 2007.
37. D. Gabor, W. Wildes, and R. Woodcock, "A universal nonlinear filter, predictor and simulator which optimizes itself by a learning process," Proc. Inst. Electr. Eng., vol. 108B, pp. 422-438, 1961.
38. Y. Xia and M. S. Kamel, "A generalized least absolute deviation method for parameter estimation of autoregressive signals," IEEE Trans. Neural Netw., vol. 19, no. 1, pp. 107-118, Jan. 2008.
39. C. J. C. Burges and B. Schölkopf, "Improving the accuracy and speed of support vector machines," in Advances in Neural Information Processing Systems, M. C. Mozer, M. I. Jordan, and T. Petsche, Eds. Cambridge, MA: MIT Press, 1997, vol. 9, p. 375.
40. IDA, "IDA Benchmark Repository Used in Several Boosting, KFD and SVM Papers," Tech. Rep. [Online]. Available: http://ida.first.gmd.de/ raetsch/data/benchmarks.htm
41. K. Muller, S. Mika, G. Ratsh, K. Tsuda, and B. Scholkopf, "An introduction to Kernel-based learning algorithms," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 181-201, Mar. 2001.
42. J. Weston and R. Herbrich, "Adaptive margin support vector machines," in Advances in Large Margin Classifiers, A. Smola, P. Bartlett, B. Scholkopf, and D. Schuurmans, Eds. Cambridge, MA: MIT Press, 2000, pp. 281-295.
43. G. C. Cawley and N. L. C. Talbot, "Efficient leave-one-out cross-validation of kernel Fisher discriminant classifiers," Pattern Recognit., vol. 36, pp. 2585-2592, November 2003.
44. V. Roth, "The generalized LASSO," IEEE Trans. Neural Netw., vol. 15, no. 1, pp. 16-28, Jan. 2004.
45. G. C. Cawley and N. L. C. Talbot, "The evidence framework applied to sparse kernel logistic regression," Neurocomputing, vol. 65, pp. 119-135, March 2005.
46. Q. Tao, G.-W. Wu, Fei-Yue, and J. Wang, "Posterior probability support vector machines for unbalanced data," IEEE Trans. Neural Netw., vol. 16, no. 6, pp. 1561-1573, Nov. 2005.
47. J. Demsar, "Statistical comparisons of classifiers over multiple data sets," J. Mach. Learn. Res., vol. 7, pp. 1-30, 2006.
48. B. Zhang, Is the maximal margin hyperplane special in a feature space? Hewlett-Packards Labs, Palo Alto, CA, Tech. Rep., Apr. 2001.
49. B. Sholkopf, C. Burges, and V. Vapnik, "Extracting support data for a given task," in Proc. Ist Int. Conf. Knowl. Disc. Data Mining, 1995, pp. 252-257.
50. S. Amari and S. Wu, "Improving support vector machine classifier by modifying kernel functions," Neural Netw., vol. 12, pp. 783-789, 1999.
51. K. Duan, S. S. Keerthi, and A. N. Poo, "Evaluation of simple performance measures for tuning SVM hyperparameters," Neurocomputing vol. 51, pp. 41-59, 2003.
52. Y. Tan and J. Wuang, "A support vector machine with a hybrid kernel and minimal Vapnik-Chervonenks dimension," IEEE Trans. Knowl. Data Eng. vol. 16, no. 4, pp. 385-395, Apr. 2004.
53. M. Yoon, Y. Yun, and H. Nakayama, "A role on total margin support vector machines," in Proc. Int. Joint Conf. Neural Netw., 2003, pp. 2049-2053.
54. G. Baudat and F. Anouar, "Kernel-based methods and function approximation," in Proc. Int. Joint Conf. Neural Netw., Washington, DC, 2001, pp. 1244-1249.
55. E. Lacerda, A. Carvalho, A. P. Braga, and T. B. Ludermir, "Evolutionary radial basis functions for credit assessment," Appl. Intell., vol. 22, no. 3, pp. 167-182, 2005.
56. S. P. Llyod, "Least square quantization in PCM," IEEE Trans. Inf. Theory, vol. IT-28, no. 2, pp. 129-137, Mar. 1982.
57. M. A. Ismail, S. Z. Selim, and S. K. Aror, "Efficient clustering of multidimensional data," in Proc. IEEE Int. Conf. Syst. Man Cybern. 1984, pp. 120-123.
58. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis New York: Wiley-Interscience, 1973.
59. M. A. Ismail and M. S. Kamel, "Multidimensional data clustering utilizing hybrid strategies," Pattern Recognit., vol. 22, pp. 75-89, 1989.
60. J. MacQueen, "Some methods for classification and analysis of multivariate observations," in Proc. 5th Berkeley Symp. Math, 1967, vol. 1, pp. 281-297.
61. C. Chinrungrueng and C. H. Séquin, "Optimal adaptive k-means algorithm with dynamic adjustment of learning rate," IEEE Trans. Neural Netw., vol. 6, no. 1, pp. 157-169, Jan. 1995.
62. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Computational Models of Cognition and Perception. Cambridge, MA: MIT Press, 1986, vol. 1, ch. 8, pp. 319-362.
63. S. E. Fahlman and C. Lebiere, "The cascade-correlation learning architecture," in Advances in Neural Information Processing Systems D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, vol. 2, pp. 524-532.
64. R. Parekh, J. Yang, and V. Honavar, "Constructive neural network learning algorithms for multi-category real-value pattern classification," Dept. Comput. Sci., Iowa State Univ., Ames, IA, Tech. Rep., 1987.
65. B. Scholkopf, Support Vector Learning. New York: Springer-Verlag, 1997.
66. J. S. R. Jang, "ANFIS: Adaptive-network-based fuzzy inference systems," IEEE Trans. Syst. Man Cybern., vol. 23, no. 3, pp. 665-685, May 1993.
67. T. Yamakwa, E. Uchino, T. Miki, and Kusanagi, "A neo fuzzy neuron and it applications to system identification and predictions to system behavior," in Proc. 2nd Int. Conf. Fuzzy Logic Neural Netw., Japan, 1992, pp. 477-483.
68. S. Geman, E. Bienenstock, and R. Doursat, "Neural networks and the bias-variance dilemma," Neural Comput., vol. 4, no. 1, pp. 1-58, 1992.