Sensitivity analysis for selective learning by feedforward neural networks

Fundamenta Informaticae

Volumn 45, Issue 4, 2001, Pages 295-328

  Engelbrecht, Andries P ^a  

^a UNIVERSITY OF PRETORIA   (South Africa)

Author keywords

Decision Boundaries; Dynamic Pattern Selection; Feedforward Neural Networks; Pattern Informativeness; Sensitivity Analysis

Indexed keywords

LEARNING SYSTEMS; PERTURBATION TECHNIQUES; SENSITIVITY ANALYSIS;

SELECTIVE LEARNING;

FEEDFORWARD NEURAL NETWORKS;

EID: 0035270526     PISSN: 01692968     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (63)

1. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Internal Representation by Error Propagation, in: Parallel Distributed Processing (D.E. Rumelhart, J.L. McClelland, Eds.), Vol. 1, MIT Press, Cambridge, Mass, 1986.
2. Denoeux, T., Lengellé, R.: Initializing Back Propagation Networks using Prototypes, Neural Networks, 6(3), 1993, 351-363.
3. Wesseles, L.F.A., Barnard, E.: Avoiding False Local Minima by Proper Initialization of Connections, IEEE Transactions on Neural Networks, 3(6), 1992, 899-905.
4. Magoulas, G.D., Vrahatis, M.N., Androulakis, G.S.: Effective Backpropagation Training with Variable Stepsize, Neural Networks, 10(1), 1997, 69-82.
5. Salomon, R., Van Hemmen, J.L.: Accelerating Backpropagation through Dynamic Self-Adaptation, Neural Networks, 9(4), 1996, 589-601.
6. Weir, M.K.: A Method for Self-Determination of Adaptive Learning Rates in Back Propagation, Neural Networks, 1990, 371-379.
7. Yu, X-H., Chen, G-A.: Efficient Backpropagation Learning using Optimal Learning Rate and Momentum, Neural Networks, 10(3), 1997, 517-527.
8. Cibas, T., Fogelman Soulié, F., Gallinari, P., Raudys, S.: Variable Selection with Neural Networks, Neurocomputing, 12, 1996, 223-248.
9. Engelbrecht, A.P., Cloete, I.: A Sensitivity Analysis Algorithm for Pruning Feedforward Neural Networks, IEEE International Conference in Neural Networks, Washington, Vol. 2, 1996, 1274-1277.
10. Fletcher, L., Katkovnik, V., Steffens, F.E., Engelbrecht, A.P.: Optimizing the Number of Hidden Nodes of a Feedforward Artificial Neural Network, IEEE World Congress on Computational Intelligence, International Joint Conference on Neural Networks, Anchorage, Alaska, 1998, 1608-1612.
11. Hassibi, B., Stork, D.G., Wolff, G.: Optimal Brain Surgeon: Extensions and Performance Comparisons, in: Advances in Neural Information Processing Systems (J.D. Cowan, G. Tesauro, J. Alspector, Eds), Vol. 6, 1994, 263-270.
12. Le Cun, Y., Denker, J.S., Solla, S.A.: Optimal Brain Damage, Advances in Neural Information Processing systems (D. Touretzky, Ed), Vol. 2, 1990, 598-605.
13. Zurada, J.M., Malinowski, A., Usui, S.: Perturbation Method for Deleting Redundant Inputs of Perceptron Networks, Neurocomputing, 14, 1997, 177-193.
14. Hirose, Y., Yamashita, K., Hijiya, S.: Back-Propagation Algorithm which Varies the Number of Hidden Units, Neural Networks, 4, 1991, 61-66.
15. Kwok, T-Y., Yeung, D-Y.: Constructive Feedforward Neural Networks for Regression Problems: A Survey, Technical Report HKUST-CS95-43, Department of Computer Science, The Hong Kong University of Science & Technology, 1995.
16. Lim, S-F., Ho, S-B.: Dynamic Creation of Hidden Units with Selective Pruning in Backpropagation, World Congress on Neural Networks, Vol. 3, 1994, 492-497.
17. Battiti, R.: First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method, Neural Computation, 4, 1992, 141-166
18. Engelbrecht, A.P., Ismail, A.: Training Product Unit Neural Networks, Stability and Control: Theory and Applications, 2(1-2), 1999, 59-74.
19. Møller, M.F.: A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning, Neural Networks, 6, 1993, 525-533.
20. Rosen, B.E., Goodwin, J.M.: Optimizing Neural Networks using Very Fast Simulated Annealing, Neural, Parallel & Scientific Computations, 1997.
21. Tang, Z., Koehler, G.J.: Deterministic Global Optimal FNN Training Algorithm, Neural Networks, 7(2), 1994, 301-311.
22. Van den Bergh, F., Engelbrecht, A.P.: Cooperative Learning in Neural Networks using Particle Swarm Optimizers, South Africam Computer Journal, 26, 2000, 84-90.
23. Engelbrecht, A.P., Cloete, I., Geldenhuys, J., Zurada, J.M.: Automatic Scaling using Gamma Learning for Feedforward Neural Networks, International Workshop on Artificial Neural Networks, Torremolinos, Spain, in: "From Natural to Artificial Neural Computing" in the series Lecture Notes in Computer Science (J. Mira, F. Sandoval, Eds), Vol. 930, 1995, 374-381.
24. Fletcher, G.P., Hinde, C.J.: Learning the Activation Function for the Neurons in Neural Networks, International Conference on Artificial Neural Networks, Vol. 1, 1994, pp 611-614.
25. Zurada, J.: Lambda Learning Rule for Feedforward Neural Networks, Proceedings of the IEEE International Conference in Neural Networks, San Francisco, 1992.
26. Abu-Mostafa, Y.S.: The Vapnik-Chervonenkis Dimension: Information versus Complexity in Learning, Neural Computation, Vol. 1, 1989, 312-317.
27. Cohn, D., Tesauro, G.: Can Neural Networks do Better than the Vapnik-Chervonenkis Bounds?, Advances in Neural Information Processing Systems (R. Lippmann, J. Moody, D.S. Touretzky, Eds), Vol. 3, 1991, 911-917.
28. Hole, A.: Vapnik-Chervonenkis Generalization Bounds for Real Valued Neural Networks, Neural Computation, 8, 1996, 1277-1299.
29. Opper, M.: Learning and Generalization in a Two-Layer Neural Network: The Role of the Vapnik-Chervonenkis Dimension, Physical Review Letters, 72(13), 1994, 2133-2166.
30. Gu, H., Takahashi, H.: Estimating Learning Curves of Concept Learning, Neural Networks, 10(6), 1997, 1089-1102.
31. Röbel, A.: The Dynamic Pattern Selection Algorithm: Effective Training and Controlled Generalization of Backpropagation Neural Networks, Technical Report, Institute für Angewandte Informatik, Technische Universität Berlin, 1994, 497-500.
32. Zhang, B-T.: Accelerated Learning by Active Example Selection, International Journal of Neural Systems, 5(1), 1994, 67-75.
33. Sung, K.K., Niyogi, P.: A Formulation for Active Learning with Applications to Object Detection, AI Memo No 1438, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1996.
34. Lange, R., Männer, R.: Quantifying a Critical Training Set Size for Generalization and Overfitting using Teacher Neural Networks, International Conference on Artificial Neural Networks, Vol. 1, 1994, 497-500.
35. Lange, S., Zeugmann, T.: Incremental Learning from Positive Data, Journal of Computer and System Sciences, 53, 1996, 88-103.
36. Cohn, D., Atlas, L., Ladner, R.: Improving Generalization with Active Learning, Machine Learning, 15, 1994, 201-221.
37. Hunt, S.D., Deller J.R. (jr): Selective Training of Feedforward Artificial Neural Networks using Matrix Perturbation Theory, Neural Networks, 8(6), 1995, 931-944.
38. Cohn, D.A.: Neural Network Exploration using Optimal Experiment Design, AI Memo No 1491, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1994.
39. Seung, H.S., Opper, M., Sompolinsky, H.: Query by Committee, Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory, 1992, 287-299.
40. Barnard, E.: Performance and Generalization of the Classification Figure of Merit Criterion Function, IEEE Transactions on Neural Networks, 2(2), 1991, 322-325.
41. Hampshire, J.B., Waibel, A.H.: A Novel Objective Function for Improved Phoneme Recognition using Time-Delay Neural Networks, IEEE Transactions on Neural Networks, 1(2), 1990, 216-228.
42. Ohnishi, N., Okamoto, A., Sugiem, N.: Selective Presentation of Learning Samples for Efficient Learning in Multi-Layer Perceptron, International Joint Conference on Neural Networks Vol. 1, 1990, 688-691.
43. Kohara, K.: Selective Presentation Learning for Forecasting by Neural Networks, International Workshop on Applications of Neural Networks in Telecommunications, Stockolm, Sweden, 1995, 316-323.
44. Slade, P., Gedeon, T.D.: Bimodal Distribution Removal, International Workshop on Artificial Neural Networks, in: "New Trends in Neural Computation," in the series Lecture Notes in Computer Science (J. Mira, J. Cabestany, A. Prieto, Eds), Springer-Verlag, Berlin, 1993, 249-254.
45. Gedeon, T.D., Wong, P.M., Harris, D.: Balancing Bias and Variance: Network Topology and Pattern Set Reduction Techniques, International Workshop on Artificial Neural Networks, in: "From Natural to Artificial Neural Computing," in the series Lecture Notes in Computer Science (J. Mira, F. Sandoval, Eds), Vol. 930, 1995, 551-558.
46. Ludik, J.: Training, Dynamics, and Complexity of Architecture-Specific Recurrent Neural Networks, PhD Thesis, Department of Computer Science, University of Stellenbosch, February 1995.
47. Drucker, H.: Boosting using Neural Networks, in: Combining Artificial Neural Nets, Perspectives in Neural Computing (A. Sharkey, Ed), Springer, 1999, 51-78.
48. Quinlan, J.R.: C4.5: Programs for Machine Learning, Morgan Kaufmann, 1993.
49. Stitson, M.O., Weston, J.A.E., Gammerman, A., Vovk, V., Vapnik, V.: Theory of Support Vector Machines, Technical Report CSD-TR-96-17, Department of Computer Science, Royal Holloway University of London, 1996.
50. MacKay, D.J.C.: Bayesian Methods for Adaptive Models, PhD Thesis, California Institute of Technology, 1992.
51. Fukumizu, K.: Active Learning in Multilayer Perceptrons, Advances in Neural Information Processing Systems (D.S. Touretzky, M.C. Mozer, M.E. Hasselmo, Eds), Vol. 8, 1996.
52. Plutowski, M., White, H.: Selecting Concise Training Sets from Clean Data, IEEE Transactions on Neural Networks, 4(2), 1993, 305-318.
53. Ahmad, S., Tesauro, G.: Scaling and Generalization in Neural Networks: A Case Study, Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1989, 3-10.
54. Baum, E.B.: Neural Net Algorithms that Learn in Polynomial Time from Examples and Queries, IEEE Transactions on Neural Networks, 2(1), 1991, 5-19.
55. Hwang, J-N., Choi, J.J., Oh, S., Marks, R.J. II: Query-Based Learning Applied to Partially Trained Multilayer Perceptrons, IEEE Transactions on Neural Networks, 2(1), 1991, 131-136.
56. Engelbrecht, A.P., Cloete, I.: Selective Learning using Sensitivity Analysis, IEEE World Congress on Computational Intelligence, International Joint Conference on Neural Networks, Anchorage, Alaska, 1998, 1150-1155.
57. Engelbrecht, A.P., Cloete, I.: Feature Extraction from Feedforward Neural Networks using Sensitivity Analysis, International Conference on Systems, Signals, Control, Computers, Durban, South Africa, Vol. 2, 1998, 221-225.
58. Engelbrecht, A.P.: Sensitivity Analysis for Decision Boundaries, Neural Processing Letters, 10(3), 1999, 253-266.
59. Mitchell, T.M.: Machine Learning, MacGraw Hill, 1997.
60. Prechelt, L.: PROBEN1 - A Set of Neural Network Benchmark Problems and Benchmarking Rules, Technical Report 21/94, Fakultät für Informatik, Universität Karslruhe, 1994.
61. University California Irvene Machine Learning Repository, http://www.ics.uci.edu/mlearn/MLRepository.html
62. Hoaglin, D.C., Mosteller, F., Tukey, J.W.: Understanding Robust and Exploratary Data Analysis, John Wiley & Sons, 1983.
63. Huber, P.J.: Robust Statistics, John Wiley & Sons, 1981.