Csiszár's Generalized Error Measures for Gradient-descent-based Optimizations in Neural Networks Using the Backpropagation Algorithm

Connection Science

Volumn 8, Issue 1, 1996, Pages 79-114

  Neelakanta, P S ^a   Abusalah, S ^a   De Groff, D ^a   Sudhakar, R ^a   Park, J C ^b  

^a Florida Atlantic Univ   (United States)

^b Evaluation Cent Detachment   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0030520369     PISSN: 09540091     EISSN: None     Source Type: Journal    
DOI: 10.1080/095400996116965     Document Type: Article

References (31)

1. Ackley, D.H., Hinton, G.E. & Sejnowski, T.J. (1985) A learning algorithm for Boltzmann machines. Cognitive Science, 9, 147.
2. Aczél, J. (1984) Measuring information beyond communication theory. Information Management & Processing, 20, 383-395.
3. Anderson, J.A. & Rosenfeld, E. (Eds) (1988) Neurocomputing: Foundation of Research. Cambridge, MA: MIT Press.
4. Baum, E.B. & Wilczek, F. (1988) Supervised learning of probability distributions by neural networks. In D.A. Anderson (Ed.), Neural Information Processing Systems. New York: American Institute of Physics.
5. Blahut, R.E. (1987) Principle and Practice of Information Theory. Reading, MA: Addison-Wesley.
6. Bose, N.K. & Liang, P. (1996) Neural Network Fundamentals with Graphs, Algorithms, and Applications. New York: McGraw-Hill.
7. Csiszár, I. (1967) Information-type measures of difference of probability distributions and indirect observations. Studia Sci. Mathematica Hungaria, 2, 299-318.
8. Csiszár, I. (1972) A class of measures of informativity of observation channels. Periodica Mathematica Hungaria, 2, 191-213.
9. Csiszár, I. & Körner, J. (1981) Information Theory, Coding Theorems for Discrete Memoryless Systems. Budapest: Akademiai Kaidó.
10. De Groff, D., Neelakanta, P.S., Sudhakar, R. & Medina, F. (1993) Liquid-crystal model of neural networks. Complex Systems, 7, 43-57.
11. Jefferys, H. (1946) An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London, Series A, 186, 453-461.
12. Havrda, J. & Charvát, F. (1967) Quantification method of classification processes. Kybernetika, 3, 30-35.
13. Kapur, J.N. (1967) Generalized entropy of order α and β. The Mathematics Seminar, 4, 78-94.
14. Kapur, J.N. (1972) Measures of uncertainty mathematical programming and physics. Journal of the Indian Society of Agricultural Statistics, 24, 47-66.
15. Kapur, J.N. & Kesavan, H.K. (1992) Entropy Optimization Principles with Applications. Boston: Academic Press (Harcourt Brace Jovanovich).
16. Kjaer, T.W., Hertz, J.A. & Richmond, B.J. (1994) Decoding of cortical neuronal signals: network models, information estimation and spatial tuning. Journal of Computational Neuroscience, 1, 109-139.
17. Kosko, B. (1992) Neural Network and Fuzzy Systems: A Dynamical Systems Approach to Machine Intelligence. Englewood Cliffs, NJ: Prentice Hall.
18. Kullback, S. (1959) Information Theory and Statistics. New York: John Wiley and Sons.
19. Kullback, S. & Leibler, R.A. (1951) On information and sufficiency. Ann. Math. Statist., 22, 79-86.
20. Lin, J. (1991) Divergence measures based on Shannon entropy. IEEE Transactions on Information Theory, 17, 145-151.
21. Liou, C.Y. & Lin, S.L. (1990) The other variant Boltzmann machine. In Proceedings of Joint Conference on Neural Networks, 18-22 June 1990, Washington, DC, I 449-454.
22. Neelakanta, P.S. & De Groff, D. (1994) Neural Networks Modeling. Boca Raton, FL: CRC Press.
23. Neelakanta, P.S., Sudhakar, R. & De Groff, D. (1991) Langevin machine: a neural network based on stochastically justifiable sigmoidal function. Biological Cybernetics, 65, 331-338.
24. Österreicher, F. & Vajda, I. (1993) Statistical information and discrimination. IEEE Transactions on Information Theory, 39, 1036-1039.
25. Rényi, A. (1961) On measures of entropy and information. In Proceeding of the Fourth Berkeley Symposium of Mathematical Statistics and Probability, Vol. 1, pp. 547-561.
26. Rosenblatt, F. (1961) Principle of Neurodynamics: Perceptron and the Theory of Brain Mechanisms. Washingtan, DC: Sparton Books.
27. Rumelhart, D.E. & McClelland, J.L. (Eds) (1986) Parallel Distributed Processing: Vol. I. Cambridge, MA: MIT Press.
28. Shannon, C.E. (1948) A mathematical theory of communication. Bell System Technical Journal, 27, 329-423, 623-659.
29. Sharma, B.D. & Mittal, D.P. (1975) New non-additive measures of entropy for discrete probability distributions. Journal of Mathematical Sciences, 10, 28-40.
30. Solla, S.A., Levin, E. & Fleisher, M. (1988) Accelerated learning in a layered neural network. Complex Systems, 2, 625-640.
31. Watrous, R.L. (1992) A comparison between squared error and relative entropy metrics using several optimization algorithms. Complex Systems, 6, 495-505.