Greedy information acquisition algorithm: A new information theoretic approach to dynamic information acquisition in neural networks

Connection Science

Volumn 14, Issue 2, 2002, Pages 137-162

  Kamimura, Ryotaro ^a   Kamimura, Taeko ^b   Takeuchi, Haruhiko ^c  

^a TOKAI UNIVERSITY   (Japan)

^b SENSHU UNIVERSITY   (Japan)

^c NATIONAL INSTITUTE OF ADVANCED INDUSTRIAL SCIENCE AND TECHNOLOGY AIST   (Japan)

Author keywords

Competitive learning; Feature detection; Greedy information acquisition; Information maximization; Network growing

Indexed keywords

GREEDY INFORMATION ACQUISITION ALGORITHMS;

ALGORITHMS; INFORMATION THEORY; LEARNING SYSTEMS; NEURAL NETWORKS;

INFORMATION ANALYSIS;

EID: 0036626892     PISSN: 09540091     EISSN: None     Source Type: Journal    
DOI: 10.1080/09540090210162065     Document Type: Article

References (31)

1. Ahalt, S. C., Krishnamurthy, A. K., Chen, P., and Melton, D. E., 1990, Competitive learning algorithms for vector quantization. Neural Networks, 3: 277-290.
2. Atick, J. J., and Redlich, A. N., 1990, Toward a theory of early visual processing. Neural Computation, 2: 308-320.
3. Becker, S., and Hinton, G. E., 1993, Learning mixture models of spatial coherence. Neural Computation, 5: 267-277.
4. DeSieno, D., 1988, Adding a conscience to competitive learning. In Proceedings of IEEE International Conference on Neural Networks, San Diego, pp. 117-124.
5. Dopazo, J., and Carazo, J. M., 1997, Phylogenetic reconstruction using a growing neural network that adopts the topology of a phylogenetic tree. Journal of Molecular Evolution, 44: 226-233.
6. Fahlman, S. E., and Lebiere, C., 2000, The cascade-correlation learning architecture. In Advances in Neural Information Processing, Vol. 2 (San Mateo CA: Morgan Kaufmann), pp. 524-532.
7. Fritzke, B., 1991, Unsupervised clustering with growing cell structures. In Proceedings of IEEE International Joint Conference on Neural Networks, IEEE and INNS, IEEE, pp. 531-536.
8. Fritzke, B., 1994a, Fast learning with incremental RBF networks. Neural Processing Letters, 1 (1): 2-5.
9. Fritzke, B., 1994b, Growing cell structures-a self-organizing network for unsupervised and supervised learning. Neural Networks, 7 (9): 1441-1460.
10. Fritzke, B., 1996, Unsupervised ontogenetic networks. In E. Fiesler and R. Beale (eds) Handbook of Neural Computation. (Bristol: Institute of Physics and Oxford: Oxford University Press).
11. Gatlin, L. L., 1972, Information Theory and Living Systems. (New York: Columbia University Press).
12. Grossberg, S., 1987, Competitive learning: from interactive activation to adaptive resonance. Cognitive Science, 11: 23-63.
13. Hagan, M. T., and Demuth, H. B., 1996, Neural Network Design (Boston MA: PWS).
14. Herrero, J., Valencia, A., and Dopazo, J., 2001, A hierarchical unsupervised growing neural network for clustering gene expression patterns. Bioinformatics, 17 (2): 126-136.
15. Kamimura, R., Kamimura, T., and Shultz, T. R., 2001a, Information theoretic competitive learning and linguistic rule acquistion. Transactions of the Japanese Society for Artificial Intelligence, 16 (2): 287-298.
16. Kamimura, R., Kamimura, T., and Uchida, O., 2001b, Flexible feature discovery and structural information. Connection Science, 13 (4): 323-347.
17. Kamimura, R., and Nakanishi, S., 1995, Hidden information maximization for feature detection and rule discovery. Network, 6: 577-622.
18. Kohonen, T., 1982, Self-organized formation of topological correct feature maps. Biological Cybernetics, 43: 59-69.
19. Lehtokangas, M., 1999, Modelling with constructive backpropagation architecture. Neural Networks, 12: 707-716.
20. Linsker, R., 1988, Self-organization in a perceptual network. Computer, 21: 105-117.
21. Linsker, R., 1989, How to generate ordered maps by maximizing the mutual information between input and output. Neural Computation, 1: 402-411.
22. Linsker, R., 1992, Local synaptic rules suffice to maximize mutual information in a linear network. Neural Computation, 4: 691-702.
23. Luk, A., and Lien, S., 2000, Properties of the generalized lotto-type competitive learning. In Proceedings of International Conference on Neural Information Processing, (San Mateo CA: Morgan Kaufmann), pp. 1180-1185.
24. Marchman, V. A., 1993, Constraints on plasticity in a connectionist model of the English past tense. Journal of Cognitive Neuroscience, 5 (2): 215-234.
25. Poggio, T., and Girosi, F., 1990, Regularization algorithms for learning that are equivalent to multiplayer networks. Science, 247: 978-982.
26. Reed, R., 1993, Pruning algorithms: a survey. IEEE Transactions on Neural Networks, 4 (5): 740-747.
27. Rumelhart, D. E., and Zipser, D., 1986, Feature discovery by competitive learning. In D. E. Rumelhart, G. E. Hinton, R. J. and Williams (eds) Parallel Distributed Processing, Vol. 1 (Cambridge MA: MIT Press), pp. 151-193.
28. Shultz, T. R., and Rivest, F., 2000, Knowledge-based cascade-correlation. In Proceedings of the International Joint Conference on Neural Networks (Los Alamitos CA: IEEE Computer Society Press).
29. Thacker, N. A., Abraham, I., and Courtney, P., 1997, Supervised learning extensions to the clam network. Neural Networks, 10 (2): 315-326.
30. Thacker, N. A., and Mayhew, J. E. W., 1990, Supervised learning extensions to the clam network. Neural Networks, 3 (3): 291-299.
31. Xu, L., 1993, Rival penalized competitive learning for clustering analysis, RBF net, and curve detection. IEEE Transaction on Neural Networks, 4 (4): 636-649.