Analog VLSI Circuits for Competitive Learning Networks

Analog Integrated Circuits and Signal Processing

Volumn 15, Issue 3, 1998, Pages 291-314

  Card, H C ^a,b,c,d,e   Mcneill, D K ^a   Schneider, C R ^a,f  

^a UNIVERSITY OF MANITOBA   (Canada)

^b UNIVERSITY OF MANCHESTER   (United Kingdom)

^c IBM T J WATSON RESEARCH CENTER   (United States)

^d LUCENT TECHNOLOGIES   (United States)

^e IEEE   (Canada)

^f Niche Technology   (Canada)

Author keywords

Analog VLSI Circuits; Artificial Neural Networks; Competitive Learning; Vector Quantization

Indexed keywords

COMPUTER SIMULATION; LEARNING ALGORITHMS; LINEAR INTEGRATED CIRCUITS; NEURAL NETWORKS; VECTOR QUANTIZATION;

COMPETITIVE LEARNING NETWORKS;

VLSI CIRCUITS;

EID: 0032028645     PISSN: 09251030     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1008222414574     Document Type: Article

References (113)

1. B. Ackland, "The role of VLSI in multimedia." IEEE J. Solid St. Ccts. 29, pp. 381-388, Apr. 1994.
2. S. Sheng, A. Chandrakasan, and R. W. Brodersen, "A portable multimedia terminal." IEEE Commun. Mag. pp. 64-75, Dec. 1992.
3. S. Singhal, D. L. Gall, and C. T. Chen, "Source coding of speech and video signals." Proc. IEEE 78, pp. 1233-1249, July 1990.
4. R. M. Gray, "Vector quantization." IEEE ASSP Mag. pp. 4-29, Apr. 1984.
5. T. Kohonen, "The self-organizing map." Proc. IEEE 78, pp. 1464-1480, Sept. 1990.
6. F. Rosenblatt, Principles of Neurodynamics. Spartan: New York, 1962.
7. C. von der Malsburg, "Self organization of orientation selective cells in striate cortex." Kybernetik 14, pp. 85-100, 1973.
8. D. J. Willshaw and C. von der Malsburg, "How patterned neural connections can be set up by self organization." Proc. Royal Soc. London B 194, pp. 431-445, 1976.
9. C. von der Malsburg, "Network self-organization," in An Introduction to Neural and Electronic Networks, S. F. Zornetzer, J. L. Davis, and C. Lau, editors, Academic Press: San Diego, 1990.
10. K. Fukushima, "Cognitron: A self-organizing multilayered neural network." Biol. Cybern. 20, pp. 121-136, 1975.
11. K. Fukushima, "Neocognitron: A neural network model for a mechanism of visual pattern recognition." IEEE Trans. Syst., Man and Cybern. 13, pp. 826-834, 1983.
12. S. Grossberg, "Adaptive pattern classification and universal receding: I. Parallel development and coding of neural feature detectors." Biol. Cybernetics 23, pp. 121-134, 1976.
13. S. Grossberg, "Competitive learning: From interactive activation to adaptive resonance." Cognitive Science 11, pp. 23-63, 1987.
14. S. A. Amari, "Dynamics of pattern formation in lateral-inhibition type neural fields." Biol. Cybern. 27, pp. 77-87, 1977.
15. T. Kohonen, "Self-organized formation of topologically correct feature maps." Biol. Cybern. 43, pp. 59-69, 1982.
16. T. Kohonen, Self-organization and associative memory. Springer-Verlag: Berlin, 1984.
17. D. E. Rumelhart and D. Zipser, "Feature discovery by competitive learning." Cognitive Science 9, pp. 75-112, 1985.
18. C. A. Mead, Analog VLSI and Neural systems. Addison Wesley: Reading, 1989.
19. J. Burr, "Digital neurochip design," in Parallel Digital Implementations of Neural Networks, K. W. Przytula and V K. Prasanna, editors, Prentice Hall: Englewood Cliffs, 1993, Chap. 8, pp. 223-281.
20. P. Thiran, V. Peiris, P. Heim, and B. Hochet, "Quantization effects in digitally behaving circuit implementations of Kohonen networks." IEEE Trans. NNs 5, pp. 450-458, May 1994.
21. A. F. Murray, D. Del Corso, and L. Tarassenko, "Pulse-stream VLSI neural networks mixing analog and digital techniques." IEEE Trans. on Neural Networks 2, pp. 193-204, Mar. 1991.
22. B. Gilbert, "A precise four quadrant multiplier with subnanosecond response." IEEE J. Solid State Ccts. SC-3, p. 365, 1968.
23. F. J. Kub, K. K. Moon, I. A. Mack, and F. M. Long, "Programmable analog vector-matrix multipliers." IEEE J. Solid St. Ccts. 25, pp. 207-214, Feb. 1990.
24. C. R. Schneider and H. C. Card, "CMOS implementation of analog Hebbian synaptic learning circuits," in Proc Int. Joint Conf. on Neural Networks, Seattle, WA, 1991, Vol. 1, pp. 437-442.
25. C. R. Schneider and H. C. Card, "Analog CMOS deterministic Boltzmann circuits." IEEE J. Solid St. Ccts. 28, pp. 907-914, Aug. 1993.
26. H. C. Card, C. R. Schneider, and R. S. Schneider, "Learning capacitive weights in analog CMOS neural networks." J. VLSI Sign. Proc. 8, pp. 209-225, 1994.
27. D. B. Schwartz, R. E. Howard, and W. E. Hubbard, "A programmable neural network chip." J. Solid St. Ccts. 24, pp. 313-319, Apr 1989.
28. E. Vittoz, H. Oguey, M. A. Maher, O. Nys, E. Dijkstra, and M. Chevroulet, "Analog storage of adjustable synaptic weights," in VLSI Design of Neural Networks, U. Ramacher and U. Ruckert, editors, Kluwer: Boston, 1991, pp. 47-63.
29. Y. Horio and S. Nakamura, "Analog memories for VLSI neurocomputing," in Artificial Neural Networks: Paradigms, Applications, and Hardware Implementations, E. Sanchez-Sinencio and C. Lau, editors, 1992, pp. 344-363.
30. B. Hochet, V. Peiris, S. Abdo, and M. J. Declercq, "Implementation of a learning Kohonen neuron based on a new multilevel storage technique." IEEE J. Solid St. Ccts. 26, pp. 262-267, Mar. 1991.
31. E. K. F. Lee and P. G. Gulak, "Error correction technique for multivalued MOS memory." Elect. Lett. 27, pp. 1321-1323, 1991.
32. E. K. F. Lee and P. G. Gulak, "Field programmable analogue array based on MOSFET transconductors." Elect. Lett. 28, pp. 28-29, 1992.
33. R. Castello, D. D. Caviglia, M. Franciotta, and F. Montecchi, "Self refreshing analogue memory cell for variable synaptic weights." Elect. Lett. 27, pp. 1871-1873, 1991.
34. E. El-Masry and B. Maundy, "Analog ANN weight storage elements with on-chip learning capabilities using switched capacitors," in Proc. Can. Conf. on VLSI, Halifax, Canada, Oct. 1992, pp. 220-227.
35. Y. Arima, M. Murasaki, T. Yamada, A. Maeda, and H. Shinohara, "A refreshable analog VLSI neural network chip with 400 neurons and 40K synapses." IEEE J. Solid St. Ccts. 27, pp. 1854-1861, 1992.
36. Y. Arima, K. Mashiko, K. Okada, T. Yamada, A. Maeda, H. Notani, H. Kondoh, and S. Kayano, "A 336-neuron, 28K synapse, self-learning neural network chip with branch-neuron-unit architecture." IEEE J. Solid St. Ccts. 26, pp. 1637-1644, Nov. 1991.
37. A. F. Murray, "Analogue noise enhanced learning in neural network circuits." Elect. Lett. 27, pp. 1546-1548, 1991.
38. M. Hohfeld and S. E. Fahlman, "Probabilistic rounding in neural network learning with limited precision." Neurocomputing 4, pp. 291-299, 1992.
39. H. C. Card, "Analog circuits for relaxation networks." Int. J. Neural Syst. 4, pp. 359-379, Dec. 1993.
40. B. K. Dolenko and H. C. Card, "Tolerance to analog hardware of on-chip learning in backpropagation networks." IEEE Trans. Neural Networks 6, pp. 1045-1052, Sept. 1995.
41. M. Jabri and B. Flower, "Weight perturbation: An optimal architecture and learning technique for analog VLSI feedforward and recurrent multilayer networks." IEEE Trans. Neural Networks 3, pp. 154-157, 1992.
42. J. Alspector, R. Meir, B. Yuhas, A. Jayakumar, and D. Lippe, "A parallel gradient descent method for learning in analog VLSI neural networks," in Adv. Neural Info. Proc. Syst. 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Morgan Kaufmann, 1993, pp. 836-844.
43. J. Alspector, J. W. Gannett, S. Haber, M. B. Parker, and R. Chu, "A VLSI efficient technique for generating multiple uncorrelated noise sources and its application to stochastic neural networks." IEEE Trans. Circuits and Syst. 38, p. 109, Jan. 1991.
44. B. Flower and M. Jabri, "Summed-weight neuron perturbation: An O(N) improvement over weight perturbation," in Adv. Neural Info. Proc. Syst. 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Morgan Kaufmann, 1993, pp. 244-251.
45. G. Cauwenberghs, "A Fast stochastic error-descent algorithm for supervised learning and optimization," in Proc. NIPS-5, Denver, 1992.
46. D. B. Kirk, "Accurate and precise computation using analog VLSI with applications to computer graphics and neural networks," Ph.D thesis, Department of Computer Science, Caltech, 1993.
47. D. B. Kirk, D. Kerns, K. Fleischer, and A. H. Barr, "Analog VLSI implementation of gradient descent," in Adv. Neural Info. Proc. Syst. 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Morgan Kaufmann, 1993, pp. 789-796.
48. D. A. Kerns, J. E. Tanner, M. A. Sivilotti, and J. Luo. "CMOS UV writeable non-volatile analog storage," in Advanced Research in VLSI, Santa Cruz, 1991, pp. 245-261.
49. G. Cauwenberghs, C. F. Neugebauer, and A. Yariv, "Analysis and verification of an analog VLSI incremental outer product learning system." IEEE Trans. Neural Networks 3, pp. 488-497, May 1992.
50. R. G. Benson and D. A. Kerns, "UV activated conductances allow for multiple time-scale learning." IEEE Trans. Neural Networks 4, pp. 434-440, May 1993.
51. S. Tam, M. Holler, J. Brauch, A. Pine, A. Peterson, S. Anderson, and S. Deiss, "A reconfigurable multichip analog neural network: Recognition and backpropagation training." Proc. IJCNN 92, Baltimore, Vol. 2, pp. 625-630, 1992.
52. J. Alspector, R. B. Allen, A. Jayakumar, T. Zeppenfeld, and R. Meir, "Relaxation networks for large supervised learning problems," in Adv. in Neural Info. Proc. Syst. 3, D. S. Touretzky, J. Moody, and R. Lippmann, editors, Morgan Kaufmann: San Mateo, CA, 1991, pp. 1015-1021.
53. J. Alspector, A. Jayakumar, and S. Luna, "Experimental evaluation of learning in a neural microsystem," in Advances in Neural Info. Proc. Syst. 4, Morgan Kaufmann: San Mateo, CA, 1992.
54. J. Raffel, J. Mann, R. Berger, A. Soares and S. Gilbert, "A generic architecture for wafer scale neuromorphic systems," in Proc. IEEE Int. Conf. Neural Networks III, 1987, pp. 501-513.
55. J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation, Addison-Wesley: Menlo Park, CA, 1991.
56. R. O. Duda and P. E. Hart, Pattern Recognition and Scene Analysis, John Wiley: New York, 1973.
57. D. DeSieno, "Adding a conscience to competitive learning," Proc. IEEE Int. Conf. Neural Networks, Vol. 1, pp. 117-124, San Diego, 1988.
58. A. K. Krishnamurthy, S. C. Ahalt, D. E. Melton, and P. Chen, "Neural networks for vector quantization of speech and images." IEEE J. Sel. Areas in Commun. 8, pp. 1449-1457, Oct. 1990.
59. J. Lazzaro, S. Ryckebusch, M. A. Mahowald, and C. A. Mead, "Winner-take-all networks of O(N) complexity," in Adv. in Neural Info. Proc. Syst. I, D. S. Touretzky, editor, Morgan Kaufmann: San Mateo, CA, 1989, pp. 703-711.
60. J. Choi and B. J. Sheu, "A high precision VLSI winner take all circuit for self-organizing neural networks." IEEE J. Solid St. Ccts. 28, pp. 576-584, May 1993.
61. W. C. Fang, B. J. Sheu, O. T. C. Chen, and J. Choi, "A VLSI neural processor for image data compression using self-organizing networks." IEEE Trans. Neural Networks 3, pp. 506-518, May 1992.
62. B. J. Sheu and J. Choi, Neural Information Processing and VLSI, Kluwer: Boston, 1995.
63. J. L. Wyatt and D. L. Standley, "Criteria for robust stability in a class of lateral inhibition networks coupled through resistive grids." Neural Computation 1, pp. 58-67, 1989.
64. J. R. Mann and S. Gilbert, "An analog self-organizing neural network chip," in Advances in Neural Information Processing Systems I, D.S. Touretzky, editor, Morgan Kaufmann: San Mateo, CA, 1990, pp. 739-747.
65. Y. P. Tsividis and D. Anastassiou, "Switched capacitor neural networks." Elect. Lett. 23, pp. 958-959, 1987.
66. M. Brownlow, L. Tarassenko, A. F. Murray, A. Hamilton, S. Han and H. M. Reekie, "Pulse firing neural chips for hundreds of neurons," in Advances in Neural Information Processing Systems 2, D. S. Touretzky, editor, Morgan Kaufmann: San Mateo, CA, 1989, pp. 785-792.
67. A. Rodriguez-Vazquez, R. Dominguez-Castro, A. Rueda, J. L. Huertas, and E. Sanchez-Sinencio, "Nonlinear switched capacitor neural networks for optimization problems." IEEE Trans. Ccts. Systems 37, pp. 384-398, Mar. 1990.
68. B. Maundy and E. El-Masry, "A switched capacitor bidirectional associative memory." IEEE Trans. Ccts. and Systems 37, pp. 1568-1572, 1990.
69. B. Maundy and E. El-Masry, "A self organizing switched capacitor neural network." IEEE Trans. Ccts. and Systems 38. pp. 1556-1563, 1991.
70. H. C. Card and C. R. Schneider, "Analog CMOS neural circuits - In situ learning." Int. J. Neural Syst. 3, pp. 103-124, 1992.
71. R. Linares-Barranco, E. Sanchez-Sinencio, A. Rodriguez-Vazquez, and J. L. Huertas, "A modular T-mode design approach for analog neural network hardware implementations." IEEE J. Solid St. Ccts. 27, pp. 701-713, 1992.
72. J. Moody and C. J. Darken, "Fast learning in networks of locally tuned processing units." Neural Computation 1, pp. 281-294, 1989.
73. S. J. Nowlan, "Soft competitive adaptation: Neural network algorithms based on fitting statistical mixtures," Ph.D thesis, School of Computer Science, Carnegie-Mellon University, 1991.
74. G. E. Hinton and S. J. Nowlan, "The bootstrap Widrow-Hoff rule as a cluster formation algorithm." Neural Computation 2, pp. 355-362, 1990.
75. G. E. Hinton, Neural Networks for Industry. Information Technology Research Center, University of Toronto, 1991.
76. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm." J. Royl Sta. Soc. 39, pp. 1-38, 1977.
77. E. Oja, "Neural networks, principal components, and subspaces." Int. J. Neural Systems 1, pp. 61-68, 1989.
78. J. T. Tou and R. C. Gonzalez, Pattern Recognition Principles. Addison-Wesley: Reading, 1974.
79. J. MacQueen, "Some methods for classification and analysis of multivariate data," in Proc. 5th Berkeley Symp. on Prob. and Statistics, University of California Press, 1967.
80. N. R. Pal, J. C. Bezdek, and E. C. K. Tsao, "Generalized clustering networks and Kohonen's self-organizing scheme." IEEE Trans. Neural Networks 4, pp. 549-557, July 1993.
81. G. A. Carpenter and S. Grossberg, "A massively parallel architecture for a self-organizing neural pattern recognition machine." Computer Vision, Graphics and Image Processing 37, pp. 54-115, 1987.
82. G. A. Carpenter and S. Grossberg, "The ART of adaptive pattern recognition by a self-organizing neural network." IEEE Computer, pp. 77-88, Mar. 1988.
83. G. A. Carpenter and S. Grossberg, "A self-organizing neural network for supervised learning, recognition, and prediction." IEEE Commun. Mag. pp. 38-49, Sept. 1992.
84. S. S. Watkins and P. M. Chau, "A radial basis function neurocomputer implemented with analog VLSI circuits," Proc. IJCNN 92, Baltimore, 1992, Vol. 2, pp. 607-612.
85. J. Anderson, J. C. Platt, and D. B. Kirk, "An analog VLSI chip for radial basis functions," in Adv. Neural Info. Proc. Syst. 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Morgan Kaufmann, 1993, pp. 765-772.
86. T. Delbruck, "Bump circuits for computing similarity and dissimilarity of analog voltages," in Proc. Int. Joint Conf. on Neural Networks, Seattle, July 1991, Vol. 1, pp. 475-479.
87. D. B. Schwartz and V. K. Samalam, "An analog VLSI splining circuit," in Advances in Neural Information Processing Systems 3, D. S. Touretzky, J. Moody, and R. Lippmann, editors, Morgan Kaufmann: San Mateo, CA, Apr. 1991, pp. 1008-1014.
88. M. H. Cohen and A. C. Andreou, "MOS circuit for nonlinear Hebbian learning." Elect. Lett. 28, pp. 591-593, 1992.
89. M. H. Cohen and A. G. Andreou, "Current mode subthreshold MOS implementation of the Herault-Jutten autoadaptive network." IEEE J. Solid St. Ccts. 27, pp. 714-723, 1992.
90. E. Vittoz and X. Arreguit, "CMOS integration of Herault-Jutten cells for separation of sources," in Analog VLSI Implementation of Neural Systems, C. A. Mead and M. Ismail, editors, Kluwer: Boston, 1989, pp. 57-83.
91. S. W. Tsay and R. W. Newcomb, "VLSI implementation of ART1 memories." IEEE Trans. Neural Networks 2, pp. 214-221, 1991.
92. P. Heim, "CMOS analogue VLSI implementation of a Kohonen map," D.Sc.T. Thesis, Ecole Polytechnique Federale de Lausanne, Switzerland, 1993.
93. E. Vittoz, E. Sorouchyari, P. Heim, X. Arreguit, and F. Krummenacher, "Analog VLSI implementation of a Kohonen map," in Proc. Journees D'Electronique 1989, Lausanne, Oct. 1989, pp. 292-301.
94. P. Heim, B. Hochet, and E. Vittoz, "Generation of learning neighbourhood in Kohonen feature maps by means of simple nonlinear network." Elect. Lett. 27, pp. 275-277, 1991.
95. B. Hochet, V. Peiris, S. Abdo, and M. J. Declercq, "Implementation of a learning Kohonen neuron based on a new multilevel storage technique." IEEE J. Solid St. Ccts. 26, pp. 262-267, Mar. 1991.
96. O. Landolt, E. Vittoz, and P. Heim, "CMOS self-biased Euclidean distance computing circuit with high dynamic range." Elect. Lett. 28, pp. 352-354, 1992.
97. D. Macq, M. Verleysen, P. Jespers, and J. D. Legat, "Analog implementation of a Kohonen map with on-chip learning." IEEE Trans. Neural Networks 4, pp. 456-461, May 1993.
98. Y. He and U. Cilingiroglu, "A charge-based on-chip adaptation Kohonen neural network." IEEE Trans. Neural Networks 4, pp. 462-469, May 1993.
99. V. Peiris, B. Hochet, and M. Declercq, "Implementation of a fully parallel Kohonen map: A mixed analog digital approach." Proc. ICNN 4, pp. 2064-2069, 1994.
100. P. Thiran, V. Peiris, P. Heim, and B. Hochet, "Quantization effects in digitally behaving circuit implementations of Kohonen networks." IEEE Trans. NNs 5, pp. 450-458, May 1994.
101. O. Landolt, "An analog CMOS implementation of a Kohonen network with learning capability," in VLSI for Neural Networks and Artificial Intelligence, J. G. Delgado-Frias and W. R. Moore, editors, Plenum Press: New York, 1994, pp. 25-34.
102. A. Mortara and E. Vittoz, "A communication architecture tailored for analog VLSI artificial neural networks: Intrinsic performance and limitations." IEEE Trans. Neural Networks 5, pp. 549-466, May 1994.
103. B. E. Shi and L. O. Chua, "Resistive grid image filtering: Input/output analysis via the CNN framework." IEEE Trans. Ccts. and Syst. I: Fund. Th. and Appl. 39, pp. 531-548, 1992.
104. H. Harrer, J. A. Nossek, and R. Stelzl, "An analog implementation of discrete time cellular neural networks." IEEE Trans. Neural Networks 3, pp. 466-476, May 1992.
105. H. C. Card, D. K. McNeill, C. R. Schneider and R. S. Schneider, "The impact of VLSI fabrication on neural learning," in Proc. IEEE ISCAS, Seattle, 1995, Vol. 2, pp. 985-988.
106. D. K. McNeill and H. C. Card, "Analog hardware tolerance of soft competitive learning," in Proc. IEEE ICNN, Orlando, 1994, Vol. 4, pp. 2004-2008.
107. M. R. Schroeder and B. S. Atal, "Code-excited linear prediction (CELP); High quality speech at very low bit rates," in Proc. IEEE Int. Conf. Acoust., Speech, and Signal Processing, 1985, pp. 937-940.
108. Y. M. Cheng and D. O'Shaughnessy, "On 450-600b/s natural sounding speech coding." IEEE Trans. on Speech and Audio Processing 1, pp. 207-220, Apr. 1993.
109. S. Kamarsu, "Neural code-excited linear prediction for low power adaptive speech coding," MSc. thesis, Dept. of Electrical and Computer Engineering, University of Manitoba, 1995.
110. S. Kamarsu and H. C. Card, "Limited-precision unsupervised learning algorithms for speech coding," in Proc. INNS World Congress on Neural Networks, Washington, D.C., July 1995, Vol. 1, pp. 128-131.
111. T. C. Lee and A. M. Peterson, "Adaptive vector quantization using a self-development neural network." IEEE J. Sel. Areas in Commun. 8, pp. 1458-1471, Oct. 1990.
112. L. V. Veleva and R. K. Kunchev, "Adaptive speech coding with DCT and neural net vector quantization." Elect. Lett. 29, pp. 704-705, Apr. 1993.
113. B. Frey, "Code and wavelet excited linear prediction of speech," MSc. thesis, Dept. of Electrical and Computer Engineering, University of Manitoba, 1994.