A 32 × 32 pixel convolution processor chip for address event vision sensors with 155 ns event latency and 20 Meps throughput

IEEE Transactions on Circuits and Systems I: Regular Papers

Volumn 58, Issue 4, 2011, Pages 777-790

  Camuñas Mesa, Luis ^a   Acosta Jiménez, Antonio ^a   Zamarreño Ramos, Carlos ^a   Serrano Gotarredona, Teresa ^a   Linares Barranco, Bernabé ^a  

^a INSTITUTO DE MICROELECTRÓNICA DE SEVILLA   (Spain)

Author keywords

Address event representation; convolution processing; event based processing; feature extraction; frame less vision; neuromorphic circuits and systems; object recognition; vision processing

Indexed keywords

CONVOLUTION; FEATURE EXTRACTION; OBJECT RECOGNITION;

ADDRESS-EVENT REPRESENTATION; ASYNCHRONOUS EVENT; EVENT-BASED; FRAMES PER SECONDS; NEUROMORPHIC CIRCUITS; PROCESSING CAPABILITY; PROCESSING SYSTEMS; VISION PROCESSING;

PIXELS;

EID: 79953273390     PISSN: 15498328     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCSI.2010.2078851     Document Type: Article

References (80)

1. U.Mallik,M. Clapp, E. Choi, G. Cauwenberghs, and R. Etienne-Cummings, "Temporal change threshold detection imager," in IEEE ISSCC Dig. Tech. Papers, 2005, pp. 362-363.
2. P. Lichtsteiner, C. Posch, and T.Delbruck, "A 128 128 12 dB30mW asynchronous vision sensor that responds to relative intensity change," in IEEE ISSCC Dig. Tech. Papers, 2006, pp. 508-509.
3. C. Posch, M. Hofstatter, D.Matolin, G. Vanstraelen, P. Schon, N. Donath, and M. Litzenberger, "A dual-line optical transient sensor with on-chip precision time-stamp generation," in IEEE ISSCC Dig. Tech. Papers, 2007, pp. 500-618.
4. N. Massari, M. Gottardi, and S. Jawed, "A 100 64 128-pixel contrast-based asynchronous binary vision sensor for wireless sensor networks," in IEEE ISSCC Dig. Tech. Papers, 2008, pp. 588-638.
5. P. F. Ruedi, P. Heim, S. Gyger, F. Kaess, C. Arm, R. Caseiro, J.-L. Nagel, and S. Todeschini, "An SoC combining a 132 dB QVGA pixel array and a 32 b DSP/MCU processor for vision applications," in IEEE ISSCC Dig. Tech. Papers, 2009, pp. 46-47, 47a.
6. C. Posch, D. Matolin, and R Wohlgenannt, "A QVGA 143 dB DR asynchronous address-event PWM dynamic vision and image sensor with lossless pixel-level video compression and time-domain CDS," in ISSCC Dig. Tech. Papers, 2010, pp. 400-401.
7. R. Sarpeshkar, M. W. Baker, C. D. Salthouse, J.-J. Sit, L. Turicchia, and S. M. Zhak, "An analog bionic ear processor with zero-crossing detection," in ISSCC Dig. Tech. Papers, 2005, pp. 78-79. (Pubitemid 41696464)
8. B. Wen and K. Boahen, "A 360-channel speech preprocessor that emulates the cochlear amplifier," in ISSCC Dig. Tech. Papers, 2006, pp. 556-557.
9. M. Sivilotti, "Wiring considerations in analog VLSI systems with application to field-programmable networks," Ph.D. dissertation, Comput. Neural Syst., California Inst. Technol., Pasadena, CA, 1991.
10. M. A. Mahowald, "VLSI analogs of neuronal visual processing: A synthesis of form and function," Ph.D. dissertation, Comput. Neural Syst., California Inst. Technol., Pasadena, CA, 1992.
11. J. Lazzaro, J. Wawrzynek, M. Mahowald, M. Silvilotti, and D. Gillespie, "Silicon auditory processors as computer peripherals," IEEE Trans. Neural Netw., vol. 4, pp. 523-528, May 1993. (Pubitemid 23684030)
12. G. Cauwenberghs, N. Kumar, W. Himmelbauer, and A. G. Andreou, "An analog VLSI chip with asynchronous interface for auditory feature extraction," IEEE Trans. Circuits Syst. Part II, Analog Digit. Signal Process., vol. 45, pp. 600-606, May 1998. (Pubitemid 128745245)
13. K. Boahen, "Retinomorphic chips that see quadruple images," in Proc. Int. Conf. Microelectron. Neural, Fuzzy Bio-Inspired Syst. (Microneuro99), Granada, Spain, pp. 12-20.
14. K. Boahen, "A retinomorphic chip with parallel pathways: Encoding INCREASING, ON, DECREASING, and OFF visual signals," Int. J. Analog Integr. Circuits Signal Process., vol. 30, pp. 121-135, Feb. 2002. (Pubitemid 34153573)
15. A. J. Martin and M. Nyström, "Asynchronous techniques for system-on-chip design," Proc. IEEE, vol. 94, no. 6, pp. 1089-1120, Jun. 2006. (Pubitemid 46444961)
16. J. Sparsø and S. B. Furber, Principles of Asynchronous Circuit Design: A Systems Perspective. Norwell, MA: Kluwer, 2001.
17. A.Mortara, E. A. Vittoz, and P. Venier, "A communication scheme for analog VLSI perceptive systems," IEEE J. Solid-State Circuits, vol. 30, pp. 660-669, Jun. 1995.
18. K. Boahen, "Retinomorphic vision systems," presented at the Microneuro' 96: Fifth Int. Conf. Neural Netw. Fuzzy Syst., Laousanne, Switzerland, Feb. 1996.
19. K. Boahen, "Point-to-Point connectivity between neuromorphic chips using address events," IEEE Trans. on Circuits Syst. II, Analog Digit. Signal Process., vol. 47, no. 5, pp. 416-434, May 2000.
20. E. Culurciello, R. Etienne-Cummings, and K. A. Boahen, "A biomorphic digital image sensor," IEEE J. Solid-State Circuits, vol. 38, pp. 281-294, 2003.
21. P. F. Ruedi, P. Heim, F. Kaess, E. Grenet, F. Heitger, P.-Y. Burgi, S. Gyger, and P. Nussbaum, "A 128 128, pixel 120-dB dynamic-range vision-sensor chip for image contrast and orientation extraction," IEEE J. Solid-State Circuits, vol. 38, pp. 2325-2333, 2003.
22. S. Chen and A. Bermak, "Arbitrated time-to-first spike CMOS image sensor with on-chip histogram equalization," IEEE Trans. Very Large Scale Integr. (VLSI) Syst., vol. 15, no. 3, pp. 346-357, Mar. 2007. (Pubitemid 46600152)
23. M. Azadmehr, J. Abrahamsen, and P. Häfliger, "A foveated AER imager chip," in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS2005), Kobe, Japan, pp. 2751-2754.
24. J. Costas-Santos, T. Serrano-Gotarredona, R. Serrano-Gotarredona, and B. Linares-Barranco, "A contrast retina with on-chip calibration for neuromorphic spike-based AER vision systems," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 54, no. 7, pp. 1444-1458, Jul. 2007. (Pubitemid 47104011)
25. J. A. Leñero-Bardallo, T. Serrano-Gotarredona, and B. Linares-Barranco, "A five-decade dynamic-range ambient-light-independent calibrated signed-spatial-contrast AER retina with 0.1 ms latency and optional time-to-first-spike mode," IEEE Trans. Circuits Syst. I, Reg. Papers, to be published.
26. P. Lichtsteiner, C. Posch, and T. Delbruck, "A 128 128 120 dB 30 mW asynchronous vision sensor that responds to relative intensity change," IEEE J. Solid-State Circuits, vol. 43, pp. 566-576, Feb. 2008. (Pubitemid 351190224)
27. K. A. Zaghloul and K. Boahen, "Optic nerve signals in a neuromorphic chip: Part 1," IEEE Trans. Biomed. Eng., vol. 51, pp. 657-666, 2004. (Pubitemid 38375141)
28. K. A. Zaghloul and K. Boahen, "Optic nerve signals in a neuromorphic chip: Part 2," IEEE Trans. Biomed Eng., vol. 51, pp. 667-675, 2004. (Pubitemid 38375142)
29. V. Chan, S.-C. Liu, and A. van Schaik, "Aer EAR: A matched silicon cochlea pair with address event representation interface," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 54, pp. 48-59, Jan. 2007. (Pubitemid 46351018)
30. E. Chicca, A. M. Whatley, P. Lichtsteiner, V. Dante, T. Delbruck, P. Del Giudice, R. J. Douglas, and G. Indiveri, "A multichip pulse-based neuromorphic infrastructure and its application to a model of orientation selectivity," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 54, no. 5, pp. 981-993, May 2007. (Pubitemid 46773770)
31. M. Oster, Y. Wang, R. Douglas, and S.-C. Liu, "Quantification of a spike-based winner-take-all VLSI network," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 55, no. 10, pp. 3160-3169, Nov. 2008.
32. T. Teixeira, A. G. Andreou, and E. Culurciello, "Event-based imaging with active illumination in sensor networks," in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS2005), Kobe, Japan, pp. 644-647.
33. [Online]. Available: http://jaer.wiki.sourceforge.net
34. T. Delbrück, "Frame-free dynamic digital vision," in Proc. Int. Symp. Secure-Life Electron., Adv. Electron. Quality Life Soc.,Mar. 6-7, 2008, pp. 21-26.
35. R. J. Vogelstein, U. Mallik, E. Culurciello, G. Cauwenberghs, and R. Etienne-Cummings, "A multi-chip neuromorphic system for spike-based visual information processing," Neural Comput., vol. 19, no. 9, pp. 2281-300, 2007.
36. D. H. Hubel and T. N. Wiesel, "Receptive fields of single neurones in the Cat's striate cortex," J. Physiol., vol. 148, pp. 574-591, 1959.
37. E. T. Rolls andG.Deco, Computational Neuroscience of Vision. New York: Oxford Univ. Press, 2002.
38. P. Vernier, A. Mortara, X. Arreguit, and E. A. Vittoz, "An integrated cortical layer for orientation enhancement," IEEE J. Solid-State Circuits, vol. 32, pp. 177-186, Feb. 1997.
39. T. Y. W. Choi, P. Merolla, J. Arthur, K. Boahen, and B. E. Shi, "Neuromorphic implementation of orientation hypercolumns," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 52, no. 6, pp. 1049-1060, Jun. 2005. (Pubitemid 40970277)
40. T. Serrano-Gotarredona, A. G. Andreou, and B. Linares-Barranco, "Aer image filtering architecture for vision processing systems," IEEE Trans. Circuits Syst. I, Fundam. Theory Appl., vol. 46, no. 9, pp. 1064-1071, Sep. 1999. (Pubitemid 30503703)
41. R. Serrano-Gotarredona, T. Serrano-Gotarredona, A. Acosta-Jiménez, and B. Linares-Barranco, "A neuromorphic cortical-layer microchip for spike-based event processing vision systems," IEEE Trans. Circuits Systems I, Reg. Papers, vol. 53, no. 12, pp. 2548-2566, Dec. 2006. (Pubitemid 46002253)
42. R. Serrano-Gotarredona, T. Serrano-Gotarredona, A. Acosta-Jiménez, C. Serrano-Gotarredona, J. A. Pérez-Carrasco, B. Linares-Barranco, A. Linares-Barranco, G. Jiménez-Moreno, and A. Civit-Ballcels, "On real-time AER 2-D convolution hardware for neuromorphic spike-based cortical processing," IEEE Trans. Neural Netw., vol. 19, no. 7, pp. 1196-1219, Jul. 2008.
43. K. Fukushima, "Visual feature extraction by a multilayered network of analog threshold elements," IEEE Trans. Syst. Sci. Cybern., vol. SSC-5, no. 4, pp. 322-333, 1969.
44. K. Fukushima and N. Wake, "Handwritten alphanumeric character recognition by the neocognitron," IEEE Trans. Neural Netw., vol. 2, no. 3, pp. 355-365, May 1991.
45. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
46. Y. LeCun and Y. Bengio, "Convolutional networks for images, speech, and time series," in The Handbook of Brain Science and Neural Networks, M. Arbib, Ed. Cambridge, MA: MIT Press, 1995, pp. 255-258.
47. S.Grossberg, E.Mingolla, and J.Williamson, "Synthetic aperture radar processing by a multiple scale neural system for boundary and surface representation," Neural Netw., vol. 8, no. 7/8, pp. 1005-1028, 1995. (Pubitemid 26040654)
48. S. Lawrence, C. L. Giles, A. Tsoi, and A. Back, "Face recognition: A convolutional neural network approach," IEEE Trans. Neural Netw., vol. 8, no. 1, pp. 98-113, 1997. (Pubitemid 127767784)
49. C. Neubauer, "Evaluation of convolution neural networks for visual recognition," IEEE Trans. Neural Netw., vol. 9, no. 4, pp. 685-696, 1998. (Pubitemid 128750029)
50. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, "Backpropagation applied to handwritten zip code recognition," Neural Comput., vol. 1, no. 4, pp. 541-551, 1989.
51. K. Chellapilla, M. Shilman, and P. Simard, "Optimally combining a cascade of classifiers," presented at the Int. Symp.. Document Recog. Retrieval, San Jose, CA, 2006.
52. R. Vaillant, C. Monrocq, and Y. LeCun, "Original approach for the localisation of objects in images," IEE Proc. Vis., Image, Signal Process., vol. 141, no. 4, pp. 245-250, Aug. 1994.
53. M. Osadchy, Y. LeCun, and M. Miller, "Synergistic face detection and pose estimation with energy-based models," J. Mach. Learn. Res., vol. 8, pp. 1197-1215, May 2007. (Pubitemid 46827844)
54. C. Garcia and M. Delakis, "Convolutional face finder: A neural architecture for fast and robust face detection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 11, pp. 1408-1423, 2004. (Pubitemid 40168019)
55. F. Nasse, C. Thurau, and G. A. Fink, "Face detection using gpu-based convolutional neural networks," Lecture Notes in Computer Science, Computer Analysis of Images and Patterns, vol. 5702/2009, pp. 83-90, 2009.
56. A. Frome, G. Cheung,A.Abdulkader,M. Zennaro, B.Wu, A. Bissacco, H. Adam, H. Neven, and L. Vincent, "Large-scale privacy protection in Google street view," in Proc. Int. Conf.Comp. Vis. (ICCV'09), 2009, pp. 2373-2380.
57. T. Serre, L.Wolf, S. Bileschi,M. Riesenhuber, and T. Poggio, "Robust object recognition with cortex-like mechanisms," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 3, pp. 411-426, Mar. 2007. (Pubitemid 46336417)
58. V. Jain and H. S. Seung, "Natural image denoising with convolutional networks," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2008, vol. 21.
59. S. Nowlan and J. Platt, "A convolutional neural network hand tracker," in Advances in Neural Information Processing Systems, G. Tesauro,D. Touretzky, and T. Leen, Eds. Cambridge,MA: MIT Press, 1995, vol. 7, pp. 901-908.
60. R. Hadsell, P. Sermanet, M. Scoffier, A. Erkan, K. Kavackuoglu, U. Muller, and Y. LeCun, "Learning long-range vision for autonomous off-road driving," J. Field Robot., vol. 26, no. 2, pp. 120-144, Feb. 2009.
61. F.Ning, D. Delhomme, Y. LeCun, F. Piano, L. Bottou, and P. Barbano, "Toward automatic phenotyping of developing embryos from videos," IEEE Trans. Image Process., vol. 14, no. 9, pp. 1360-1371, Sep. 2005. (Pubitemid 41430898)
62. V. Jain, J. F. Murray, F. Roth, S. Turaga, V. Zhigulin, K. Briggman, M. Helmstaedter, W. Denk, and H. S. Seung, "Supervised learning of image restoration with convolutional networks," in Proc. Int. Conf. Comp. Vis. (ICCV'07), pp. 1-8.
63. J. A. Pérez-Carrasco, B. Acha, C. Serrano, L. Camuñas-Mesa, T. Serrano-Gotarredona, and B. Linares-Barranco, "Fast vision through frame-less event-based sensing and convolutional processing. Application to texture recognition," IEEE Trans. Neural Netw., vol. 21, no. 4, pp. 609-620, Apr. 2010.
64. R. VanRullen and S. J. Thorpe, "Rate coding versus temporal order coding: What the retinal ganglion cells tell the visual cortex," Neural Comput., vol. 13, no. 6, pp. 1255-1283, Jun. 2001. (Pubitemid 33595838)
65. A. Delorme, J. Gautrais, R. Van Rullen, and S. S. Thorpe, "Spikenet: A simulator for modeling large networks of integrate and fire neurons," Neurocomputing, vol. 26-27, pp. 989-996, 1999. (Pubitemid 29447413)
66. J. Lazzaro and J.Wawrzynek, "A multi-sender asynchronous extension to the AER protocol," in Proc. 16th Conf. Adv. Res. VLSI (ARVLSI'95), pp. 158-158.
67. S. R. Deiss, R. J. Douglas, and A.M.Whatley, "A pulse-coded communications infrastructure for neuromorphic systems," in Pulsed Neural Networks, W. Maass and C.M. Bishop, Eds. Cambridge, MA:MIT Press, 1999, ch. 6, pp. 157-178.
68. R. Serrano-Gotarredona, M. Oster, P. Lichtsteiner, A. Linares-Barranco, R. Paz-Vicente, F. Gómez-Rodríguez, L. Camuñas-Mesa, R. Berner, M. Rivas, T. Delbrück, S. C. Liu, R. Douglas, P. Häfliger, G. Jiménez-Moreno, A. Civit, T. Serrano-Gotarredona, A. Acosta- Jiménez, and B. Linares-Barranco, "Caviar: A 45 k neuron, 5 M synapse, 12 G connects/s AER hardware sensory-processing- learning-actuating system for high-Speed visual object recognition and tracking," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1417-1438, Sep. 2009.
69. P. Merolla, J. Arthur, B. E. Shi, and K. Boahen, "Expandable networks for neuromorphic chips," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 54, no. 2, pp. 301-311, Feb. 2007. (Pubitemid 46444261)
70. M. J. Bellido, A. J. Acosta, J. Luque, A. Barriga, and M. Valencia, "Evaluation of metastability transfer models: An application to an N-bistable CMOS synchronizer," Int. J. Electron., vol. 79, no. 5, pp. 585-593, 1995.
71. K. A. Boahen, "A burst-mode word-serial address-event link - Part I: Transmitter design," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 51, no. 7, pp. 1269-1280, Jul. 2004.
72. A. Linares-Barranco, G. Jiménez-Moreno, B. Linares-Barranco, and A. Civit-Ballcels, "On algorithmic rate-coded AER generation," IEEE Trans. Neural Netw., vol. 17, no. 3, pp. 771-788, May 2006.
73. F. Gómez-Rodríguez, R. Paz, A. Linares-Barranco,M. Rivas, L.Miro, S. Vicente, G. Jimenez, and A. Civit, "AER tools for communications and debugging," in Proc. IEEE Int. Symp. Circuits Syst. 2006 (ISCAS'06), pp. 3253-3256. (Pubitemid 47132245)
74. J. A. Pérez-Carrasco, T. Serrano-Gotarredona, C. Serrano-Gotarredona, B. Acha, and B. Linares-Barranco, "High-speed character recognition system based on a complex hierarchical AER architecture," in Proc. 2008 IEEE Int. Symp. Circuits Syst. (ISCAS08), pp. 2150-2153.
75. S. R. Vangal, J. Howard, G. Ruhl, S. Dighe, H. Wilson, J. Tschanz, D. Finan, A. Singh, T. Jacob, S. Jain, V. Erraguntla, C. Roberts, Y. Hoskote, N. Borkar, and S. Borkar, "An 80-Tile 1.28 TFLOPS network- on-chip in 65 nm CMOS," in Proc. IEEE Int. Solid-State Circuits Conf. (ICCSC07), pp. 98-99.
76. C. Fabaret, C. Poulet, J. Y. Han, and Y. LeCun, "Cnp: An FPGA-based processor for convolutional networks," in Int. Conf. Field Progr. Logic Appl., 2009.
77. C. Farabet, B. Martini, P. Akselrod, S. Talay, Y. LeCun, and E. Culurciello, "Hardware accelerated convolutional neural networks for synthetic vision systems," in Proc. 2010 IEEE Int. Symp. Circuits Syst. (ISCAS10), 2010, pp. 257-260.
78. F. Mamalet, S. Roux, and C. Garcia, "Embedded facial image processing with convolutional neural networks," in Proc. 2010 IEEE Int. Symp. Circuits Syst. (ISCAS10), 2010, pp. 261-263.
79. J. M. Nageswaran, N. Dutt, Y. Wang, and T. Delbrück, "Computing spike-based convolutions on GPUs," in Proc. 2009 IEEE Int. Symp. Circuits Syst. (ISCAS09), pp. 1917-1920.
80. A. Linares-Barranco, R. Paz-Vicente, F. Gómez-Rodríguez, A. Jiménez, M. Rivas, G. Jiménez, and A. Civit, "On the AER convolution processors for FPGA," in Proc. 2010 IEEE Int. Symp. Circuits Syst. (ISCAS10), 2010, pp. 4237-4240.