Robustness of spiking Deep Belief Networks to noise and reduced bit precision of neuro-inspired hardware platforms

Frontiers in Neuroscience

Volumn 9, Issue JUN, 2015, Pages

  Stromatias, Evangelos ^a   Neil, Daniel ^b   Pfeiffer, Michael ^b   Galluppi, Francesco ^c   Furber, Steve B ^a   Liu, Shih Chii ^b  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

^b UNIVERSITY OF ZURICH   (Switzerland)

^c INSTITUT DE LA VISION   (France)

Author keywords

Deep belief networks; Neuro inspired hardware; Noise robustness; Spiking neural networks; SpiNNaker

Indexed keywords

ACCURACY; ARTICLE; ARTIFICIAL NEURAL NETWORK; CLASSIFICATION ALGORITHM; CODING; COMPARATIVE STUDY; COMPUTER PROGRAM; DEEP BELIEF NETWORK; ENERGY COST; MACHINE LEARNING; MATHEMATICAL COMPUTING; NERVE CELL PLASTICITY; NOISE; NONLINEAR SYSTEM; PROBABILITY; RECEPTIVE FIELD; SIMULATION; SPIKE WAVE;

EID: 84935098619     PISSN: 16624548     EISSN: 1662453X     Source Type: Journal    
DOI: 10.3389/fnins.2015.00222     Document Type: Article

References (62)

1. Alibart, F., Gao, L., Hoskins, B. D., and Strukov, D. B. (2012), High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm, Nanotechnology, 23, 7, 075201
2. Arthur, J. and Boahen, K. (2007), Synchrony in silicon: The gamma rhythm, IEEE Transactions on Neural Networks, 18, 6, 1815-1825, doi:10.1109/TNN.2007.900238
3. Basu, A., Shuo, S., Zhou, H., Lim, M. H., and Huang, G.-B. (2013), Silicon spiking neurons for hardware implementation of Extreme Learning Machines, Neurocomputing, 102, 125-134, doi:http://dx.doi. org/10.1016/j.neucom.2012.01.042
4. Benjamin, B., Gao, P., Mcquinn, E., Choudhary, S., Chandrasekaran, A., Bussat, J.-M., et al. (2014), Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations, Proceedings of the IEEE, 102, 5, 699-716
5. Bill, J. and Legenstein, R. (2014), A compound memristive synapse model for statistical learning through STDP in spiking neural networks, Frontiers in Neuroscience, 8, 412, doi:10.3389/fnins.2014.00412
6. Brändli, C., Berner, R., Yang, M.-H., Liu, S.-C., and Delbrück, T. (2014), A 240 × 180 130 dB 3us latency global shutter spatiotemporal vision sensor, IEEE J. Solid-State Circuits, 49, 10, 2333-2341
7. Brink, S., Nease, S., Hasler, P., Ramakrishnan, S., Wunderlich, R., Basu, A., et al. (2013), A learning-enabled neuron array IC based upon transistor channel models of biological phenomena, IEEE Transactions onBiomedical Circuits and Systems, 7, 1, 71-81, doi:10.1109/TBCAS.2012.2197858
8. Courbariaux, M., David, J., and Bengio, Y. (2014), Low precision arithmetic for deep learning, in arXiv, Cornell, doi:arxiv.org/pdf/1412.7024, in Submission
9. Cruz-Albrecht, J. M., Derosier, T., and Srinivasa, N. (2013), A scalable neural chip with synaptic electronics using CMOS integrated memristors, Nanotechnology, 24, 38, 384011
10. Das, S., Pedroni, B. U., Merolla, P., Arthur, J., Cassidy, A. S., Jackson, B. L., et al. (2015), Gibbs sampling with low-power spiking digital neurons, in 2015 IEEE International Symposium on Circuits and Systems (ISCAS), (accepted).
11. Davidson, A. P., Brüderle, D., Eppler, J.M., Kremkow, J.,Muller, E., Pecevski, D., et al. (2009), fPyNNg: a common interface for neuronal network simulators, Frontiers in Neuroinformatics, 2, 1-10, doi:10. 3389/neuro.11.011.2008
12. Dean, J., Corrado, G., Monga, R., Chen, M., K.and Devin, Mao, M., Senior, A., et al. (2012), Large scale distributed deep networks, in Advances in Neural Information Processing Systems, 1223-1231
13. Farabet, C., Martini, B., Corda, B., Akselrod, P., Culurciello, E., and LeCun, Y. (2011), Neuflow: A runtime reconfigurable dataflow processor for vision, in 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 109-116
14. Furber, S., Galluppi, F., Temple, S., and Plana, L. (2014), The SpiNNaker project, Proceedings of the IEEE, 102, 5, 652-665, doi:10.1109/JPROC.2014.2304638
15. Furber, S. and Temple, S. (2008), Neural Systems Engineering, Computational Intelligence: A Compendium, 763-796
16. Furber, S. B., Lester, D. R., Plana, L. A., Garside, J. D., Painkras, E., Temple, S., et al. (2013), Overview of the SpiNNaker system architecture, IEEE Trans. Comput., 62, 12, 2454-2467, doi:10.1109/TC. 2012.142
17. Galluppi, F., Davies, S., Rast, A., Sharp, T., Plana, L., and Furber, S. (2012), A hierarchical configuration system for a massively parallel neural hardware platform, in ACM International Conference on Computing Frontiers
18. Galluppi, F., Lagorce, X., Stromatias, E., Pfeiffer, M., Plana, L. A., Furber, S., et al. (2014), A framework for plasticity implementation on the SpiNNaker neural architecture, Frontiers in Neuroscience, 8, 429, doi:10.3389/fnins.2014.00429
19. Goodman, D. F. M. and Brette, R. (2008), Brian: a simulator for spiking neural networks in python, Frontiers in Neuroinformatics, 2, 5, doi:10.3389/neuro.11.005.2008
20. Hinton, G. and Salakhutdinov, R. (2006), Reducing the dimensionality of data with neural networks, Science, 313, 5786, 504-507
21. Jug, F., Cook, M., and Steger, A. (2012), Recurrent competitive networks can learn locally excitatory topologies, in Proceedings of 2012 International Joint Conference on Neural Networks (IJCNN), 1-8, doi:10.1109/IJCNN.2012.6252786
22. Kinget, P. (2005), Device mismatch and tradeoffs in the design of analog circuits, IEEE J. Solid-State Circuits, 40, 6, 1212-1224
23. Kuzum, D., Jeyasingh, R. G. D., Lee, B., and Wong, H.-S. P. (2012), Nanoelectronic programmable synapses based on phase change materials for brain-inspired computing, Nano Letters, 12, 5, 2179- 2186, doi:10.1021/nl201040y, pMID: 21668029
24. Le, Q. V., Ranzato, M. A., Monga, R., Devin, M., Chen, K., Corrado, G., et al. (2012), Building high-level features using large scale unsupervised learning, in Proc. of ICML
25. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998), Gradient-based learning applied to document recognition, Proceedings of the IEEE, 86, 11, 2278-2324
26. Linares-Barranco, B., Serrano-Gotarredona, T., and Serrano-Gotarredona, R. (2011), Compact low-power calibration mini-DACs for neural arrays with programmable weights, IEEE Transactions on Neural Networks, 14, 5, 1207-1216, doi:10.1109/TNN.2003.816370
27. Liu, S.-C. and Delbruck, T. (2010), Neuromorphic sensory systems, Curr Opin in Neurobiol, 20, 3, 288-295
28. Liu, S.-C., Delbrück, T., Indiveri, G., Whatley, A., and Douglas, R. (2015), Event-Based Neuromorphic Systems (John Wiley & Sons, Ltd, UK)
29. Liu, S.-C. and Douglas, R. (2004), Temporal coding in a silicon network of integrate-and-fire neurons, IEEE Transactions on Neural Networks, 15, 5, 1305-1314, doi:10.1109/TNN.2004.832725
30. Liu, S.-C., van Schaik, A., Minch, B., and Delbrück, T. (2014), Asynchronous binaural spatial audition sensor with 2 × 64 × 4 channel output, IEEE Trans. Biomed. Circuits Syst., 8, 4, 453-464, doi:10.1109/ TBCAS.2013.2281834
31. Merolla, P., Arthur, J., Alvarez, R., Bussat, J.-M., and Boahen, K. (2014a), A multicast tree router for multichip neuromorphic systems, IEEE Trans. Circuits and Syst. I, 61, 3, 820-833, doi:10.1109/TCSI. 2013.2284184
32. Merolla, P. A., Arthur, J. V., Alvarez-Icaza, R., Cassidy, A. S., Sawada, J., Akopyan, F., et al. (2014b), A million spiking-neuron integrated circuit with a scalable communication network and interface, Science, 345, 6197, 668-673, doi:DOI:10.1126/science.1254642
33. Mitra, S., Fusi, S., and Indiveri, G. (2009), Real-time classification of complex patterns using spike-based learning in neuromorphic VLSI, IEEE Trans. on Biomedical Circuits and Systems, 3, 1, 32-42
34. Moore, S., Fox, P., Marsh, S., Markettos, A., and Mujumdar, A. (2012), Bluehive - a field-programable custom computing machine for extreme-scale real-time neural network simulation, in 2012 IEEE 20th Annual International Symposiumon Field-Programmable CustomComputingMachines (FCCM), 133- 140, doi:10.1109/FCCM.2012.32
35. Moradi, S. and Indiveri, G. (2014), An event-based neural network architecture with an asynchronous programmable synaptic memory, IEEE Trans. Biomed. Circuits Syst., 8, 1, 98-107, doi:10.1109/ TBCAS.2013.2255873
36. Neftci, E., Chicca, E., Indiveri, G., and Douglas, R. (2011), A systematic method for configuring VLSI networks of spiking neurons, Neural Computation, 23, 10, 2457-2497
37. Neftci, E., Das, S., Pedroni, B., Kreutz-Delgado, K., and Cauwenberghs, G. (2014), Event-driven contrastive divergence for spiking neuromorphic systems, Frontiers in Neuroscience, 7, 272, doi:10. 3389/fnins.2013.00272
38. Neil, D. and Liu, S.-C. (2014), Minitaur, an event-driven FPGA-based spiking network accelerator, IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 22, 12, 2621-2628, doi:10.1109/TVLSI. 2013.2294916
39. O'Connor, P., Neil, D., Liu, S.-C., Delbruck, T., and Pfeiffer, M. (2013), Real-time classification and sensor fusion with a spiking deep belief network, Frontiers in Neuroscience, 7, 178, doi:10.3389/fnins. 2013.00178
40. Oster, M., Wang, Y., Douglas, R., and Liu, S.-C. (2008), Quantification of a spike-based winner-take620 VLSI network, IEEE Transactions on Circuits and Systems I: Regular Papers, 55, 10, 3160-3169, doi:10.1109/TCSI.2008.923430
41. Pavasovic, A., Andreou, A., and Westgate, C. (1994), Characterization of subthreshold MOS mismatch in transistors for VLSI systems, Journal of VLSI Signal Processing, 8, 1, 75-85
42. Pelgrom, M., Duinmaijer, A., and Welbers, A. (1989), Matching properties of MOS transistors, IEEE J. Solid-State Circuits, 25, 5, 1212-1224
43. Pfeil, T., Grübl, A., Jeltsch, S., Müller, E., Müller, P., Petrovici, M. A., et al. (2013), Six networks on a universal neuromorphic computing substrate, Frontiers in Neuroscience, 7, 11, doi:10.3389/fnins.2013. 00011
44. Pfeil, T., 629 Potjans, T. C., Schrader, S., Potjans, W., Schemmel, J., Diesmann, M., et al. (2012), Is a 4- bit synaptic weight resolution enough?-constraints on enabling spike-timing dependent plasticity in neuromorphic hardware, Frontiers in neuroscience, 6, doi:10.3389/fnins.2012.00090
45. Pham, P.-H., Jelaca, D., Farabet, C.,Martini, B., LeCun, Y., and Culurciello, E. (2011), Neuflow: Dataflow vision processing system-on-a-chip, in 2012 IEEE 55th International Midwest Symposium on Circuits and Systems (MWSCAS), 1044-1047
46. Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B., and Delbruck, T. (2014), Retinomorphic event636 based vision sensors: Bioinspired cameras with spiking output, Proceedings of the IEEE, 102, 10, 1470-1484
47. Querlioz, D., Bichler, O., Dollfus, P., and Gamrat, C. (2013), Immunity to device variations in a spiking neural network with memristive nanodevices, IEEE Trans on Nanotechnology, 12, 3, 288-295
48. Schemmel, J., Brüderle, D., Grübl, A., Hock, M., Meier, K., and Millner, S. (2010), A wafer641 scale neuromorphic hardware system for large-scale neural modeling, in 2010 IEEE International Symposium on Circuits and Systems (ISCAS), 1947-1950, doi:10.1109/ISCAS.2010.5536970
49. Schmidhuber, J. (2014), Deep learning in neural networks: An overview, CoRR, abs/1404.7828
50. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. (2013), OverFeat: Integrated recognition, localization and detection using convolutional networks, arXiv preprint arXiv:1312.6229
51. Serrano-Gotarredona, R., Serrano-Gotarredona, T., Acosta-Jimenez, A., Serrano-Gotarredona, C., Perez-Carrasco, J., Barranco, B., et al. (2008), On real-time AER 2-D convolutions hardware for neuromorphic spike-based cortical processing, IEEE Transactions on Neural Networks, 19, 7, 1196-1219, doi:10.1109/TNN.2008.2000163
52. Serrano-Gotarredona, T. and Linares-Barranco, B. (1999), Systematic width-and-length dependent CMOS transistor mismatch characterization and simulation, Analog Integrated Circuits and Signal Processing, 21, 271-296
53. Serrano-Gotarredona, T., Masquelier, T., Prodromakis, T., Indiveri, G., and Linares-Barranco, B. (2013), STDP and STDP variations with memristors for spiking neuromorphic learning systems, Frontiers in Neuroscience, 7, 2, doi:10.3389/fnins.2013.00002
54. Siegert, A. J. F. (1951), On the first passage time probability problem, Physical Review, 81, 4, 617
55. Stromatias, E., Galluppi, F., Patterson, C., and Furber, S. (2013), Power analysis of large-scale, real658 time neural networks on spiNNaker, in Proceedings of 2013 International Joint Conference on Neural Networks (IJCNN), 1-8, doi:10.1109/IJCNN.2013.6706927
56. Stromatias, E., Neil, D., Galluppi, F., Pfeiffer, M., Liu, S.-C., and Furber, S. (2015a), Live demonstration: Handwritten digit recognition using Spiking Deep Belief Networks on SpiNNaker, in 2015 IEEE International Symposium on Circuits and Systems (ISCAS), (accepted).
57. Stromatias, E., Neil, D., Galluppi, F., Pfeiffer, M., Liu, S.-C., and Furber, S. (2015b), Scalable energy-efficient, low-latency implementations of spiking deep belief networks on SpiNNaker, in 2015 International Joint Conference on Neural Networks (IJCNN), (accepted).
58. Underwood, K. (2004), FPGAs vs. CPUs: Trends in peak floating-point performance, in Proceedings of the 2004 ACM/SIGDA 12th International Symposium on Field Programmable Gate Arrays
59. Vogelstein, R., Mallik, U., Vogelstein, J., and Cauwenberghs, G. (2007), Dynamically reconfigurable silicon array of spiking neurons with conductance-based synapses, IEEE Transactions on Neural Networks, 18, 1, 253-265
60. Wang, Y.-X. and Liu, S.-C. (2006), Programmable synaptic weights for an aVLSI network of spiking neurons, in Proc. IEEE Int. Symp. Circuits and Syst., doi:10.1109/ISCAS.2006.1693637
61. Wang, Y.-X. and Liu, S.-C. (2011), A two-dimensional configurable active silicon dendritic neuron array, IEEE Transactions on Circuits and Systems I: Regular Papers, 58, 9, 2159-2171, doi:10.1109/TCSI. 2011.2112570
62. Wang, Y.-X. and Liu, S.-C. (2013), Active processing of spatio-temporal input patterns in silicon dendrites, IEEE Transactions on Biomedical Circuits and Systems, 7, 3, 307-318, doi:10.1109/TBCAS. 2012.2199487