Efficient feedforward categorization of objects and human postures with address-event image sensors

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 34, Issue 2, 2012, Pages 302-314

  Chen, Shoushun ^a   Akselrod, Polina ^b   Zhao, Bo ^a   Perez Carrasco, Jose Antonio ^c   Linares Barranco, Bernabe ^c   Culurciello, Eugenio ^c  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b Yale University   (United States)

^c INSTITUTO DE MICROELECTRÓNICA DE SEVILLA   (Spain)

Author keywords

address event image sensor; bio inspired categorization; event based circuits; Human posture categorization

Indexed keywords

BIO-INSPIRED; CATEGORIZATION METHODS; CLUSTER-BASED; COMPUTATION COMPLEXITY; EVENT-BASED; FEED-FORWARD; HARDWARE IMPLEMENTATIONS; HARDWARE RESOURCES; HAUSDORFF DISTANCE; HUMAN POSTURES; INPUT VIDEOS; INVARIANT LINES; LINE FEATURES; LINE SEGMENT; ON-THE-FLY; REAL TIME VIDEOS; SIZE AND POSITION; SOFTWARE MODULES; TEMPORAL DIFFERENCES; TRAINING SAMPLE; VISUAL CORTEXES;

ALGORITHMS; FEATURE EXTRACTION; HARDWARE; INFORMATION SCIENCE; SENSORS; SOFTWARE ARCHITECTURE; VIDEO RECORDING;

IMAGE SENSORS;

ALGORITHM; ARTICLE; AUTOMATED PATTERN RECOGNITION; BIOLOGICAL MODEL; BODY POSTURE; HUMAN; IMAGE PROCESSING; METHODOLOGY; PHYSIOLOGY; VIDEORECORDING; VISUAL CORTEX;

ALGORITHMS; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MODELS, BIOLOGICAL; PATTERN RECOGNITION, AUTOMATED; POSTURE; VIDEO RECORDING; VISUAL CORTEX;

EID: 84055212084     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2011.120     Document Type: Article

References (66)

1. R. Van Rullen and S. Thorpe, "Rate Coding versus Temporal Order Coding: What the Retinal Ganglion Cells Tell the Visual Cortex," Neural Computation, vol. 13, pp. 1255-1283, 2001. (Pubitemid 33595838)
2. S. Thorpe, A. Delorme, R. Van Rullen, and W. Paquier, "Reverse Engineering of the Visual System Using Networks of Spiking Neurons," Proc. IEEE Int'l Symp. Circuits and Systems, vol. 4, pp. 405-408, 2000.
3. C. Koch, The Quest for Consciousness: A Neurobiological Approach. Roberts and Company Publishers, 2004.
4. T. Serre, "Learning a Dictionary of Shape-Components in Visual Cortex: Comparison with Neurons, Humans and Machines," PhD dissertation, MIT, Apr. 2006.
5. T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio, "Robust Object Recognition with Cortex-Like Mechanisms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp. 411-426, Mar. 2007. (Pubitemid 46336417)
6. J.J. Peissig and M.J. Tarr, "Visual Object Recognition: Do We Know More Now than We Did 20 Years Ago?" Ann. Rev. Psychology, vol. 58, pp. 75-96, 2007.
7. J.K. Tsotsos, A.J. Rodriguez-Sanchez, A.L. Rothenstein, and E. Simine, "Different Binding Strategies for the Different Stages of Visual Recognition," Proc. Second Int'l Conf. Advances in Brain, Vision and Artificial Intelligence, pp. 150-160, 2007.
8. S. Chikkerur, T. Serre, C. Tan, and T. Poggio, "What and Where: A Bayesian Inference Theory of Attention," Vision Research, vol. 50, no. 22, pp. 2233-2247, 2010.
9. R. Serrano-Gotarredona, T. Serrano-Gotarredona, A. Acosta-Jimenez, and B. Linares-Barranco, "A Neuromorphic Cortical-Layer Microchip for Spike-Based Event Processing Vision Systems," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 53, no. 12, pp. 2548-2566, Dec. 2006. (Pubitemid 46002253)
10. L. Camunas-Mesa, A. Acosta-Jimenez, C. Zamarreno-Ramos, T. Serrano-Gotarredona, and B. Linares-Barranco, "A 32 x 32 Pixel Convolution Processor Chip for Address Event Vision Sensors with 155ns Event Latency and 20Meps Throughput," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 58, no. 4, pp. 777-790, Apr. 2011.
11. T. Teixeira, A. Andreou, and E. Culurciello, "An Address-Event Image Sensor Network," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 4467-4470, May 2006. (Pubitemid 47132547)
12. E. Culurciello and A. Savvides, "Address-Event Image Sensor Network," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 955-958, May 2006. (Pubitemid 47131673)
13. E. Culurciello and A. Andreou, "CMOS Image Sensors for Sensor Networks," Analog Integrated Circuits and Signal Processing, vol. 49, no. 1, pp. 39-51, 2006.
14. T. Teixeira, E. Culurciello, E. Park, D. Lymberopoulos, A. Barton-Sweeney, and A. Savvides, "Address-Event Imagers for Sensor Networks: Evaluation and Modeling," Proc. Information Processing in Sensor Networks, pp. 458-466, Apr. 2006.
15. Z.M. Fu and E. Culurciello, "A 1.2mw CMOS Temporal-Difference Image Sensor for Sensor Networks," Proc. IEEE Int'l Symp. Circuits and Systems, May 2008.
16. S. Pellegrini and L. Iocchi, "Human Posture Tracking and Classification through Stereo Vision and 3D Model Matching," J. Image and Video Processing, vol. 2008, pp. 7:1-7:12, 2008.
17. C. Wu, H. Aghajan, and R. Kleihorst, "Real-Time Human Posture Reconstruction in Wireless Smart Camera Networks," Proc. Int'l Conf. Information Processing in Sensor Networks, pp. 321-331, 2008.
18. R. Jafari, W. Li, R. Bajcsy, S. Glaser, and S. Sastry, "Physical Activity Monitoring for Assisted Living at Home," Proc. Int'l Workshop Wearable and Implantable Body Sensor Networks, Mar. 2007.
19. Z. Fu and E. Culurciello, "Fall Detection Using an Address-Event Temporal Contrast Vision Sensor," Proc. IEEE Int'l Symp. Circuits and Systems, May 2008.
20. G. Virone, M. Alwan, S. Dalal, S.W. Kell, B. Turner, J.A. Stankovic, and R. Felde, "Behavioral Patterns of Older Adults in Assisted Living," IEEE Trans. Information Technology in Biomedicine, vol. 12, no. 3, pp. 387-398, May 2008. (Pubitemid 351761571)
21. H. Su and F.-G. Huang, "Human Gait Recognition Based on Motion Analysis," Proc. Int'l Conf. Machine Learning and Cybernetics, vol. 7, pp. 464-468, Aug. 2005.
22. B. Boulay, F. Bremond, and M. Thonnat, "Posture Recognition with a 3D Human Model," Proc. IEEE Int'l Symp. Imaging for Crime Detection and Prevention, pp. 135-138, June 2005. (Pubitemid 41549133)
23. J. Isaacs and S. Foo, "Hand Pose Estimation for American Sign Language Recognition," Proc. 36th Southeastern Symp. System Theory, pp. 132-136, 2004.
24. K. Takahashi, T. Sakaguchi, and J. Ohya, "Remarks on Real-Time Human Posture Estimation from Silhouette Image Using Neural Network," Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, pp. 370-375, Oct. 2004. (Pubitemid 40415884)
25. E. H-Jaraha, C. Urunuela, and J. Senar, "Detected Motion Classification with a Doublebackground and a Neighborhood-Based Difference," Pattern Recognition Letters, vol. 24, pp. 2079-2092, 2003. (Pubitemid 36558025)
26. L.H.W. Aloysius, G. Dong, H. Zhiyong, and T. Tan, "Human Posture Recognition in Video Sequence Using Pseudo 2-D Hidden Markov Models," Proc. Eighth Control, Automation, Robotics, and Vision Conf., pp. 712-716, Dec. 2004. (Pubitemid 40893549)
27. P. Spagnolo, M. Leo, A. Leone, G. Attolico, and A. Distante, "Posture Estimation in Visual Surveillance of Archaeological Sites," Proc. IEEE Conf. Advanced Video and Signal Based Surveillance, pp. 277-283, July 2003.
28. J. Triesch and C. von der Malsburg, "A System for Person-Independent Hand Posture Recognition against Complex Backgrounds," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 12, pp. 1449-1453, Dec. 2001. (Pubitemid 34035577)
29. M. Gottardi, N. Massari, and S. Jawed, "A 100/iw 128 x 64 Pixels Contrast-Based Asynchronous Binary Vision Sensor for Sensor Networks Applications," IEEE J. Solid-State Circuits, vol. 44, no. 5, pp. 1582-1592, May 2009.
30. E. Artyomov and O. Yadid-Pecht, "Adaptive Multiple-Resolution CMOS Active Pixel Sensor," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 53, no. 10, pp. 2178-2186, 2006. (Pubitemid 44637785)
31. A. Teman, S. Fisher, L. Sudakov, A. Fish, and O. Yadid-Pecht, "Autonomous CMOS Image Sensor for Real Time Target Detection and Tracking," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 2138-2141, May 2008.
32. A. Fish, L. Sudakov-Boresha, and O. Yadid-Pecht, "Low-Power Tracking Image Sensor Based on Biological Models of Attention," Int'l J. Information Theory and Applications, vol. 14, no. 2, pp. 103-114, 2006.
33. S. Mizuno, K. Fujita, H. Yamamoto, N. Mukozaka, and H. Toyoda, "A 256 x 256 Compact Cmos Image Sensor with On-Chip Motion Detection Function," IEEE J. Solid-State Circuits, vol. 38, no. 6, pp. 1072-1075, June 2003.
34. J. Choi, S.-W. Han, S.-J. Kim, S.-I. Chang, and E. Yoon, "A Spatial-Temporal Multiresolution CMOS Image Sensor with Adaptive Frame Rates for Tracking the Moving Objects in Region-of-Interest and Suppressing Motion Blur," IEEE J. Solid-State Circuits, vol. 42, no. 12, pp. 2978-2989, Dec. 2007.
35. P. Lichtsteiner, C. Posch, and T. Delbruck, "A 128 x 1280 120dB 15/is Latency Asynchronous Temporal Contrast Vision Sensor," IEEE J. Solid-State Circuits, vol. 43, no. 2, pp. 566-576, Feb. 2008. (Pubitemid 351190224)
36. D. Kim, Z. Fu, and E. Culurciello, "A 1-mw CMOS Temporal-Difference AER Sensor for Wireless Sensor Networks," IEEE Trans. Electron Devices, vol. 56, no. 11, pp. 2586-2593, Nov. 2009.
37. E-Lab, E. Culurciello's Laboratory at Yale Univ., http://www.eng. yale.edu/elab/, 2011.
38. C. Farabet, B. Martini, P. Akselrod, S. Talay, Y. LeCun, and E. Culurciello, "Hardware Accelerated Convolutional Neural Networks for Synthetic Vision Systems," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 257-260, 2010.
39. C. Farabet, B. Martini, P. Akselrod, B. Corda, S. Talay, Y. LeCun, and E. Culurciello, "Bio-Inspired Vision Processor for Ultra-Fast Object Categorization," Proc. High Performance Embedded Computing Workshop, 2010.
40. E. Culurciello, R. Etienne-Cummings, and K. Boahen, "Arbitrated Address Event Representation Digital Image Sensor," Proc. IEEE Int'l Solid-State Circuits Conf. Digest of Technical Papers, pp. 92-93, May 2001. (Pubitemid 32327484)
41. J.A. Lenero-Bardallo, T. Serrano-Gotarredona, and B. Linares-Barranco, "A 3.6/is Asynchronous Frame-Free Event-Driven Dynamic-Vision-Sensor," IEEE J. Solid-State Circuits, vol. 46, no. 6, pp. 1443-1455, June 2011.
42. L. Chen, G. Lu, and D. Zhang, "Effects of Different Gabor Filter Parameters on Image Retrieval by Texture," Proc. 10th Int'l Multimedia Modelling Conf, vol. 1, pp. 273-278, July 2004.
43. D. Huttenlocher, G. Klanderman, and W. Rucklidge, "Comparing Images Using the Hausdorff Distance," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 9, pp. 850-863, Sept. 1993. (Pubitemid 23703609)
44. D.-G. Sim, O.-K. Kwon, and R.-H. Park, "Object Matching Algorithm Using Robust Hausdorff Distance Measures," IEEE Trans. Image Processing, vol. 8, no. 3, pp. 425-429, Mar. 1999.
45. J. Chen, M.K. Leung, and Y. Gao, "Noisy Logo Recognition Using Line Segment Hausdorff Distance," Pattern Recognition, vol. 36, no. 4, pp. 943-955, 2003.
46. C.-H. T. Yang and S.-H. Lai, "Hybrid Image Matching Combining Hausdorff Distance with Normalized Gradient Matching," Pattern Recognition, vol. 40, no. 4, pp. 1173-1181, 2007. (Pubitemid 44894484)
47. Y. Gao and M. Leung, "Face Recognition Using Line Edge Map," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 6, pp. 764-779, June 2002. (Pubitemid 34669590)
48. M. Barnsley, Fractals Everywhere. Academic Press, 1988.
49. M. Litzenberger, C. Posch, D. Bauer, A. Belbachir, B.K.P. Schon, and H. Garn, "Embedded Vision System for Real-Time Object Tracking Using an Asynchronous Transient Vision Sensor," Proc. 12th Signal Processing Education Workshop, pp. 173-178, Sept. 2006. (Pubitemid 351245948)
50. T. Delbruck and P. Lichtsteiner, "Fast Sensory Motor Control Based on Event-Based Hybrid Neuromorphic-Procedural System," Proc. IEEE Int'l Symp. Circuits and Systems, pp. 845-848, May 2007. (Pubitemid 47448637)
51. R. Serrano-Gotarredona, M. Oster, P. Lichtsteiner, A. Linares-Barranco, R. Paz-Vicente, F. Gomez-Rodriguez, L. Camunas-Mesa, R. Berner, M. Rivas-Perez, T. Delbruck, S.-C. Liu, R. Douglas, P. Hafliger, G. Jimenez-Moreno, A. Ballcels, T. Serrano-Gotarredona, A. Acosta-Jimenez, and B. Linares-Barranco, "CAVIAR: A 45k Neuron, 5M Synapse, 12G Connects/s AER Hardware Sensory Processing Learning Actuating System for High-Speed Visual Object Recognition and Tracking," IEEE Trans. Neural Networks, vol. 20, no. 9, pp. 1417-1438, Sept. 2009.
52. http://www.eng.yale.edu/elab/research/svision/svision.html, 2011.
53. http://www.eng.yale.edu/elab/research/postures/postures. html, 2011.
54. Java AER Open Source Project, http://jaer.wiki.sourceforge. net/, 2011.
55. Y. LeCun, F. Huang, and L. Bottou, "Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2004.
56. F.J. Huang and Y. LeCun, "NORB Object Database," http://www.cs.nyu.edu/~ylclab/data/norb-v1.0-small/, 2009.
57. M. Riesenhuber and T. Poggio, "Hierarchical Models of Object Recognition in Cortex," Nature Neuroscience, vol. 2, pp. 1019-1025, 1999. (Pubitemid 30599567)
58. K. Takahashi and T. Uemura, "Real-Time Human Body Posture Estimation Using Neural Networks," JSME Int'l J. Series C Mechanical Systems, Machine Elements, and Manufacturing, vol. 44, no. 3, pp. 618-625, 2001. (Pubitemid 33082925)
59. T. Masquelier and S.J. Thorpe, "Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity," PLoS Computational Biology, vol. 3, no. 2, p. e31, Feb. 2007.
60. R. Vogelstein, U. Mallik, E. Culurciello, R. Etienne-Cummings, and G. Cauwenberghs, "Saliency-Driven Image Acuity Modulation on a Reconfigurable Silicon Array of Spiking Neurons," Proc. Advances in Neural Information Processing Systems, Dec. 2004.
61. M. Oster, Y. Wang, R. Douglas, and S.-C. Liu, "Quantification of a Spike-Based Winner-Take-All VLSI Network," IEEE Trans. Circuits and Systems I: Regular Papers, vol. 55, no. 10, pp. 3160-3169, Nov. 2008.
62. A. Linares-Barranco, M. Oster, D. Cascado, G. Jimenez, A. Civit, and B. Linares-Barranco, "Inter-Spike-Intervals Analysis of AER Poisson Like Generator Hardware," Neurocomputing, vol. 70, no. 70, pp. 2692-2700, May 2007. (Pubitemid 47308621)
63. B. Zhao and S. Chen, "Realtime Feature Extraction Using MAXLike Convolutional Network for Human Posture Recognition," Proc. IEEE Int'l Symp. Circuits and Systems, May 2011.
64. Z. Fu, T. Delbruck, P. Lichsteiner, and E. Culurciello, "An Address-Event Fall Detector for Assisted Living Applications," IEEE Trans. Biomedical Circuits and Systems, vol. 2, no. 2, pp. 88-96, June 2008.
65. T. Teixeira, D. Jung, G. Dublon, and A. Savvides, "PEM-ID: Identifying People by Gait-Matching Using Cameras and Wearable Accelerometers," Proc. ACM/IEEE Int'l Conf. Distributed Smart Cameras, pp. 1-8, 2009.
66. T. Teixeira and A. Savvides, "Lightweight People Counting and Localizing in Indoor Spaces Using Camera Sensor Nodes," Proc. ACM/IEEE Int'l Conf. Distributed Smart Cameras, pp. 36-43, 2007.