Deep convolutional neural networks for image classification: A comprehensive review

Neural Computation

Volumn 29, Issue 9, 2017, Pages 2352-2449

  Rawat, Waseem ^a   Wang, Zenghui ^a  

^a UNIVERSITY OF SOUTH AFRICA   (South Africa)

Author keywords

[No Author keywords available]

Indexed keywords

CONVOLUTION; IMAGE CLASSIFICATION; NEURAL NETWORKS; VISION;

COMPUTING POWER; CONVOLUTIONAL NEURAL NETWORK; DEEP CONVOLUTIONAL NEURAL NETWORKS; LABELED DATA; LARGE AMOUNTS; RECENT STATE; VISUAL TASKS;

DEEP NEURAL NETWORKS;

EID: 85031680076     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00990     Document Type: Review

References (306)

1. Abdulkader, A. (2006). A two-tier Arabic offline handwriting recognition based on conditional joining rules. In Proceedings of the 10th International Workshop on Frontiers in Handwriting Recognition (pp. 1-6). Los Alamitos, CA: IEEE Computer Society
2. Agostinelli, F., Hoffman, M., Sadowski, P., & Baldi, P. (2014). Learning activation functions to improve deep neural networks. arXiv 1412.6830
3. Ahmed, A., Yu, K., Xu, W., Gong, Y., & Xing, E. (2008). Training hierarchical feedforward visual recognition models using transfer learning from pseudo-tasks. In Proceedings of the European Conference on Computer Vision (pp. 69-82). Berlin: Springer
4. Arora, S., Bhaskara, A., Ge, R., & Ma, T. (2014). Provable bounds for learning some deep representations. In Proceedings of the 31th International Conference Machine Learning (pp. 584-592). N.p.: International Machine Learning Society
5. Ba, J., & Frey, B. (2013). Adaptive dropout for training deep neural networks. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 26 (pp. 3084-3092). Red Hook, NY: Curran
6. Ba, J., Mnih, V., & Kavukcuoglu, K. (2015).Multiple object recognition with visual attention. In Proceedings of the 3rd International Conference on Learning Representations (pp. 1-10). N.p.: Computational and Biological Learning Society
7. Bachman, P. (2016). An architecture for deep, hierarchical generative models. In D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, &R.Garnett (Eds.), Advances in neural information processing systemsm, 29 (pp. 4826-4834). Red Hook, NY: Curran
8. Baldi, P., & Sadowski, P. J. (2013). Understanding dropout. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 26 (pp. 3084-3092). Red Hook, NY: Curran
9. Baldi, P., &Sadowski, P. (2014). The dropout learning algorithm. Artificial Intelligence, 210, 78-122
10. Barkan, O., Weill, J., Wolf, L., & Aronowitz, H. (2013). Fast high dimensional vector multiplication face recognition. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1960-1967). Red Hook, NY: Curran
11. Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A., & Criminisi, A. (2016).Measuring neural net robustnesswith constraints. InD. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in neural information processing systems, 29 (pp. 2613-2621). Red Hook, NY: Curran
12. Basu, S., Karki, M., DiBiano, R., Mukhopadhyay, S., Ganguly, S., Nemani, R., & Gayaka, S. (2016). A theoretical analysis of deep neural networks for texture classification. arXiv 1605.02699
13. Belkin, M., Niyogi, P., & Sindhwani, V. (2006). Manifold regularization: Ageometric framework for learning from labeled and unlabeled examples. Journal ofMachine Learning Research, 7, 2399-2434
14. Bell, S., & Bala, K. (2015). Learning visual similarity for product design with convolutional neural networks. ACM Transactions on Graphics, 34(4), 98-107
15. Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1), 1-127
16. Bengio, Y. (2013). Deep learning of representations: Looking forward. In Proceedings of International Conference on Statistical Language and Speech Processing (pp. 1-37). Berlin: Springer
17. Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: Areview and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798-1828
18. Bengio, Y., Lamblin, P., Popovici, D., &Larochelle, H. (2006).Greedy layer-wise training of deep networks. In J. C. Platt, D. Koller, Y. Singer, & S. T. Roweis (Eds.), Advances in neural information processing systems, 19 (pp. 2814-2822). Red Hook, NY: Curran
19. Bengio, Y., Mesnard, T., Fischer, A., Zhang, S., &Wu, Y. (2017). STDP-compatible approximation of backpropagation in an energy-based model. Neural Computation, 29(3), 555-577
20. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157-166
21. Bengio, Y., Thibodeau-Laufer, E., Alain, G., & Yosinski, J. (2014). Deep generative stochastic networks trainable by backprop. In Proceedings of the 31st International Conference Machine Learning (pp. 226-234). N.p.: International Machine Learning Society
22. Bottou, L. (1998). Online learning and stochastic approximations. On-Line Learning in Neural Networks, 17(9), 142-177
23. Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In Proceedings of the International Conference on Computational Statistics (pp. 177-186). Berlin: Physica-Verlag Heidelberg
24. Boureau, Y., Ponce, J., & LeCun, Y. (2010). A theoretical analysis of feature pooling in visual recognition. In Proceedings of the 27th International Conference on Machine Learning (pp. 111-118). N.p.: International Machine Learning Society
25. Bromley, J., Bentz, J. W., Bottou, L., Guyon, I., LeCun, Y., Moore, C., . Shah, R. (1993). Signature verification using a "Siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(4), 669-688
26. Bulo, S., & Kontschieder, P. (2014). Neural decision forests for semantic image labelling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 81-88). Los Alamitos, CA: IEEE Computer Society
27. Bruna, J., Szlam, A., & LeCun, Y. (2013). Signal recovery from pooling representations. arXiv 1311.4025
28. Cao, X., Wipf, D., Wen, F., Duan, G., & Sun, J. (2013). A practical transfer learning algorithm for face verification. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3208-3215). Red Hook, NY: Curran
29. Chang, J., & Chen, Y. (2015). Batch-normalized maxout network in network. arXiv 1511.02583
30. Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014). Return of the devil in the details: Delving deep into convolutional nets. arXiv 1405.3531
31. Chellapilla, K., & Puri, S., & Simard, P. (2006). High performance convolutional neural networks for document processing. In Proceedings of the 10th InternationalWorkshop on Frontiers in Handwriting Recognition. Los Alamitos, CA: IEEE Computer Society
32. Chellapilla, K., Shilman, M., & Simard, P. (2006). Optimally combining a cascade of classifiers. In Proceedings of the 18th Annual Symposium on Electronic Imaging (pp. 6067-6126). N.p: International Society for Optical Engineering
33. Chellapilla, K., &Simard, P. (2006). Two-tier approach for Arabic offline handwriting recognition. In Proceedings of the 10th Internstional Workshop on Frontiers in Handwriting Recognition (pp. 1-6). Los Alamitos, CA: IEEE Computer Society
34. Chen, D., Cao, X., Wen, F., & Sun, J. (2013). Blessing of dimensionality: Highdimensional feature and its efficient compression for face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3025-3032). Red Hook, NY: Curran
35. Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., & Chen, Y. (2015). Compressing neural networks with the hashing trick. arXiv 1504.04788v1
36. Cheng, Y., Felix, X. Y., Feris, R. S., Kumar, S., Choudhary, A., & Chang, S. (2015). Fast neural networks with circulant projections. arXiv 1502.03436
37. Cheng, Y., Yu, F. X., Feris, R. S., Kumar, S., Choudhary, A., & Chang, S. (2015). An exploration of parameter redundancy in deep networks with circulant projections. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2857-2865). Red Hook, NY: Curran
38. Cheng, Z., Soudry, D., Mao, Z., & Lan, Z. (2015). Training binary multilayer neural networks for image classification using expectation backpropagation. arXiv 1503.03562
39. Chollet, F. (2016). Xception: Deep learning with depthwise separable convolutions. arXiv 1610.02357
40. Chopra, S., Hadsell, R., & LeCun, Y. (2005). Learning a similarity metric discriminatively, with application to face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 539-546). Los Alamitos, CA: IEEE Computer Society
41. Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., & LeCun, Y. (2015). The loss surfaces of multilayer networks. In Proceedings 18th International Conference on Artificial Intelligence and Statistics (pp. 192-204). www.jmlr.org/proceedings/papers/v38/choromanska15.pdf
42. Ciresan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2010). Deep, big, simple neural nets for handwritten digit recognition. Neural Computation, 22(12), 3207-3220
43. Ciresan, D. C., Meier, U., Masci, J., Maria Gambardella, L., & Schmidhuber, J. (2011). Flexible, high performance convolutional neural networks for image classification. In Proceedings of the International Joint Conference on Artificial Intelligence (vol. 1, pp. 1237-1242). Menlo Park, CA: AAAI Press
44. Ciresan, D., Meier, U., &Schmidhuber, J. (2012).Multi-column deep neural networks for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3642-3649). Red Hook, NY: Curran
45. Clevert, D., Unterthiner, T., & Hochreiter, S. (2016). Fast and accurate deep network learning by exponential linear units (ELUs). In Proceedings of the 4th International Conference on Learning Representations (pp. 1-14). N.p.: Computational and Biological Learning Society
46. Coates, A., Lee, H., & Ng, A. Y. (2011). An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (pp. 215-223). www.jmlr.org/proceedings/papers/v15/coates11a/coates11a.pdf
47. Collobert, R., & Bengio, S. (2004). A gentle Hessian for efficient gradient descent. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (pp. 517-520). N.p.: IEEE Signal Processing Society
48. Collobert, R., Sinz, F., Weston, J., & Bottou, L. (2006). Large scale transductive SVMs. Journal of Machine Learning Research, 7, 1687-1712
49. Courbariaux, M., Bengio, Y., & David, J. P. (2015). BinaryConnect: Training deep neural networks with binary weights during propagations. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems, 28 (pp. 3123-3131). Red Hook, NY: Curran
50. Courbariaux, M., Hubara, I., Soudry, D., & Yaniv, R. E. (2016). Binarized neural networks. In D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in neural information processing systems, 29 (pp. 1-9). N.p.: Preproceedings
51. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., . Le, Q. V. (2012). Large scale distributed deep networks. In F. Pereira, C. J. C. Burges, L. Bottou, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 25 (pp. 1223-1231). Red Hook, NY: Curran
52. Decoste, D., & Schölkopf, B. (2002). Training invariant support vector machines. Machine Learning, 46(1-3), 161-190
53. Deng, L. (2014). A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPATransactions on Signal and Information Processing, 3(2), 1-29
54. Deng, L., & Yu, D. (2014). Deep learning:Methods and applications. Foundations and Trends in Signal Processing, 7(3-4), 197-387
55. Denil, M., Shakibi, B., Dinh, L., & de Freitas, N. (2013). Predicting parameters in deep learning. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 26 (pp. 2148-2156). Red Hook, NY: Curran
56. Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y., &Fergus, R. (2014). Exploiting linear structure within convolutional networks for efficient evaluation. In Z. Ghahramani, M.Welling, C. Cortes, N. D. Lawrence, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 1269-1277). Red Hook, NY: Curran
57. Dettmers, T. (2016). 8-bit approximations for parallelism in deep learning. In Proceedings of the 4th International Conference on Learning Representations (pp. 1-14). N.p.: Computational and Biological Learning Society
58. Dreyfus, S. (1962). The numerical solution of variational problems. Journal of Mathematical Analysis and Applications, 5(1), 30-45
59. Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12, 2121-2159
60. Dumoulin, V., & Visin, F. (2016). A guide to convolution arithmetic for deep learning. arXiv 1603.07285
61. Erhan, D., Bengio, Y., Courville, A., Manzagol, P. A., Vincent, P., & Bengio, S. (2010). Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11, 625-660
62. Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2012). Scene parsing with multiscale feature learning, purity trees, and optimal covers. arXiv 1202.2160
63. Fawzi, A., Fawzi, O., & Frossard, P. (2015a). Analysis of classifiers' robustness to adversarial perturbations. arXiv 1502.02590
64. Fawzi, A., Fawzi, O., & Frossard, P. (2015b). Fundamental limits on adversarial robustness. In Proceedings of the 32nd International Conference on Machine Learning (pp. 1-7). N.p.: International Machine Learning Society
65. Fei-Fei, L. (2006). Knowledge transfer in learning to recognize visual objects classes. In Proceedings of the 4th International Conference on Development and Learning (pp. 11-17). N.p.: IEEE Computational Intelligence Society
66. Fei-Fei, L., Fergus, R., & Perona, P. (2006). One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 594-611
67. Filho, D. P., Luiz, P., Oliveira, L. S., & Britto, A. S. Jr. (2009). A database for forest species recognition. In Proceedings of the XXII Brazilian Symposium on Computer Graphics and Image Processing (pp. 1-2). Red Hook, NY: Curran
68. Finn, C., Tan, X. Y., Duan, Y., Darrell, T., Levine, S., & Abbeel, P. (2015). Deep spatial autoencoders for visuomotor learning. arXiv 1509.06113
69. Floreano, D., &Mattiussi, C. (2008). Bio-inspired artificial intelligence: Theories, methods, and technologies. Cambridge, MA: MIT Press
70. Fukushima, K. (1979). Self-organization of a neural network which gives positioninvariant response. In Proceedings of the 6th International Joint Conference on Artificial Intelligence (vol. 1, pp. 291-293). San Francisco: Morgan Kaufmann
71. Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4), 193-202
72. Fukushima, K., & Miyake, S. (1982). Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognition, 15(6), 455-469
73. Garcia, C., & Delakis, M. (2002). A neural architecture for fast and robust face detection. In Proceedings of the 16th International Conference on Pattern Recognition (pp. 44-47). Los Alamitos, CA: IEEE Computer Society
74. Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 580-587). Red Hook, NY: Curran
75. Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (pp. 249-256). jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf
76. Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse rectifier neural networks. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (pp. 315-323).www.jmlr.org/proceedings/papers/v15/glorot11a/glorot11a.pdf
77. Gong, Y., Liu, L., Yang, M., & Bourdev, L. (2014). Compressing deep convolutional networks using vector quantization. arXiv 1412.6115
78. Gong, Y., Wang, L., Guo, R., & Lazebnik, S. (2014, September). Multi-scale orderless pooling of deep convolutional activation features. In Proceedings of the European Conference on Computer Vision (pp. 392-407). Berlin: Springer
79. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Cambridge, MA: MIT Press
80. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., . Bengio, Y. (2014). Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 2672-2680). Red Hook, NY: Curran
81. Goodfellow, I. J., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. In Proceedings of the 3rd International Conference on Learning Representations (pp. 1-11). N.p.: Computational and Biological Learning Society
82. Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A. C., & Bengio, Y. (2013). Maxout networks. In Proceedings of the 30th International ConferenceMachine Learning (pp. 1319-1327). N.p.: International Machine Learning Society
83. Graham, B. (2014). Fractional max-pooling. arXiv 1412.6071
84. Grauman, K., & Darrell, T. (2005). The pyramid match kernel: Discriminative classification with sets of image features. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1458-1465). Red Hook, NY: Curran
85. Griffin, G., Holub, A., &Perona, P. (2007). Caltech-256 object category dataset. Pasadena: California University of Technology
86. Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., .Wang, G. (2015). Recent advances in convolutional neural networks. arXiv 1512.07108
87. Gu, S., & Rigazio, L. (2014). Towards deep neural network architectures robust to adversarial examples. arXiv 1412.5068
88. Gulcehre, C., Cho, K., Pascanu, R., & Bengio, Y. (2014). Learned-norm pooling for deep feedforward and recurrent neural networks. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (pp. 530-546). New York: Springer-Verlag
89. Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016). Deep learning for visual understanding: Areview. Neurocomputing, 187, 27-48
90. Hadsell, R., Chopra, S., & LeCun, Y. (2006). Dimensionality reduction by learning an invariant mapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1735-1742). Los Alamitos, CA: IEEE Computer Society
91. Hadsell, R., Sermanet, P., Ben, J., Erkan, A., Scoffier, M., Kavukcuoglu, K., . LeCun, Y. (2009). Learning long-range vision for autonomous off-road driving. Journal of Field Robotics, 26(2), 120-144
92. Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016). EIE: Efficient inference engine on compressed deep neural network. arXiv 1602.01528
93. Han, S., Mao, H., & Dally, W. J. (2016). Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In Proceedings of the 3rd International Conference on Learning Representations (pp. 1-14).N.p.: Computational and Biological Learning Society
94. Hardt, M., & Ma, T. (2016). Identity matters in deep learning. arXiv 1611.04231
95. Harris, K. D., & Shepherd, G. M. (2015). The neocortical circuit: Themes and variations. Nature Neuroscience, 18(2), 170-181
96. He, K., & Sun, J. (2015). Convolutional neural networks at constrained time cost. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5353-5360). Red Hook, NY: Curran
97. He, K., Zhang, X., Ren, S., & Sun, J. (2014). Spatial pyramid pooling in deep convolutional networks for visual recognition. In Proceedings of the European Conference on Computer Vision (pp. 346-361). Berlin: Springer
98. He, K., Zhang, X., Ren, S., & Sun, J. (2015a). Deep residual learning for image recognition. arXiv 1512.03385
99. He, K., Zhang, X., Ren, S., & Sun, J. (2015b). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1026-1034). Red Hook, NY: Curran
100. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Identity mappings in deep residual networks. arXiv 1603.05027
101. Hinton, G. E. (1989). Connectionist learning procedures. Artificial Intelligence, 40(1), 185-234
102. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1771-1800
103. Hinton, G. E., Osindero, S., & Teh, Y. (2006). Afast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527-1554
104. Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504-507
105. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv 1207.0580
106. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv 1503.02531
107. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780
108. Huang, F. J., & LeCun, Y. (2006). Large-scale learning with SVM and convolutional nets for generic object categorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 284-291). Red Hook, NY: Curran
109. Huang, G. B., Ramesh, M., Berg, T., & Learned-Miller, E. (2007). Labeled faces in the wild: A database for studying face recognition in unconstrained environments (vol. 1, p. 3) (Technical Report 07-49). Amherst: University of Massachusetts
110. Huang, G., Liu, Z., Weinberger, K. Q., & van der Maaten, L. (2016). Densely connected convolutional networks. arXiv 1608.06993
111. Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016). Deep networks with stochastic depth. In Proceedings of the European Conference on Computer Vision (pp. 646-661). Heidelberg, Berlin: Springer
112. Huang, R., Xu, B., Schuurmans, D., & Szepesvári, C. (2016). Learning with a strong adversary. arXiv 1511.03034v6
113. Hubel, D. H., & Wiesel, T. N. (1959). Receptive fields of single neurones in the cat's striate cortex. Journal of Physiology, 148(1), 574-591
114. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. Journal of Physiology, 160(1), 106-154
115. Hyvärinen, A., & Köster, U. (2007). Complex cell pooling and the statistics of natural images. Network: Computation in Neural Systems, 18(2), 81-100
116. Iandola, F. N., Moskewicz, M. W., Ashraf, K., Han, S., Dally, W. J., & Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size. arXiv 1602.07360
117. Ioffe, S. (2017). Batch renormalization: Towards reducing minibatch dependence in batchnormalized models. arXiv 1702.03275
118. Ioffe, S., &Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference Machine Learning (pp. 448-456). N.p.: International Machine Learning Society
119. Ivakhnenko, A. G., &Lapa, V. G. (1966). Cybernetic predicting devices. New York:CCM Information Corp
120. Jaderberg, M., Simonyan, K., & Zisserman, A. (2015). Spatial transformer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems, 28 (pp. 2017-2025). Red Hook, NY: Curran
121. Jaderberg, M., Vedaldi, A., &Zisserman, A. (2014). Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference (pp. 1-12). Durham, UK: BMVAPress
122. Jarrett, K., Kavukcuoglu, K., & LeCun, Y. (2009). What is the best multi-stage architecture for object recognition? In Proceedings of the IEEE International Conference on Computer Vision (pp. 2146-2153). Red Hook, NY: Curran
123. Jegou, H., Perronnin, F., Douze, M., Sánchez, J., Perez, P., & Schmid, C. (2012). Aggregating local image descriptors into compact codes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(9), 1704-1716
124. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., . Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM International Conference on Multimedia (pp. 675-678). New York: ACM
125. Jin, J., Dundar, A., &Culurciello, E. (2016). Robust convolutional neural networks under adversarial noise. In Proceedings of the 4th International Conference on Learning Representations (pp. 1-8). N.p.: Computational and Biological Learning Society
126. Jin, X., Xu, C., Feng, J., Wei, Y., Xiong, J., & Yan, S. (2015). Deep learning with S-shaped rectified linear activation units. arXiv 1512.07030
127. Kalchbrenner, N., Danihelka, I., & Graves, A. (2015). Grid long short-term memory. arXiv 1507.01526
128. Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016). Neural machine translation in linear time. arXiv 1610.10099
129. Karpathy, A. (2016). CS231n: Convolutional neural networks for visual recognition. http://cs231n.github.io/classification/
130. Karpathy, A., & Fei-Fei, L. (2016). Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3128-3137). Red Hook, NY: Curran
131. Kavukcuoglu, K., Ranzato, M., Fergus, R., & LeCun, Y. (2009). Learning invariant features through topographic filter maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1605-1612). Red Hook, NY: Curran
132. Kavukcuoglu, K., Ranzato, M., & LeCun, Y. (2010). Fast inference in sparse coding algorithms with applications to object recognition. arXiv 1010.3467
133. Kennedy, J., &Eberhart, R. C. (1995, October).Anew optimizer using particle swarm theory. In Proceedings of the 6th International Symposium on Micro Machine and Human Science (pp. 39-43). Piscataway, NJ: IEEE
134. Kim, M., & Smaragdis, P. (2016). Bitwise neural networks. arXiv 1601.06071
135. Kim, Y., Park, E., Yoo, S., Choi, T., Yang, L., & Shin, D. (2015). Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv 1511.06530
136. Kingma, D., &Ba, J. (2014). Adam: Amethod for stochastic optimization. arXiv 1412.6980
137. Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. arXiv 1312.6114 v10
138. Koushik, J. (2016). Understanding convolutional neural networks. arXiv 1605.09081
139. Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, Canada
140. Krizhevsky, A. (2014). One weird trick for parallelizing convolutional neural networks. arXiv 1404.5997
141. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 25 (pp. 1097-1105). Red Hook, NY: Curran
142. Kulkarni, T. D., Whitney, W. F., Kohli, P., & Tenenbaum, J. (2015). Deep convolutional inverse graphics network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systemsm, 28 (pp. 2539-2547). Red Hook, NY: Curran
143. Kumar, N., Berg, A. C., Belhumeur, P. N., & Nayar, S. K. (2009). Attribute and simile classifiers for face verification. In Proceedings of the IEEE International Conference on Computer Vision (pp. 365-372). Red Hook, NY: Curran
144. Laptev, D., Savinov, N., Buhmann, J. M., & Pollefeys, M. (2016). TI-POOLING: Transformation-invariant pooling for feature learning in convolutional neural networks. arXiv 1604.06318
145. Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th International Conference on Machine Learning (pp. 473-480). N.p.: International Machine Learning Society
146. Lavin, A., & Gray, S. (2016). Fast algorithms for convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4013-4021). Red Hook, NY: Curran
147. Lawrence, S., Giles, C. L., Tsoi, A. C., & Back, A. D. (1997). Face recognition: A convolutional neural-network approach. IEEE Transactions on Neural Networks, 8(1), 98-113
148. Lazebnik, S., Schmid, C., & Ponce, J. (2005). A sparse texture representation using local affine regions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 1265-1278
149. Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2169-2178). Red Hook, NY: Curran
150. Learned-Miller, E., Huang, G. B., RoyChowdhury, A., Li, H., & Hua, G. (2016). Labeled faces in thewild:Asurvey. In M. Kawulok, M. E. Celebi, &B. Smolka (Eds.), Advances in face detection and facial image analysis (pp. 189-248). Cham, Switzerland: Springer
151. Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I., & Lempitsky, V. (2014). Speedingup convolutional neural networks using fine-tuned CP-decomposition. arXiv 1412.6553
152. LeCun, Y. (1989). Generalization and network design strategies. In R. Pfeifer, Z. Schreter, F. Fogelman, & L. Steels (Eds.), Connections in perspective (pp. 143-155). Zurich, Switzerland: Elsevier
153. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444
154. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989a). Handwritten digit recognition with a back-propagation network. In D. S. Touretzky (Ed.), Advances in neural information processing systems, 2 (pp. 396-404). Cambridge, MA: MIT Press
155. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989b). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4), 541-551
156. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324
157. LeCun, Y., Huang, F. J., & Bottou, L. (2004). Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 97-104). Red Hook, NY: Curran
158. LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010). Convolutional networks and applications in vision. In Proceedings of the IEEE International Symposium on Circuits and Systems (pp. 253-256). Red Hook, NY: Curran
159. Lee, C., Gallagher, P. W., & Tu, Z. (2016). Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (pp. 464-472). www.jmlr.org/proceedings/papers/v51/lee16a.pdf
160. Lee, C., Xie, S., Gallagher, P., Zhang, Z., & Tu, Z. (2015). Deeply-supervised nets. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics (pp. 562-570). jmlr.org/proceedings/papers/v38/lee15a.pdf
161. Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th International ConferenceMachine Learning (pp. 609-616).N.p.: International Machine Learning Society
162. Leibo, J. Z., Mutch, J., & Poggio, T. (2011). Why the brain separates face recognition from object recognition. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 24 (pp. 711-719). Red Hook, NY: Curran
163. Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016). End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39), 1-40
164. Li, F., Zhang, B., & Liu, B. (2016). Ternary weight networks. arXiv 1605.04711
165. Li, S., Jiao, J., Han, Y., &Weissman, T. (2016). Demystifying ResNet. arXiv 1611.01186
166. Li, Z., Gong, B., & Yang, T. (2016). Improved dropout for shallow and deep learning. In D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in neural information processing systemsm (pp. 1-9). N.p.: Preproceedings
167. Liang, M., & Hu, X. (2015). Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3367-3375). Red Hook, NY: Curran
168. Liao, Z., & Carneiro, G. (2016). On the importance of normalisation layers in deep learning with piecewise linear activation units. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (pp. 1-8). Red Hook, NY: Curran
169. Lin, J., Morere, O., Chandrasekhar, V., Veillard, A., & Goh, H. (2015). Deephash: Getting regularization, depth and fine-tuning right. arXiv 1501.04711
170. Lin, M., Chen, Q., & Yan, S. (2013). Network in network. arXiv 1312.4400
171. Lin, Y., Lv, F., Zhu, S., Yang, M., Cour, T., Yu, K., . Huang, T. (2011). Large-scale image classification: Fast feature extraction and SVM training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1689-1696). Red Hook, NY: Curran
172. Linnainmaa, S. (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's thesis, University of Helsinki, Finland
173. Littwin, E., &Wolf, L. (2016). The loss surface of residual networks: Ensembles and the role of batch normalization. arXiv 1611.02525
174. Liu, H., Tian, Y., Yang, Y., Pang, L., & Huang, T. (2016). Deep relative distance learning: Tell the difference between similar vehicles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2167-2175). Red Hook, NY: Curran
175. Liu, W., Wen, Y., Scut, M., Yu, Z., & Yang, M. (2016). Large-margin softmax loss for convolutional neural networks. In Proceedings of the 33rd International Conference Machine Learning (pp. 507-516). N.p.: International Machine Learning Society
176. Maas, A. L., Hannun, A. Y., &Ng, A. Y. (2013). Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th International ConferenceMachine Learning (pp. 1-8). N.p.: International Machine Learning Society
177. Maharaj, A. V. (2015). Improving the adversarial robustness of ConvNets by reduction of input dimensionality. Stanford, CA: Department of Physics, Stanford University
178. Mairal, J., Bach, F., Ponce, J., Sapiro, G., & Zisserman, A. (2008). Discriminative learned dictionaries for local image analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1-8). Red Hook, NY: Curran
179. Malinowski, M., & Fritz, M. (2013). Learnable pooling regions for image classification. arXiv 1301.3516
180. Mallat, S. (2012). Group invariant scattering. Communications on Pure and Applied Mathematics, 65(10), 1331-1398. doi:10.1002/cpa.21413
181. Masci, J., Meier, U., Cireşan, D., & Schmidhuber, J. (2011). Stacked convolutional autoencoders for hierarchical feature extraction. In Proceedings of the 21th International Conference on Artificial Neural Networks (pp. 52-59). Berlin: Springer
182. Mathieu, M., Henaff, M., & LeCun, Y. (2013). Fast training of convolutional networks through FFTs. arXiv 1312.5851
183. Mishkin, D., & Matas, J. (2016). All you need is a good init. In Proceedings of the 4th International Conference on Learning Representations (pp. 1-13). N.p.: Computational and Biological Learning Society
184. Miyato, T., Maeda, S., Koyama, M., Nakae, K., & Ishii, S. (2016). Distributional smoothing with virtual adversarial training. In Proceedings of the 5th International Conference on Learning Representations (pp. 1-12). N.p.: Computational and Biological Learning Society
185. Montavon, G., Orr, G. B., & Müller, K. (Eds.). (2012). Neural networks: Tricks of the trade (2nd ed.). Berlin: Springer
186. Muller, U., Ben, J., Cosatto, E., Flepp, B., & LeCun, Y. (2005). Off-road obstacle avoidance through end-to-end learning. In Y.Weiss, P. B. Schölkopf, & J. C. Platt (Eds.), Advances in neural information processing systems, 18 (pp. 739-746). Cambridge, MA: MIT Press
187. Nagi, J., Di Caro, G. A., Giusti, A., Nagi, F., & Gambardella, L. M. (2012). Convolutional neural support vector machines: Hybrid visual pattern classifiers for multi-robot systems. In Proceedings of the 11th International Conference on Machine Learning and Applications (pp. 27-32). Los Alamitos, CA: IEEE Computer Society
188. Nagi, J., Ducatelle, F., Di Caro, G. A., Cireşan, D., Meier, U., Giusti, A., . Gambardella, L. M. (2011). Max-pooling convolutional neural networks for visionbased hand gesture recognition. In Proceedings of the IEEE International Conference on Signal and Image Processing Applications (pp. 342-347). Red Hook, NY: Curran
189. Nair, V., & Hinton, G. E. (2009). 3D object recognition with deep belief nets. In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, & A. Culotta (Eds.), Advances in neural information processing systems, 22 (pp. 1339-1347). Red Hook, NY: Curran
190. Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (pp. 807-814). N.p.: International Machine Learning Society
191. Nasse, F., Thurau, C., & Fink, G. A. (2009). Face detection using GPU-based convolutional neural networks. In Proceedings of the 13th International Conference on Computer Analysis of Images and Patterns (pp. 83-90). Berlin: Springer
192. National Data Science Bowl | Kaggle. (2016). Kaggle.com. https://www.kaggle.com/c/datasciencebowl
193. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., & Ng, A. Y. (2011). Reading digits in natural imageswith unsupervised feature learning. In Advances in neural information processing systems, 24 (NIPS)Workshop on Deep Learning and Unsupervised Feature Learning (pp. 1-9). Red Hook, NY: Curran
194. Ngiam, J., Chen, Z., Chia, D., Koh, P. W., Le, Q. V., & Ng, A. Y. (2010). Tiled convolutional neural networks. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, & A. Culotta (Eds.), Advances in neural information processing systems, 23 (pp. 1279-1287). Red Hook, NY: Curran
195. Nguyen, A., Yosinski, J., & Clune, J. (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 427-436). Los Alamitos, CA: IEEE Computer Society
196. Ning, F., Delhomme, D., LeCun, Y., Piano, F., Bottou, L., & Barbano, P. E. (2005). Toward automatic phenotyping of developing embryos from videos. IEEE Transactions on Image Processing, 14(9), 1360-1371
197. Novikov, A., Podoprikhin, D., Osokin, A., & Vetrov, D. P. (2015). Tensorizing neural networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems, 28 (pp. 442-450). Red Hook, NY: Curran
198. Nowlan, S. J., & Hinton, G. E. (1992). Simplifying neural networks by soft weightsharing. Neural Computation, 4(4), 473-493
199. Oh, K., & Jung, K. (2004). GPU implementation of neural networks. Pattern Recognition, 37(6), 1311-1314
200. Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., . Kavukcuoglu, K. (2016). Wavenet: A generative model for raw audio. CoRR abs/1609.03499
201. Orhan, A. E. (2017). Skip connections as effective symmetry-breaking. arXiv 1701.09175
202. Oseledets, I. V. (2011). Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5), 2295-2317
203. Oxholm, G., Bariya, P., & Nishino, K. (2012). The scale of geometric texture. In Proceedings of the European Conference on Computer Vision (pp. 58-71). Berlin: Springer
204. Paine, T., Jin, H., Yang, J., Lin, Z., & Huang, T. (2013). GPU asynchronous stochastic gradient descent to speed up neural network training. arXiv 1312.6186
205. Papernot, N., McDaniel, P., Wu, X., Jha, S., & Swami, A. (2016). Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of the 37th IEEE Symposium on Security and Privacy (pp. 1-16). Los Alamitos, CA: IEEE Computer Society
206. Pereyra, G., Tucker, G., Chorowski, J., Kaiser, L., & Hinton, G. (2017). Regularizing neural networks by penalizing confident output distributions. arXiv 1701.06548v1
207. Perronnin, F., Sánchez, J., &Mensink, T. (2010). Improving the Fisher kernel for largescale image classification. In Proceedings of the European Conference on Computer Vision (pp. 143-156). Berlin: Springer
208. Pinto, L., & Gupta, A. (2015). Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. arXiv 1509.06825
209. Qiao, Y., Shen, J., Xiao, T., Yang, Q., Wen, M., & Zhang, C. (2016). FPGA-accelerated deep convolutional neural networks for high throughput and energy efficiency. Concurrency and Computation: Practice and Experience. doi:10.1002/cpe.3850
210. Qian, N. (1999). On the momentum term in gradient descent learning algorithms. Neural Networks, 12, 145-150
211. Ranzato, M. A., Huang, F. J., Boureau, Y., & LeCun, Y. (2007). Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (pp. 1-8). Los Alamitos, CA: IEEE Computer Society
212. Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2006). Efficient learning of sparse representations with an energy-based model. In P. B. Schölkopf, J. C. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems, 19 (pp. 1137-1144). Cambridge, MA: MIT Press
213. Ranzato, M. A., & Szummer, M. (2008). Semi-supervised learning of compact document representations with deep networks. In Proceedings of the 25th International Conference on Machine Learning (pp. 792-799). N.p.: International Machine Learning Society
214. Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016). Xnor-net: Imagenet classification using binary convolutional neural networks. In Proceedings of the European Conference on Computer Vision (pp. 525-542). Berlin: Springer
215. Razavian, A., Azizpour, H., Sullivan, J., & Carlsson, S. (2014). CNN features off-theshelf: An astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern RecognitionWorkshops (pp. 806-813). Los Alamitos, CA: IEEE Computer Society
216. Recht, B., Re, C., Wright, S., & Niu, F. (2011). Hogwild: Alock-free approach to parallelizing stochastic gradient descent. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 24 (pp. 693-701). Red Hook, NY: Curran
217. Rippel, O., Gelbart, M. A., & Adams, R. P. (2014). Learning ordered representations with nested dropout. In Proceedings of the 30th International Conference on Machine Learning (pp. 1746-1754). N.p.: International Machine Learning Society
218. Rippel, O., Snoek, J., & Adams, R. P. (2015). Spectral representations for convolutional neural networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems, 28 (pp. 2449-2457). Red Hook, NY: Curran
219. Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). Fitnets: Hints for thin deep nets. In Proceedings of the 3rd International Conference on Learning Representations (pp. 1-13). N.p.: Computational and Biological Learning Society
220. Rumelhart, D. E., Hinton, G. E., &Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536
221. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., . Bernstein, M. (2015). ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211-252
222. Sabour, S., Cao, Y., Faghri, F., & Fleet, D. J. (2016). Adversarial manipulation of deep representations. In Proceedings of the 4th International Conference on Learning Representations (pp. 1-18). N.p.: Computational and Biological Learning Society
223. Sainath, T.N., Kingsbury, B., Mohamed, A., Dahl, G. E., Saon, G., Soltau, H., . Ramabhadran, B. (2013). Improvements to deep convolutional neural networks for LVCSR. In 2013 IEEEWorkshop on Automatic Speech Recognition and Understanding (pp. 315-320). Red Hook, NY: Curran
224. Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B. (2013). Low-rank matrix factorization for deep neural network training with highdimensional output targets. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (pp. 6655-6659. N.p.: IEEE Signal Processing Society)
225. Salakhutdinov, R., & Hinton, G. E. (2007). Learning a nonlinear embedding by preserving class neighbourhood structure. In Proceedings of the 11th International Conference on Artificial Intelligence and Statistics (pp. 412-419). http://www.jmlr.org/proceedings/papers/v2/salakhutdinov07a/salakhutdinov07a.pdf
226. Sánchez, J., & Perronnin, F. (2011). High-dimensional signature compression for large-scale image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1665-1672). Los Alamitos, CA: IEEE Computer Society
227. Saxe, A. M., McClelland, J. L., & Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv 1312.6120
228. Sercu, T., & Goel, V. (2016). Dense prediction on sequences with time-dilated convolutions for speech recognition. arXiv 1611.09288
229. Scherer, D., Müller, A., & Behnke, S. (2010). Evaluation of pooling operations in convolutional architectures for object recognition. In Proceedings of the 20th International Conference on Artificial Neural Networks (pp. 92-101). Berlin: Springer
230. Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85-117
231. Schroff, F., Kalenichenko, D., & Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 815-823). Los Alamitos, CA: IEEE Computer Society
232. Sermanet, P., Chintala, S., & LeCun, Y. (2012). Convolutional neural networks applied to house numbers digit classification. In Proceedings of the 21st International Conference on Pattern Recognition (pp. 3288-3291). Red Hook, NY: Curran
233. Simard, P. Y., Steinkraus, D., & Platt, J. C. (2003, August). Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the 7th International Conference on Document Analysis and Recognition (vol. 3, pp. 958-963). Washington, DC: IEEE Computer Society
234. Simoncelli, E. P., & Heeger, D. J. (1998). Amodel of neuronal responses in visual area MT. Vision Research, 38(5), 743-761
235. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv 1409.1556
236. Smirnov, E. A., Timoshenko, D. M., & Andrianov, S. N. (2014). Comparison of regularization methods for ImageNet classification with deep convolutional neural networks. AASRI Procedia, 6, 89-94
237. Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. In F. Pereira, C. J. C. Burges, L. Bottou, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 25 (pp. 2951-2959). Red Hook, NY: Curran
238. Sontag, E. D. (1998). VC dimension of neural networks. NATO ASI Series F Computer and Systems Sciences, 168, 69-96
239. Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014). Striving for simplicity: The all convolutional net. arXiv 1412.6806
240. Springenberg, J. T., & Riedmiller, M. (2013). Improving deep neural networks with probabilistic maxout units. arXiv 1312.6116
241. Srinivas, S., Sarvadevabhatla, R. K., Mopuri, K. R., Prabhu, N., Kruthiventi, S. S., & Babu, R. V. (2016). A taxonomy of deep convolutional neural nets for computer vision. arXiv 1601.06615
242. Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: Asimple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1), 1929-1958
243. Srivastava, N., & Salakhutdinov, R. R. (2013). Discriminative transfer learning with tree-based priors. In C. J. C. Burges, L. Bottou, M.Welling, Z. Ghahramani, &K. Q. Weinberger (Eds.), Advances in neural information processing systems, 26 (pp. 2094-2102). Red Hook, NY: Curran
244. Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015a). Training very deep networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems, 28 (pp. 2377-2385). Red Hook, NY: Curran
245. Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015b). Highway networks. arXiv 1505.00387
246. Stallkamp, J., Schlipsing, M., Salmen, J., & Igel, C. (2011, July). The German traffic sign recognition benchmark: A multi-class classification competition. In Proceedings of the IEEE International Joint Conference on Neural Networks (pp. 1453-1460). Red Hook, NY: Curran
247. Steinkrau, D., Simard, P. Y., & Buck, I. (2005). Using GPUs for machine learning algorithms. In Proceedings of the 8th International Conference on Document Analysis and Recognition (pp. 1115-1119).Washington, DC: IEEE Computer Society
248. Stollenga, M. F., Masci, J., Gomez, F., & Schmidhuber, J. (2014). Deep networks with internal selective attention through feedback connections. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 3545-3553). Red Hook, NY: Curran
249. Sun, Y., Chen, Y., Wang, X., & Tang, X. (2014). Deep learning face representation by joint identification-verification. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 1988-1996). Red Hook, NY: Curran
250. Sun, Y., Liang, D., Wang, X., & Tang, X. (2015). Deepid3: Face recognition with very deep neural networks. arXiv 1502.00873
251. Sun, Y., Wang, X., & Tang, X. (2014). Deep learning face representation from predicting 10, 000 classes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1891-1898). Los Alamitos, CA: IEEE Computer Society
252. Sun, Y., Wang, X., & Tang, X. (2015). Deeply learned face representations are sparse, selective, and robust. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2892-2900). Los Alamitos, CA: IEEE Computer Society
253. Sussillo, D., & Abbott, L. (2014). Random walk initialization for training very deep feedforward networks. arXiv 1412.6558
254. Sutskever, I., Martens, J., Dahl, G. E., & Hinton, G. E. (2013). On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference Machine Learning (pp. 1139-1147). N.p.: International Machine Learning Society
255. Szegedy, C., Ioffe, S., & Vanhoucke, V. (2016). Inception-v4, Inception-Resnet and the impact of residual connections on learning. arXiv 1602.07261
256. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., . Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1-9). Los Alamitos, CA: IEEE Computer Society
257. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2015). Rethinking the Inception architecture for computer vision. arXiv 1512.00567
258. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., &Fergus, R. (2014). Intriguing properties of neural networks. In Proceedings of the 1st International Conference on Learning Representations (pp. 1-10). N.p.: Computational and Biological Learning Society
259. Tabacof, P., & Valle, E. (2016). Exploring the space of adversarial images. In Proceedings of the International Joint Conference on Neural Networks (pp. 1-8). Red Hook, NY: Curran
260. Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014). Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1701-1708). Los Alamitos, CA: IEEE Computer Society
261. Tang, Y. (2013). Deep learning using linear support vector machines. arXiv 1306.0239
262. Tompson, J., Goroshin, R., Jain, A., LeCun, Y., & Bregler, C. (2015). Efficient object localization using convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 648-656). Los Alamitos, CA: IEEE Computer Society
263. Tucker, L. R. (1966). Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3), 279-311
264. Turaga, S. C., Murray, J. F., Jain, V., Roth, F., Helmstaedter, M., Briggman, K., . Seung, H. S. (2010). Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation, 22(2), 511-538
265. Ulicný, M., Lundström, J., & Byttner, S. (2016). Robustness of deep convolutional neural networks for image recognition. In Proceedings of the 1st International Symposium on Intelligent Computing Systems (pp. 16-30). Switzerland: Springer International Publishing
266. Van Dyk, D. A., & Meng, X. (2012). The art of data augmentation. Journal of Computational and Graphical Statistics, 10(1), 1-50
267. Vapnik, V. (1995). The nature of statistical learning theory. New York: Springer Science & Business Media
268. Vapnik, V. N., & Chervonenkis, A. Y. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications. 16(2), 11-30
269. Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). Show and tell: Aneural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3156-3164). Red Hook, NY: Curran
270. Wager, S., Wang, S., & Liang, P. S. (2013). Dropout training as adaptive regularization. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 26 (pp. 351-359). Red Hook, NY: Curran
271. Wan, L., Zeiler, M., Zhang, S., LeCun, Y., & Fergus, R. (2013). Regularization of neural networks using Dropconnect. In Proceedings of the 30th International Conference Machine Learning (pp. 1058-1066). N.p.: International Machine Learning Society
272. Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., & Xu, W. (2016). CNN-RNN: A unified framework for multi-label image classification. arXiv 1604.04573
273. Wang, S. I., & Manning, C. D. (2013). Fast dropout training. In Proceedings of the 30th International Conference Machine Learning (pp. 118-126). N.p.: International Machine Learning Society
274. Wang, Y., Xu, C., You, S., Tao, D., & Xu, C. (2016). CNNpack: Packing convolutional neural networks in the frequency domain. InD. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in neural information processing systems, 29 (pp. 1-9). N.p.: Preproceedings
275. Wang, Z., & Oates, T. (2015). Encoding time series as images for visual inspection and classification using tiled convolutional neural networks. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence (pp. 40-46)
276. Warde-Farley, D., Goodfellow, I. J., Courville, A., & Bengio, Y. (2013). An empirical analysis of dropout in piecewise linear networks. arXiv 1312.6197
277. Weinberger, K. Q., Blitzer, J., & Saul, L. K. (2005). Distance metric learning for large margin nearest neighbor classification. In Y. Weiss, P. B. Schölkopf, & J. C. Platt (Eds.), Advances in neural information processing systems, 18 (pp. 1473-1480). Cambridge, MA: MIT Press
278. Werbos, P. (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences. Doctoral dissertation, Harvard University
279. Werbos, P. J. (1982). Applications of advances in nonlinear sensitivity analysis. In R. F. Drenick & F. Kozin (Eds.), System modeling and optimization (pp. 762-770). Berlin: Springer
280. Weston, J., Ratle, F., Mobahi, H., & Collobert, R. (2008). Deep learning via semisupervised embedding. In Proceedings of the 25th International Conference onMachine Learning (pp. 1168-1175). N.p.: International Machine Learning Society
281. Wiatowski, T., & Bölcskei, H. (2015). A mathematical theory of deep convolutional neural networks for feature extraction. arXiv 1512.06293
282. Winograd, S. (1980). Arithmetic complexity of computations. Philadelphia: SIAM
283. Wolf, L., Hassner, T., &Maoz, I. (2011). Face recognition in unconstrained videos with matched background similarity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 529-534). Red Hook, NY: Curran
284. Wu, H., & Gu, X. (2015). Max-pooling dropout for regularization of convolutional neural networks. In Proceedings of the 22nd International Conference Neural Information Processing (pp. 46-53). Berlin: Springer
285. Wu, J., Yu, Y., Huang, C., & Yu, K. (2015). Deep multiple instance learning for image classification and auto-annotation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3460-3469). Red Hook, NY: Curran
286. Xie, L., Wang, J., Wei, Z., Wang, M., & Tian, Q. (2016). DisturbLabel: Regularizing CNN on the loss layer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4753-4762). Red Hook, NY: Curran
287. Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2016). Aggregated residual transformations for deep neural networks. arXiv 1611.05431
288. Xu, B., Wang, N., Chen, T., & Li, M. (2015). Empirical evaluation of rectified activations in convolutional network. arXiv 1505.00853v2
289. Yadan, O., Adams, K., Taigman, Y., & Ranzato, M. (2014). Multi-GPU training of convnets. arXiv 1312.5853v4
290. Yang, J., Yu, K., Gong, Y., & Huang, T. (2009). Linear spatial pyramid matching using sparse coding for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1794-1801). Red Hook, NY: Curran
291. Yu, D., Wang, H., Chen, P., & Wei, Z. (2014). Mixed pooling for convolutional neural networks. In Proceedings of the 9th International Conference on Rough Sets and Knowledge Technology (pp. 364-375). Berlin: Springer
292. Yu, F., & Koltun, V. (2015). Multi-scale context aggregation by dilated convolutions. arXiv 1511.07122
293. Yu, W., Yang, K., Bai, Y., Yao, H., & Rui, Y. (2014a). DNN flow: DNN feature pyramid based image matching. In Proceedings of the British Machine Vision Conference (pp. 1-10). Durham, UK: BMVA Press
294. Yu, W., Yang, K., Bai, Y., Yao, H., & Rui, Y. (2014b). Visualizing and comparing convolutional neural networks. arXiv 1412.6631
295. Zagoruyko, S., & Komodakis, N. (2017). Wide residual networks. arXiv 1605.07146v3
296. Zeiler, M. D. (2012). ADADELTA: An adaptive learning rate method. arXiv 1212.5701
297. Zeiler, M. D., & Fergus, R. (2013). Stochastic pooling for regularization of deep convolutional neural networks. arXiv 1301.3557
298. Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision (pp. 818-833). Berlin: Springer
299. Zeiler, M. D., Taylor, G.W., & Fergus, R. (2011). Adaptive deconvolutional networks for mid and high level feature learning. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2018-2025). Red Hook, NY: Curran
300. Zhai, S., Cheng, Y., & Zhang, Z. M. (2016). Doubly convolutional neural networks. In D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in neural information processing systems, 29 (pp. 1-9). N.p.: Preproceedings
301. Zhang, H., Berg, A. C., Maire, M., &Malik, J. (2006). SVM-KNN: Discriminative nearest neighbor classification for visual category recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2126-2136). Red Hook, NY: Curran
302. Zhao, Q., & Griffin, L. D. (2016). Suppressing the unusual: Towards robust CNNs using symmetric activation functions. arXiv 1603.05145v1
303. Zhou, E., Cao, Z., &Yin, Q. (2015). Naive-deep face recognition: Touching the limit of LFW benchmark or not? arXiv 1501.04690
304. Zhu, Z., Luo, P., Wang, X., & Tang, X. (2014). Recover canonical-view faces in the wild with deep neural networks. arXiv 1404.3543
305. Zhuang, Y., Chin, W., Juan, Y., & Lin, C. (2013). A fast parallel SGD for matrix factorization in shared memory systems. Proceedings of the 7th ACM Conference on Recommender Systems (pp. 249-256). New York: ACM
306. Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010). Parallelized stochastic gradient descent. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, & A. Culotta (Eds.), Advances in neural information processing systemsm, 23 (pp. 2595-2603). Red Hook, NY: Curran