Normalizing the normalizers: Comparing and extending network normalization schemes

5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings

Volumn , Issue , 2017, Pages

  Ren, Mengye ^a   Liao, Renjie ^a   Urtasun, Raquel ^a   Sinz, Fabian H ^b   Zemel, Richard S ^a,c  

^a UNIVERSITY OF TORONTO   (Canada)

^b BAYLOR COLLEGE OF MEDICINE   (United States)

^c CANADIAN INSTITUTE FOR ADVANCED RESEARCH   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

CHEMICAL ACTIVATION; IMAGE ENHANCEMENT; UNIFIED MODELING LANGUAGE;

LANGUAGE MODEL; REGULARIZER; SPEED UP; SUPER RESOLUTION;

RECURRENT NEURAL NETWORKS;

EID: 85088228386     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (51)

1. Sparse coding via thresholding and local competition in neural circuits. Neural Computation, 20(10): 2526-63, 2008. ISSN 08997667. doi: 10.1162/neco.2008.03-07-486.
2. Abadi, Martín, Barham, Paul, Chen, Jianmin, Chen, Zhifeng, Davis, Andy, Dean, Jeffrey, Devin, Matthieu, Ghemawat, Sanjay, Irving, Geoffrey, Isard, Michael, Kudlur, Manjunath, Levenberg, Josh, Monga, Rajat, Moore, Sherry, Murray, Derek Gordon, Steiner, Benoit, Tucker, Paul A., Vasudevan, Vijay, Warden, Pete, Wicke, Martin, Yu, Yuan, and Zhang, Xiaoqiang. Tensorflow: A system for large-scale machine learning. CoRR, abs/1605.08695, 2016.
3. Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. CoRR, abs/1607.06450, 2016.
4. Ballé, Johannes, Laparra, Valero, and Simoncelli, Eero P. Density modeling of images using a generalized normalization transformation. ICLR, 2016.
5. Beck, J. M., Latham, P. E., and Pouget, A. Marginalization in Neural Circuits with Divisive Normalization. The Journal of neuroscience: the official journal of the Society for Neuroscience, 31(43):15310-9, oct 2011. ISSN 1529-2401. doi: 10.1523/JNEUROSCI.1706-11.2011.
6. Bevilacqua, Marco, Roumy, Aline, Guillemot, Christine, and Morel, Marie-Line Alberi. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012.
7. Bonds, A. B. Role of Inhibition in the Specification of Orientation Selectivity of Cells in the Cat Striate Cortex. Visual Neuroscience, 2(01):41-55, 1989.
8. Busse, L., Wade, A. R., and Carandini, M. Representation of Concurrent Stimuli by Population Activity in Visual Cortex. Neuron, 64(6):931-942, dec 2009. ISSN 0896-6273. doi: 10.1016/j. neuron.2009.11.004.
9. Carandini, M. and Heeger, D. J. Normalization as a canonical neural computation. Nature reviews. Neuroscience, 13(1):51-62, nov 2012. ISSN 1471-0048. doi: 10.1038/nrn3136.
10. Coen-Cagli, R., Kohn, A., and Schwartz, O. Flexible gating of contextual influences in natural vision. Nature Neuroscience, 18(11):1648-1655, 2015. ISSN 1097-6256. doi: 10.1038/nn.4128.
11. Cogswell, Michael, Ahmed, Faruk, Girshick, Ross, Zitnick, Larry, and Batra, Dhruv. Reducing overfitting in deep networks by decorrelating representations. ICLR, 2015.
12. Cooijmans, Tim, Ballas, Nicolas, Laurent, César, and Courville, Aaron. Recurrent batch normalization. CoRR, abs/1603.09025, 2016.
13. Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
14. Dong, Chao, Loy, Chen Change, He, Kaiming, and Tang, Xiaoou. Image super-resolution using deep convolutional networks. TPAMI, 38(2):295-307, 2016.
15. Froudarakis, Emmanouil, Berens, Philipp, Ecker, Alexander S, Cotton, R James, Sinz, Fabian H, Yatsenko, Dimitri, Saggau, Peter, Bethge, Matthias, and Tolias, Andreas S. Population code in mouse V1 facilitates readout of natural scenes through increased sparseness. Nature neuroscience, 17(6):851-7, apr 2014. ISSN 1546-1726. doi: 10.1038/nn.3707.
16. Gatys, Leon A., Ecker, Alexander S., and Bethge, Matthias. Image style transfer using convolutional neural networks. In CVPR, 2016.
17. Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier neural networks. In AISTATS, 2011.
18. Goodfellow, Ian, Bengio, Yoshua, and Courville, Aaron. Deep learning. Book in preparation for MIT Press, 2016.
19. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. In CVPR, 2016.
20. Heeger, D. J. Normalization of cell responses in cat striate cortex. Vis Neurosci, 9(2):181-197, 1992. ISSN 09525238.
21. Higgins, I., Matthey, L., Glorot, X., Pal, A., Uria, B., Blundell, C., Mohamed, S., and Lerchner, A. Early Visual Concept Learning with Unsupervised Deep Learning. CoRR, abs/1606.05579, 2016.
22. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
23. Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., and LeCun, Y. What is the best multi-stage architecture for object recognition? ICCV, 2009.
24. Kavukcuoglu, K., Ranzato, M.'A., Fergus, R., and LeCun, Y. Learning invariant features through topographic filter maps. In CVPR Workshops, 2009.
25. Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks. NIPS, 2012.
26. Laurent, César, Pereyra, Gabriel, Brakel, Philémon, Zhang, Ying, and Bengio, Yoshua. Batch normalized recurrent neural networks. arXiv preprint arXiv:1510.01378, 2015.
27. Le, Quoc V. Building high-level features using large scale unsupervised learning. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 8595-8598. IEEE, 2013.
28. Liao, Q. and Poggio, T. Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex. CoRR, abs/1604.03640, 2016.
29. Liao, Qianli, Kawaguchi, Kenji, and Poggio, Tomaso. Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning. CoRR, abs/1610.06160, 2016a.
30. Liao, Renjie, Schwing, Alexander, Zemel, Richard, and Urtasun, Raquel. Learning deep parsimonious representations. NIPS, 2016b.
31. Lyu, Siwei and Simoncelli, Eero P. Reducing statistical dependencies in natural signals using radial Gaussianization. NIPS, 2008.
32. Malo, J., Epifanio, I., Navarro, R., and Simoncelli, E. P. Nonlinear image representation for efficient perceptual coding. TIP, 15(1):68-80, 2006.
33. Martin, David, Fowlkes, Charless, Tal, Doron, and Malik, Jitendra. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.
34. Olsen, S. R, Bhandawat, V., and Wilson, R. I. Divisive Normalization in Olfactory Population Codes. Neuron, 66(2):287-299, 2010. ISSN 10974199. doi: 10.1016/j.neuron.2010.04.009.
35. Pinto, N., Cox, D. D., and DiCarlo, J. J. Why is Real-World Visual Object Recognition Hard? PLoS Comput Biol, 4(1):e27, jan 2008. doi: 10.1371/journal.pcbi.0040027.
36. Reynolds, J. H. and Heeger, D. J. The normalization model of attention. Neuron, 61(2):168-85, jan 2009. ISSN 1097-4199. doi: 10.1016/j.neuron.2009.01.002.
37. Ringach, D. L. Population coding under normalization. Vision Research, 50(22):2223-2232, 2009. ISSN 18785646. doi: 10.1016/j.visres.2009.12.007.
38. Salimans, Tim and Kingma, Diederik P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In NIPS, 2016.
39. Scardapane, S., Comminiello, D., Hussain, A., and Uncin, A. Group sparse regularization for deep neural networks. CoRR, abs/1607.00485, 2016.
40. Schwartz, O. and Simoncelli, E. P. Natural signal statistics and sensory gain control. Nat Neurosci, 4 (8):819-825, 2001. ISSN 1097-6256. doi: 10.1038/90526.
41. Schwartz, O., J., Sejnowski T., and P., Dayan. Perceptual organization in the tilt illusion. Journal of Vision, 9(4):1-20, apr 2009. ISSN 1534-7362.
42. Sermanet, P., Chintala, S., and LeCun, Y. Convolutional neural networks applied to house numbers digit classification. Proceedings of International Conference on Pattern Recognition ICPR12, (Icpr):10-13, 2012. ISSN 1051-4651. doi: 10.0/Linux-x86 64.
43. Simoncelli, E. P. and Heeger, D. J. A model of neuronal responses in visual area MT. Vision Research, 38(5):743-761, 1998.
44. Sinz, Fabian and Bethge, Matthias. Temporal Adaptation Enhances Efficient Contrast Gain Control on Natural Images. PLoS Computational Biology, 9(1):e1002889, jan 2013. ISSN 1553734X.
45. Sinz, Fabian H and Bethge, Matthias. The Conjoint Effect of Divisive Normalization and Orientation Selectivity on Redundancy Reduction. In NIPS, 2008.
46. Srivastava, Nitish, Hinton, Geoffrey E, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-1958, 2014.
47. Timofte, Radu, De Smet, Vincent, and Van Gool, Luc. Anchored neighborhood regression for fast example-based super-resolution. In ICCV, 2013.
48. Ulyanov, Dmitry, Vedaldi, Andrea, and Lempitsky, Victor S. Instance normalization: The missing ingredient for fast stylization. CoRR, abs/1607.08022, 2016.
49. Wang, Zhou, Bovik, Alan C, Sheikh, Hamid R, and Simoncelli, Eero P. Image quality assessment: from error visibility to structural similarity. TIP, 13(4):600-612, 2004.
50. Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization. CoRR, abs/1409.2329, 2014.
51. Zeyde, Roman, Elad, Michael, and Protter, Matan. On single image scale-up using sparse-representations. In International conference on curves and surfaces, pp. 711-730. Springer, 2010.