Deep Learning of Part-Based Representation of Data Using Sparse Autoencoders with Nonnegativity Constraints

IEEE Transactions on Neural Networks and Learning Systems

Volumn 27, Issue 12, 2016, Pages 2486-2498

  Hosseini Asl, Ehsan ^a   Zurada, Jacek M ^a,b   Nasraoui, Olfa ^a  

^a University of Louisville   (United States)

^b UNIVERSITY OF SOCIAL SCIENCES   (Poland)

Author keywords

Autoencoder; deep architectures; feature learning; nonnegativity constraints; part based representation

Indexed keywords

DEEP LEARNING; DEEP NEURAL NETWORKS; FACTORIZATION; LEARNING SYSTEMS; MATRIX ALGEBRA;

AUTO ENCODERS; DEEP ARCHITECTURES; FEATURE LEARNING; NON-NEGATIVITY CONSTRAINTS; PART-BASED REPRESENTATION;

LEARNING ALGORITHMS;

EID: 85014007036     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2015.2479223     Document Type: Article

References (43)

1. Y. Bengio and Y. LeCun, "Scaling learning algorithms towards AI," Large-Scale Kernel Mach., vol. 34, no. 5, pp. 1-41, 2007.
2. Y. Bengio, "Learning deep architectures for AI," Found. Trends Mach. Learn., vol. 2, no. 1, pp. 1-127, 2009.
3. G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
4. L. Deng, "A tutorial survey of architectures, algorithms, and applications for deep learning," APSIPA Trans. Signal Inf. Process., vol. 3, p. e2, Jan. 2014.
5. S. Bengio, L. Deng, H. Larochelle, H. Lee, and R. Salakhutdinov, "Guest editors' introduction: Special section on learning deep architectures," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1795-1797, Aug. 2013.
6. B. Hutchinson, L. Deng, and D. Yu, "Tensor deep stacking networks," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1944-1957, Aug. 2013.
7. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layer-wise training of deep networks," in Advances in Neural Information Processing Systems, vol. 19. Cambridge, MA, USA: MIT Press, 2007, p. 153.
8. G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, 2006.
9. M. Ranzato, Y.-L. Boureau, and Y. LeCun, "Sparse feature learning for deep belief networks," in Advances in Neural Information Processing Systems, vol. 20. Red Hook, NY, USA: Curran Associates, 2007, pp. 1185-1192.
10. R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng, "Self-taught learning: Transfer learning from unlabeled data," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 759-766.
11. D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio, "Why does unsupervised pre-training help deep learning?" J. Mach. Learn. Res., vol. 11, pp. 625-660, Feb. 2010.
12. P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, "Extracting and composing robust features with denoising autoencoders," in Proc. 25th Int. Conf. Mach. Learn., 2008, pp. 1096-1103.
13. G. E. Hinton, "Training products of experts by minimizing contrastive divergence," Neural Comput., vol. 14, no. 8, pp. 1771-1800, Aug. 2002.
14. H. Lee, C. Ekanadham, and A. Y. Ng, "Sparse deep belief net model for visual area V2," in Advances in Neural Information Processing Systems, vol. 7. Red Hook, NY, USA: Curran Associates, 2007, pp. 873-880.
15. Y. Bengio, A. Courville, and P. Vincent, "Representation learning: A review and new perspectives," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1798-1828, Aug. 2013.
16. D. C. Van Essen and J. L. Gallant, "Neural mechanisms of form and motion processing in the primate visual system," Neuron, vol. 13, no. 1, pp. 1-10, 1994.
17. E. Wachsmuth, M. W. Oram, and D. I. Perrett, "Recognition of objects and their component parts: Responses of single units in the temporal cortex of the macaque," Cerebral Cortex, vol. 4, no. 5, pp. 509-522, 1994.
18. B. A. Olshausen and D. J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by V1?" Vis. Res., vol. 37, no. 23, pp. 3311-3325, 1997.
19. M. Ranzato, C. Poultney, S. Chopra, and Y. LeCun, "Efficient learning of sparse representations with an energy-based model," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2006, pp. 1137-1144.
20. A. Makhzani and B. Frey. (Dec. 2013). "k-sparse autoencoders." [Online]. Available: http://arxiv.org/abs/1312.5663
21. B. A. Olshausen and D. J. Field, "Emergence of simple-cell receptive field properties by learning a sparse code for natural images," Nature, vol. 381, no. 6583, pp. 607-609, 1996.
22. D. D. Lee and H. S. Seung, "Learning the parts of objects by non-negative matrix factorization," Nature, vol. 401, no. 6755, pp. 788-791, 1999.
23. A. Lemme, R. F. Reinhart, and J. J. Steil, "Online learning and generalization of parts-based image representations by non-negative sparse autoencoders," Neural Netw., vol. 33, pp. 194-203, Sep. 2012.
24. J. Chorowski and J. M. Zurada, "Learning understandable neural networks with nonnegative weight constraints," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 1, pp. 62-69, Jan. 2015.
25. T. D. Nguyen, T. Tran, D. Phung, and S. Venkatesh, "Learning parts-based representations with nonnegative restricted Boltzmann machine," in Proc. Asian Conf. Mach. Learn., 2013, pp. 133-148.
26. A. Ng, "Sparse autoencoder," CS294A Lecture notes, 2011, p. 72. [online] Available:https://web.stanford.edu/class/cs294a/sparseAutoencoder-2011new.pdf
27. D. D. Lee and H. S. Seung, "Algorithms for non-negative matrix factorization," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2000, pp. 556-562.
28. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. (Jul. 2012). "Improving neural networks by preventing co-adaptation of feature detectors." [Online]. Available: http://arxiv.org/abs/1207.0580
29. G. E. Hinton and R. S. Zemel, "Autoencoders, minimum description length, and Helmholtz free energy," in Advances in Neural Information Processing Systems. San Mateo, CA, USA: Morgan Kaufmann, 1994, p. 3.
30. V. Nair and G. E. Hinton, "3D object recognition with deep belief nets," in Advances in Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates, 2009, pp. 1339-1347.
31. Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, "Efficient BackProp," in Neural Networks: Tricks of the Trade. Berlin, Germany: Springer-Verlag, 2012, pp. 9-48.
32. A. Krogh and J. A. Hertz, "A simple weight decay can improve generalization," in Advances in Neural Information Processing Systems, vol. 4. San Mateo, CA, USA: Morgan Kaufmann, 1995, pp. 950-957.
33. J. Nocedal and S. J. Wright, Numerical Optimization, vol. 2. New York, NY, USA: Springer-Verlag, 1999.
34. J. M. Zurada, Introduction to Artificial Neural Systems. St. Paul, MN, USA: West, 1992.
35. R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu, "A limited memory algorithm for bound constrained optimization," SIAM J. Sci. Comput., vol. 16, no. 5, pp. 1190-1208, 1995.
36. M. Schmidt. (2008). MATLAB Software. [Online]. Available: http://www.di.ens.fr/~mschmidt/Software/minConf.html
37. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
38. F. S. Samaria and A. C. Harter, "Parameterisation of a stochastic model for human face identification," in Proc. 2nd IEEE Workshop Appl. Comput. Vis., Dec. 1994, pp. 138-142.
39. Y. LeCun, F. J. Huang, and L. Bottou, "Learning methods for generic object recognition with invariance to pose and lighting," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2. Jun./Jul. 2004, pp. II-97-II-104.
40. P. O. Hoyer, "Non-negative matrix factorization with sparseness constraints," J. Mach. Learn. Res., vol. 5, pp. 1457-1469, Dec. 2004.
41. L. van der Maaten and G. Hinton, "Visualizing data using t-SNE," J. Mach. Learn. Res., vol. 9, no. 11, pp. 2579-2605, 2008.
42. K. S. Jones, "A statistical interpretation of term specificity and its application in retrieval," J. Documentation, vol. 28, no. 1, pp. 11-21, 1972.
43. P.-N. Tan, M. Steinbach, and V. Kumar, Introduction to Data Mining, 1st ed. Reading, MA, USA: Addison-Wesley, May 2005.