Measuring the usefulness of hidden units in Boltzmann machines with mutual information

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 8226 LNCS, Issue PART 1, 2013, Pages 482-489

  Berglund, Mathias ^a   Raiko, Tapani ^a   Cho, KyungHyun ^a  

^a AALTO UNIVERSITY   (Finland)

Author keywords

Deep boltzmann machine; Deep learning; Mutual information; Pruning; Restricted boltzmann machine; Structural learning

Indexed keywords

DEEP BOLTZMANN MACHINES; DEEP LEARNING; MUTUAL INFORMATIONS; PRUNING; RESTRICTED BOLTZMANN MACHINE; STRUCTURAL LEARNING;

ARTIFICIAL INTELLIGENCE; COMPUTER SCIENCE; COMPUTERS;

DATA PROCESSING;

EID: 84893348709     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-42054-2_60     Document Type: Conference Paper

References (18)

1. Cho, K., Raiko, T., Ilin, A.: Enhanced gradient for training restricted Boltzmann machines. Neural Computation 25(3), 805-831 (2013)
2. Cho, K., Raiko, T., Ilin, A., Karhunen, J.: A two-stage pretraining algorithm for deep Boltzmann machines. In: Proceedings of the 23rd International Conference on Artificial Neural Networks (to appear, September 2013)
3. Zhou, G., Sohn, K., Lee, H.: Online incremental feature learning with denoising autoencoders. In: Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2012), pp. 1453-1461 (2012)
4. Adams, R.P.,Wallach, H.M., Ghahramani, Z.: Learning the structure of deep sparse graphical models. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010), pp. 1-8 (2010)
5. Engelbrecht, A.P.: A new pruning heuristic based on variance analysis of sensitivity information. Transactions on Neural Networks 12(6), 1386-1399 (2001)
6. Reed, R.: Pruning algorithms-a survey. Transactions on Neural Networks 4(5), 740-747 (1993)
7. Smolensky, P.: Information processing in dynamical systems: foundations of harmony theory. In: Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Foundations, vol. 1, pp. 194-281. MIT Press, Cambridge (1986)
8. Salakhutdinov, R., Hinton, G.: Deep Boltzmann machines. In: Proceedings of the Twelfth Internation Conference on Artificial Intelligence and Statistics (AISTATS 2009), pp. 448-455 (2009)
9. Hinton, G., Salakhutdinov, R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504-507 (2006)
10. Hinton, G., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Computation 18(7), 1527-1554 (2006)
11. Salakhutdinov, R., Mnih, A., Hinton, G.: Restricted Boltzmann machines for collaborative filtering. In: Proceedings of the 24th International Conference on Machine Learning (ICML 2007), pp. 791-798. ACM, New York (2007)
12. Srivastava, N., Salakhutdinov, R.: Multimodal learning with deep boltzmann machines. In: Bartlett, P., Pereira, F.C.N., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 2231-2239 (2012)
13. Lee, H., Grosse, R., Ranganath, R., Ng, A.: Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: Proceedings of the 26th Annual International Conference on Machine Learning (ICML 2009), pp. 609-616. ACM (2009)
14. Salakhutdinov, R.: Learning and evaluating Boltzmann machines. Technical Report UTML TR 2008-002, Department of Computer Science, University of Toronto (June 2008)
15. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010), pp. 249-256 (May 2010)
16. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 2278-2324 (1998)
17. Ranzato, M., Boureau, Y.L., LeCun, Y.: Sparse feature learning for deep belief networks. In: Platt, J., Koller, D., Singer, Y., Roweis, S. (eds.) Advances in Neural Information Processing Systems 20, pp. 1185-1192. MIT Press, Cambridge (2008)
18. Peltonen, J., Kaski, S.: Discriminative components of data. IEEE Transactions on Neural Networks 16(1), 68-83 (2005)