A topological insight into restricted Boltzmann machines

Machine Learning

Volumn 104, Issue 2-3, 2016, Pages 243-270

  Mocanu, Decebal Constantin ^a   Mocanu, Elena ^a   Nguyen, Phuong H ^a   Gibescu, Madeleine ^a   Liotta, Antonio ^a  

^a EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Complex networks; Deep learning; Scale free networks; Small world networks; Sparse restricted Boltzmann machines

Indexed keywords

COMPLEX NETWORKS; NEURAL NETWORKS; SCALES (WEIGHING INSTRUMENTS); SMALL-WORLD NETWORKS;

BASIC BUILDING BLOCK; DEEP LEARNING; FEATURES EXTRACTION; NUMBER OF HIDDEN NEURONS; ORDERS OF MAGNITUDE; RESTRICTED BOLTZMANN MACHINE; SCALE-FREE TOPOLOGIES; SMALL WORLD TOPOLOGY;

TOPOLOGY;

EID: 84978872829     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-016-5570-z     Document Type: Article

References (47)

1. Ackley, H., Hinton, E., & Sejnowski, J. (1985). A learning algorithm for Boltzmann machines. Cognitive Science, 9, 147–169.
2. Ammar, H. B., Mocanu, D. C., Taylor, M., Driessens, K., Tuyls, K., & Weiss, G. (2013). Automatically mapped transfer between reinforcement learning tasks via three-way restricted boltzmann machines. In H. Blockeel, K. Kersting, S. Nijssen, & F. Elezn (Eds.), Machine learning and knowledge discovery in databases (Vol. 8189, pp. 449–464). Lecture Notes in Computer Science Berlin: Springer. doi:10.1007/978-3-642-40991-2_29.
3. Ba, J., & Caruana, R. (2014). Do deep nets really need to be deep? Advances in Neural Information Processing Systems, 27, 2654–2662.
4. Barabasi, A. L., & Albert, R. (1999). Emergence of scaling in random networks. Science, 286(5439), 509–512. doi:10.1126/science.286.5439.509.
5. Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1), 1–127. doi:10.1561/2200000006.
6. Brgge, K., Fischer, A., & Igel, C. (2013). The flip-the-state transition operator for restricted Boltzmann machines. Machine Learning, 93(1), 53–69. doi:10.1007/s10994-013-5390-3.
7. Carreira-Perpinan, M. A., & Hinton, G. E. (2005). On contrastive divergence learning. In 10th international workshop on artificial intelligence and statistics (AISTATS).
8. Clauset, A., Shalizi, C. R., & Newman, M. E. J. (2009). Power-law distributions in empirical data. SIAM Review, 51(4), 661–703. doi:10.1137/070710111.
9. Del Genio, C. I., Gross, T., & Bassler, K. E. (2011). All scale-free networks are sparse. Physical Review Letter, 107, 178701. doi:10.1103/PhysRevLett.107.178701.
10. Desjardins, G., Courville, A., Bengio, Y., Vincent, P., & Delalleau, O. (2010). Tempered Markov Chain Monte Carlo for training of restricted Boltzmann machines. In Y. W. Teh, & M. Titterington (Eds.), Proceedings of the thirteenth international conference on artificial intelligence and statistics (pp. 145–152), May 13–15, 2010. Sardinia: Chia Laguna Resort.
11. Dieleman, S., & Schrauwen, B. (2012). Accelerating sparse restricted Boltzmann machine training using non-gaussianity measures. In Y. Bengiom, J. Bergstra, & Q. Le (Eds.) Proceedings of deep learning and unsupervised feature learning (p. 9).
12. Gehler, P. V., Holub, A. D., & Welling, M. (2006). The rate adapting poisson model for information retrieval and object recognition. In Proceedings of the 23rd international conference on machine learning, ACM, ICML ’06 (pp. 337–344). doi:10.1145/1143844.1143887.
13. Germain, M., Gregor, K., Murray, I., & Larochelle, H. (2015). MADE: Masked autoencoder for distribution estimation. In Proceedings of the 32nd international conference on machine learning, JMLR.org, JMLR Proceedings (Vol. 37, pp. 881–889).
14. Hagberg, A. A., Schult, D. A., & Swart, P. J. (2008). Exploring network structure, dynamics, and function using NetworkX. In Proceedings of the 7th python in science conference (SciPy2008) (pp. 11–15), Pasadena: CA USA.
15. Hakimi, S. L. (1962). On realizability of a set of integers as degrees of the vertices of a linear graph. I. Journal of Society for Industrial and Applied Mathematics, 10, 496–506.
16. Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 28, pp. 1135–1143). Red Hook: Curran Associates Inc.
17. Hinton, G., (2012). A practical guide to training restricted Boltzmann machines. In Neural networks: Tricks of the trade (Vol. 7700, pp. 599–619). Lecture Notes in Computer Science. Berlin: Springer. doi:10.1007/978-3-642-35289-8_32.
18. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1771–1800. doi:10.1162/089976602760128018.
19. Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. doi:10.1126/science.1127647.
20. Jones, N. (2014). Computer science: The learning machines. Nature, 505(7482), 146–148.
21. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 1097–1105.
22. Larochelle, H., & Bengio, Y. (2008). Classification using discriminative restricted Boltzmann machines. In Proceedings of the 25th international conference on machine learning, ACM, ICML ’08 (pp. 536–543). doi:10.1145/1390156.1390224
23. Larochelle, H., & Murray, I. (2011). The neural autoregressive distribution estimator. In AISTATS, JMLR.org, JMLR Proceedings (Vol. 15, pp 29–37).
24. Latapy, M., Magnien, C., & Vecchio, N. D. (2008). Basic notions for the analysis of large two-mode networks. Social Networks, 30(1), 31–48. doi:10.1016/j.socnet.2007.04.006.
25. Lee, H., Ekanadham, C., & Ng, A. Y. (2008). Sparse deep belief net model for visual area v2. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in Neural Information Processing System (Vol. 20, pp. 873–880). Red Hook: Curran Associates Inc.
26. Lee, H., Pham, P., Largman, Y., & Ng, A. Y. (2009). Unsupervised feature learning for audio classification using convolutional deep belief networks. Advances in Neural Information Processing Systems, 22, 1096–1104.
27. Luo, H., Shen, R., Niu, C., & Ullrich, C. (2011). Sparse group restricted boltzmann machines. In W. Burgard & D. Roth (Eds.), AAAI. Menlo Park: AAAI Press.
28. Marlin, B. M., Swersky, K., Chen, B., & de Freitas, N. (2010). Inductive principles for restricted boltzmann machine learning. In AISTATS, JMLR.org, JMLR Proceedings (Vol. 9, pp. 509–516)
29. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. doi:10.1038/nature14236.
30. Mocanu, D. C., Ammar, H. B., Lowet, D., Driessens, K., Liotta, A., Weiss, G., et al. (2015). Factored four way conditional restricted boltzmann machines for activity recognition. Pattern Recognition Letters, 66, 100–108. doi:10.1016/j.patrec.2015.01.013. Pattern Recognition in Human Computer Interaction.
31. Osogami, T., & Otsuka, M. (2014). Restricted Boltzmann machines modeling human choice. Advances in Neural Information Processing Systems, 27, 73–81.
32. Pessoa, L. (2014). Understanding brain networks and brain organization. Physics of Life Reviews, 11(3), 400–435. doi:10.1016/j.plrev.2014.03.005.
33. Ranzato, M. A., lan Boureau, Y., & Cun, Y. L. (2008). Sparse feature learning for deep belief networks. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in neural information processing systems (Vol. 20, pp. 1185–1192). Red Hook: Curran Associates Inc.
34. Salakhutdinov, R., & Murray, I. (2008). On the quantitative analysis of deep belief networks. In Proceedings of the international conference on machine learning (pp. 872–879).
35. Salakhutdinov, R., Mnih, A., & Hinton, G. (2007). Restricted boltzmann machines for collaborative filtering. In Proceedings of the 24th international conference on machine learning, ACM, ICML ’07 (pp 791–798). doi:10.1145/1273496.1273596
36. Smolensky, P. (1987). Information processing in dynamical systems: Foundations of harmony theory. In D. E. Rumelhart, J. L. McClelland, et al. (Eds.), Parallel distributed processing (Vol. 1, pp. 194–281). Cambridge: MIT Press.
37. Srivastava, N., & Salakhutdinov, R. R. (2012). Multimodal learning with deep Boltzmann machines. In F. Pereira, C. Burges, L. Bottou, & K. Weinberger (Eds.), Advances in neural information processing systems (Vol. 25, pp. 2222–2230). Red Hook: Curran Associates Inc.
38. Strogatz, S. H. (2001). Exploring complex networks. Nature, 410(6825), 268–276.
39. Swersky, K., Tarlow, D., Sutskever, I., Salakhutdinov, R., Zemel, R. S., & Adams, R. P. (2012). Cardinality restricted boltzmann machines. In NIPS (pp. 3302–3310).
40. Tieleman, T. (2008). Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on machine learning, ICML ’08 (pp. 1064–1071), ACM, New York, NY, USA. doi:10.1145/1390156.1390290
41. Tieleman, T., & Hinton, G. (2009). Using fast weights to improve persistent contrastive divergence. In Proceedings of the 26th annual international conference on machine learning, ICML ’09 (pp. 1033–1040), ACM, New York, NY, USA. doi:10.1145/1553374.1553506
42. van der Hofstad, R. (2016). Random graphs and complex networks (Vol. i). http://www.win.tue.nl/~rhofstad/NotesRGCN
43. Wan, C., Jin, X., Ding, G., & Shen, D. (2015) Gaussian cardinality restricted boltzmann machines. In Twenty-ninth AAAI conference on artificial intelligence.
44. Watts, D. J., & Strogatz, S. H. (1998). Collective dynamics of ’small-world’ networks. Nature, 393, 440–442.
45. Welling, M., Rosen-zvi, M., & Hinton, G. E. (2005). Exponential family harmoniums with an application to information retrieval. In L. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems (Vol. 17, pp. 1481–1488). Cambridge: MIT Press.
46. Yosinski, J., & Yosinski and Lipson, H. (2012). Visually debugging restricted Boltzmann machine training with a 3d example. In Representation learning workshop, 29th international conference on machine learning.
47. Zhou, F., Claire, Q., & King, R. (2014). Predicting the geographical origin of music. In 2014 IEEE international conference on data mining (ICDM) (pp. 1115–1120). doi:10.1109/ICDM.2014.73