Multimodal learning with Deep Boltzmann Machines

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 2949-2980

  Srivastava, Nitish ^a   Salakhutdinov, Ruslan ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Boltzmann machines; Deep learning; Multimodal learning; Neural networks; Unsupervised learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; NEURAL NETWORKS; UNSUPERVISED LEARNING;

BOLTZMANN MACHINES; CLASSIFICATION RESULTS; CONDITIONAL DISTRIBUTION; DEEP BOLTZMANN MACHINES; DEEP LEARNING; MULTI-MODAL LEARNING; MULTIPLE KERNEL LEARNING; STATISTICAL PROPERTIES;

CLASSIFICATION (OF INFORMATION);

EID: 84916911784     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (41)

1. M. Bastan, H. Cam, U. Gudukbay, and O. Ulusoy. Bilvideo-7: An MPEG-7- compatible video indexing and retrieval system. IEEE Multimedia, 17:62-73, 2010. ISSN 1070-986X.
2. A. Bosch, A. Zisserman, and X. Munoz. Image classification using random forests and ferns. IEEE 11th International Conference on Computer Vision (2007), 23:1-8, 2007.
3. S. Cox, R. Harvey, Y. Lan, J. Newman, and B. Theobald. The challenge of multispeaker lip-reading. In International Conference on Auditory-Visual Speech Processing, pages 179-184, September 2008.
4. N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In IEEE Conference on Computer Vision and Pattern Recognition, pages 886-893, 2005.
5. W. M. Fisher, G. R. Doddington, and K. M. Goudie-Marshall. The DARPA speech recognition research database: Specifications and status. In Proceedings of DARPA Workshop on Speech Recognition, pages 93-99, 1986.
6. Y. Freund and D. Haussler. Unsupervised learning of distributions on binary vectors using two layer networks. Technical report, University of California at Santa Cruz, Santa Cruz, CA, USA, 1994.
7. M. Guillaumin, J. Verbeek, and C. Schmid. Multimodal semi-supervised learning for image classification. In IEEE Conference on Computer Vision and Pattern Recognition, pages 902-909, 2010.
8. M. Gurban and J.-P. Thiran. Information theoretic feature extraction for audio-visual speech recognition. IEEE Transactions on Signal Processing, 57(12):4765-4776, 2009.
9. G. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, 2006.
10. G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1711-1800, 2002.
11. G. E. Hinton, S. Osindero, and Y. W. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18(7):1527-1554, 2006.
12. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207.0580, 2012.
13. M. J. Huiskes and M. S. Lew. The MIR Flickr retrieval evaluation. In ACM International Conference on Multimedia Information Retrieval, 2008.
14. M. J. Huiskes, B. Thomee, and M. S. Lew. New trends and ideas in visual concept detection: the MIR Flickr retrieval evaluation initiative. In 11th ACM International Conference on Multimedia Information Retrieval, pages 527-536, 2010.
15. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th International Conference on Machine Learning, pages 609-616, 2009.
16. P. Lucey and S. Sridharan. Patch-based representation of visual speech. In Proceedings of the HCSNet workshop on Use of vision in human-computer interaction - Volume 56, VisHCI '06, pages 79-85. Australian Computer Society, Inc., 2006.
17. B. Manjunath, J.-R. Ohm, V. Vasudevan, and A. Yamada. Color and texture descriptors. Circuits and Systems for Video Technology, IEEE Transactions on, 11(6):703-715, 2001.
18. I. Matthews, T. F. Cootes, J. A. Bangham, S. Cox, and R. Harvey. Extraction of visual features for lipreading. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(2):198-213, Feb. 2002.
19. A. Mohamed, G. Dahl, and G. Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, 2011.
20. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodal deep learning. In Proceedings of the 28th International Conference on Machine Learning, 2011.
21. A. Oliva and A. Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 42:145-175, 2001.
22. B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607-609, 1996.
23. G. Papandreou, A. Katsamanis, V. Pitsikalis, and P. Maragos. Multimodal fusion and learning with uncertain features applied to audiovisual speech recognition. In IEEE 9th Workshop on Multimedia Signal Processing, pages 264-267, 2007.
24. G. Papandreou, A. Katsamanis, V. Pitsikalis, and P. Maragos. Adaptive multimodal fusion by uncertainty compensation with application to audiovisual speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 17(3):423-435, 2009.
25. E. K. Patterson, S. Gurbuz, Z. Tufekci, and J. N. Gowdy. Cuave: A new audio-visual database for multimodal human-computer interface research. In International Conference on Audio Speech and Signal Processing, pages 2017-2020, 2002.
26. H. Robbins and S. Monro. A stochastic approximation method. Ann. Math. Stat., 22: 400-407, 1951.
27. R. Salakhutdinov and G. E. Hinton. Replicated softmax: an undirected topic model. In Advances in Neural Information Processing Systems 22, pages 1607-1614, 2009a.
28. R. R. Salakhutdinov and G. E. Hinton. Deep Boltzmann machines. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 12, 2009b.
29. P. Smolensky. Information processing in dynamical systems: Foundations of harmony theory. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1, pages 194-281. MIT Press, Cambridge, MA, USA, 1986.
30. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014.
31. G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler. Convolutional learning of spatio-temporal features. In Eurpean Conference on Computer Vision, 2010.
32. T. Tieleman. Training Restricted Boltzmann Machines using Approximations to the Likelihood Gradient. In Proceedings of the 25th International Conference on Machine Learning, pages 1064-1071, 2008.
33. A. Vedaldi and B. Fulkerson. VLFeat: An open and portable library of computer vision algorithms, 2008.
34. J. Verbeek, M. Guillaumin, T. Mensink, and C. Schmid. Image annotation with TagProp on the MIRFLICKR set. In 11th ACM International Conference on Multimedia Information Retrieval, pages 537-546, 2010.
35. P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, pages 1096-1103, 2008.
36. M. Welling, M. Rosen-Zvi, and G. E. Hinton. Exponential family harmoniums with an application to information retrieval. In Advances in Neural Information Processing Systems 18, pages 1481-1488, 2005.
37. E. P. Xing, R. Yan, and A. G. Hauptmann. Mining associated text and images with dualwing harmoniums. In Uncertainty in Artificial Intelligence (UAI), pages 633-641. AUAI Press, 2005.
38. L. Younes. Parameter inference for imperfectly observed Gibbsian fields. Probability Theory Rel. Fields, 82:625-645, 1989.
39. L. Younes. On the convergence of Markovian stochastic algorithms with rapidly decreasing ergodicity rates. In Stochastics and Stochastics Models, pages 177-228, 1998.
40. A. L. Yuille. The convergence of contrastive divergences. In Advances in Neural Information Processing Systems 17, 2004.
41. G. Zhao, M. Barnard, and M. Pietikainen. Lipreading with local spatiotemporal descriptors. IEEE Transactions on Multimedia, 11(7):1254-1265, 2009.