Auto-sizing neural networks: With applications to n-gram language models

Conference Proceedings - EMNLP 2015: Conference on Empirical Methods in Natural Language Processing

Volumn , Issue , 2015, Pages 908-916

  Murray, Kenton ^a   Chiang, David ^a  

^a University of Notre Dame   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; MODELING LANGUAGES;

HIDDEN LAYERS; IMPROVE PERFORMANCE; LANGUAGE MODEL; MACHINE TRANSLATIONS; N-GRAM LANGUAGE MODELS; NATURAL LANGUAGES; NEURAL MODELS; OPTIMAL SETTING;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 84959882423     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.18653/v1/d15-1107     Document Type: Conference Paper

References (19)

1. Francis Bach, Rodolphe Jenatton, Mien Mairal, and Guillaume Obozinski. 2012. Optimization with sparsity-inducing penalties. Foundations and Trends in Machine Learning, 4(1):1-106.
2. Paul Baltescu, Phil Blunsom, and Hieu Hoang. 2014. OxLM: A neural language modelling framework for machine translation. Prague Bulletin of Mathematical Linguistics, 102(1):81-92.
3. Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Machine Learning Research, 3:1137-1155.
4. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proc. ACL, pages 1370-1380.
5. John Duchi and Yoram Singer. 2009. Efficient online and batch learning using forward backward splitting. J. Machine Learning Research, 10:2899-2934.
6. 1-ball for learning in high dimensions. In Proc. ICML, pages 272-279.
7. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondfej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. ACL, Interactive Poster and Demonstration Sessions, pages 177-180.
8. Yann LeCun, John S. Denker, Sara A. Sofia, Richard E. Howard, and Lawrence D. Jackel. 1989. Optimal brain damage. In Proc. NIPS, volume 2, pages 598-605.
9. Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky. 2011. RNNLM-recurrent neural network language modeling toolkit. In Proc. ASRU, pages 196-201.
10. Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In Proc. ICML, pages 1751-1758.
11. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve Restricted Boltzmann Machines. In Proc. ICML, pages 807-814.
12. Steven J. Nowland and Geoffrey E. Hinton. 1992. Simplifying neural networks by soft weight-sharing. Neural Computation, 4:473-493.
13. Neal Parikh and Stephen Boyd. 2014. Proximal algorithms. Foundations and Trends in Optimization, 1(3):127-239.
14. 1∞ regularization. In Proc. ICML, pages 857-864.
15. Jurgen Schmidhuber and Stefan Heil. 1996. Sequential neural text compression. IEEE Transactions on Neural Networks, 7:142-146.
16. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. J. Machine Learning Research, 15(1):1929-1958.
17. Martin Sundermeyer, Hermann Ney, and Ralf Schliiter. 2015. From feedforward to recurrent LSTM neural networks for language modeling. Trans. Audio, Speech, and Language, 23(3):517-529.
18. Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. In Proc. EMNLP, pages 1387-1392.
19. Wei Xu and Alexander I. Rudnicky. 2000. Can artificial neural networks learn language models? In Proc. International Conference on Statistical Language Processing, pages M1-13.