LSTM: A Search Space Odyssey

IEEE Transactions on Neural Networks and Learning Systems

Volumn 28, Issue 10, 2017, Pages 2222-2232

  Greff, Klaus ^a,b   Srivastava, Rupesh K ^a,b   Koutnik, Jan ^a,b   Steunebrink, Bas R ^a,b   Schmidhuber, Jurgen ^a,b  

^a DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

^b UNIVERSITY OF LUGANO   (Switzerland)

Author keywords

Functional ANalysis Of VAriance (fANOVA); long short term memory (LSTM); random search; recurrent neural networks; sequence learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; CHARACTER RECOGNITION; COMPUTER MUSIC; LEARNING SYSTEMS; NETWORK ARCHITECTURE; SPEECH RECOGNITION;

ACTIVATION FUNCTIONS; COMPUTATIONAL COMPONENTS; CRITICAL COMPONENT; HANDWRITING RECOGNITION; LARGE-SCALE ANALYSIS; LONG SHORT TERM MEMORY; MACHINE LEARNING PROBLEM; POLYPHONIC MUSIC;

RECURRENT NEURAL NETWORKS;

EID: 84979010616     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2016.2582924     Document Type: Article

References (46)

1. S. Hochreiter, "Untersuchungen zu dynamischen neuronalen Netzen, " Dept. Fakultät für Informatik, M.S. thesis, Tech. Univ. Munich, Munich, Germany, 1991.
2. S. Hochreiter, Y. Bengio, P. Frasconi, J. Schmidhuber, "Gradient flow in recurrent nets: The difficulty of learning long-term dependencies, " in A Field Guide to Dynamical Recurrent Networks, S. C. Kremer and J. F. Kolen, Eds. Piscataway, NJ, USA: IEEE Press, 2001.
3. A. Graves, M. Liwicki, S. Fernández, R. Bertolami, H. Bunke, J. Schmidhuber, "A novel connectionist system for unconstrained handwriting recognition, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 5, pp. 855-868, May 2009.
4. V. Pham, T. Bluche, C. Kermorvant, J. Louradour. (Nov. 2013). "Dropout improves recurrent neural networks for handwriting recognition." [Online]. Available: Http://arxiv.org/abs/1312.4569
5. P. Doetsch, M. Kozielski, H. Ney, "Fast and robust training of recurrent neural networks for offline handwriting recognition, " in Proc. 14th Int. Conf. Frontiers Handwriting Recognit., 2014, pp. 279-284. [Online]. Available: Http://people.sabanciuniv.edu/berrin/cs581/Papers/icfhr2014/data/4334a279.pdf
6. A. Graves. (Aug. 2013). "Generating sequences with recurrent neural networks." [Online]. Available: Http://arxiv.org/abs/1308.0850
7. W. Zaremba, I. Sutskever, O. Vinyals. (Sep. 2014). "Recurrent neural network regularization." [Online]. Available: Http://arxiv.org/abs/1409.2329
8. M.-T. Luong, I. Sutskever, Q. V. Le, O. Vinyals, W. Zaremba. (2014). "Addressing the rare word problem in neural machine translation." [Online]. Available: Http://arxiv.org/abs/1410.8206
9. H. Sak, A. Senior, F. Beaufays, "Long short-term memory recurrent neural network architectures for large scale acoustic modeling, " in Proc. Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), 2014, pp. 338-342. [Online]. Available: Http://193.6.4.39/~czap/letoltes/IS14/IS2014/PDF/AUTHOR/IS141304.PDF
10. Y. Fan, Y. Qian, F. Xie, F. K. Soong, "TTS synthesis with bidirectional LSTM based recurrent neural networks, " in Proc. Interspeech, 2014, pp. 1964-1968.
11. S. K. Sønderby and O. Winther. (Dec. 2014). "Protein secondary structure prediction with long short term memory networks." [Online]. Available: Http://arxiv.org/abs/1412.7828
12. E. Marchi, G. Ferroni, F. Eyben, L. Gabrielli, S. Squartini, B. Schuller, "Multi-resolution linear prediction based features for audio onset detection with bidirectional LSTM neural networks, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2014, pp. 2164-2168.
13. J. Donahue, et al. (Nov. 2014). "Long-term recurrent convolutional networks for visual recognition and description." [Online]. Available: Http://arxiv.org/abs/1411.4389
14. S. Hochreiter and J. Schmidhuber, "Long short-term memory, " Dept. Fakultät für Informatik, Tech. Univ. Munich, Munich, Germany, Tech. Rep. FKI-207-95, Aug. 1995. [Online]. Available: Http://citeseerx.ist.psu.edu/viewdoc/summarydoi=10.1.1.51.3117
15. S. Hochreiter and J. Schmidhuber, "Long short-term memory, " Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997. [Online]. Available: Http://www.bioinf.jku.at/publications/older/2604.pdf
16. R. L. Anderson, "Recent advances in finding best operating conditions, " J. Amer. Statist. Assoc., vol. 48, no. 264, pp. 789-798, Dec. 1953. [Online]. Available: Http://www.jstor.org/stable/2281072
17. F. J. Solis and R. J.-B. Wets, "Minimization by random search techniques, " Math. Oper. Res., vol. 6, no. 1, pp. 19-30, Feb. 1981. [Online]. Available: Http://pubsonline.informs.org/doi/abs/10.1287/moor.6.1.19
18. J. Bergstra and Y. Bengio, "Random search for hyper-parameter optimization, " J. Mach. Learn. Res., vol. 13, no. 1, pp. 281-305, 2012. [Online]. Available: Http://dl.acm.org/citation.cfmid=2188395
19. F. Hutter, H. Hoos, K. Leyton-Brown, "An efficient approach for assessing hyperparameter importance, " in Proc. 31st Int. Conf. Mach. Learn., 2014, pp. 754-762. [Online]. Available: Http://jmlr.org/proceedings/papers/v32/hutter14.html
20. A. Graves and J. Schmidhuber, "Framewise phoneme classification with bidirectional LSTM and other neural network architectures, " Neural Netw., vol. 18, nos. 5-6, pp. 602-610, Jul./Aug. 2005. [Online]. Available: Http://www.sciencedirect.com/science/article/pii/S0893608005001206
21. F. A. Gers, J. Schmidhuber, F. Cummins, "Learning to forget: Continual prediction with LSTM, " in Proc. 9th Int. Conf. Artif. Neural Netw. (ICANN), vol. 2. Sep. 1999, pp. 850-855.
22. F. A. Gers and J. Schmidhuber, "Recurrent nets that time and count, " in Proc. IEEE-INNS-ENNS Int. Joint Conf. Neural Netw. (IJCNN), vol. 3. Jul. 2000, pp. 189-194.
23. A. J. Robinson and F. Fallside, The Utility Driven Dynamic Error Propagation Network. Cambridge, U.K.: Univ. Cambridge Department of Engineering, 1987.
24. R. J. Williams, "Complexity of exact gradient computation algorithms for recurrent neural networks, " College Comput. Sci., Northeastern Univ., Boston, MA, USA, Tech. Rep. NU-CCS-89-27, 1989.
25. P. J. Werbos, "Generalization of backpropagation with application to a recurrent gas market model, " Neural Netw., vol. 1, no. 4, pp. 339-356, 1988.
26. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, N. L. Dahlgren, "DARPA TIMIT acoustic-phonetic continous speech corpus CD-ROM. NIST speech disc 1-1.1, " Nat. Inst. Standards Technol., Gaithersburg, MD, USA, Tech. Rep. LDC93S1, 1993.
27. F. A. Gers, J. A. Pérez-Ortiz, D. Eck, J. Schmidhuber, "DEKF-LSTM, " in Proc. 10th Eur. Symp. Artif. Neural Netw. (ESANN), 2002, pp. 369-376.
28. J. Schmidhuber, D. Wierstra, M. Gagliolo, F. Gomez, "Training recurrent networks by Evolino, " Neural Comput., vol. 19, no. 3, pp. 757-779, Mar. 2007.
29. J. Bayer, D. Wierstra, J. Togelius, J. Schmidhuber, "Evolving memory cell structures for sequence learning, " in Artificial Neural Networks-ICANN. Berlin, Germany: Springer, 2009, pp. 755-764. [Online]. Available: Http://link.springer.com/chapter/10.1007/978-3-642-04277-5-76
30. R. Jozefowicz, W. Zaremba, I. Sutskever, "An empirical exploration of recurrent network architectures, " in Proc. 32nd Int. Conf. Mach. Learn. (ICML), 2015, pp. 2342-2350.
31. S. Otte, M. Liwicki, A. Zell, "Dynamic cortex memory: Enhancing recurrent neural networks for gradient-based sequence learning, " in Artificial Neural Networks and Machine Learning (Lecture Notes in Computer Science), vol. 8681. Cham, Germany: Springer, Sep. 2014, pp. 1-8. [Online]. Available: Http://link.springer.com/chapter/10.1007/978-3-319-11179-7-1
32. K. Cho, et al. (2014). "Learning phrase representations using RNN encoder-decoder for statistical machine translation." [Online]. Available: Http://arxiv.org/abs/1406.1078
33. J. Chung, C. Gulcehre, K. Cho, Y. Bengio. (Dec. 2014). "Empirical evaluation of gated recurrent neural networks on sequence modeling." [Online]. Available: Http://arxiv.org/abs/1412.3555
34. D. Crystal, Dictionary of Linguistics and Phonetics, vol. 30. New York, NY, USA: Wiley, 2011.
35. P. Mermelstein, "Distance measures for speech recognition: Psychological and instrumental, " in Pattern Recognition and Artificial Intelligence, C. H. Chen, Ed. New York, NY, USA: Academic, 1976, pp. 374-388.
36. A. Graves, "Supervised sequence labelling with recurrent neural networks, " Ph.D. dissertation, Dept. Fakultät für Informatik, Tech. Univ. Munich, Munich, Germany, 2008.
37. A. K. Halberstadt, "Heterogeneous acoustic measurements and multiple classifiers for speech recognition, " Ph.D. dissertation, Dept. Elect. Eng. Comput. Sci., Massachusetts Inst. Technol., Cambridge, MA, USA, 1998.
38. M. Liwicki and H. Bunke, "IAM-OnDB-An on-line English sentence database acquired from handwritten text on a whiteboard, " in Proc. 8th Int. Conf. Document Anal. Recognit., 2005, pp. 956-961.
39. A. Graves, S. Fernández, F. Gomez, J. Schmidhuber, "Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks, " in Proc. 23rd Int. Conf. Mach. Learn., 2006, pp. 369-376. [Online]. Available: Http://dl.acm. org/citation.cfmid=1143891
40. M. Allan and C. K. I. Williams, "Harmonising chorales by probabilistic inference, " in Advances in Neural Information Processing Systems, vol. 17. Cambridge, MA, USA: MIT Press, 2005, pp. 25-32.
41. N. Boulanger-Lewandowski, Y. Bengio, P. Vincent, "Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription, " in Proc. 29th Int. Conf. Mach. Learn., 2012, pp. 1159-1166. [Online]. Available: Http://icml. cc/discuss/2012/590.html
42. I. Sutskever, J. Martens, G. Dahl, G. Hinton, "On the importance of initialization and momentum in deep learning, " J. Mach. Learn. Res., vol. 23, no. 3, pp. 1139-1147, 2013. [Online]. Available: Http://jmlr.org/proceedings/papers/v28/sutskever13.html
43. J. Snoek, H. Larochelle, R. P. Adams, "Practical Bayesian optimization of machine learning algorithms, " in Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, K. Q. Weinberger, Eds. Red Hook, NY, USA: Curran Associates, Inc., 2012, pp. 2951-2959.
44. F. Hutter, H. H. Hoos, K. Leyton-Brown, "Sequential model-based optimization for general algorithm configuration, " in Proc. LION 5, 2011, pp. 507-523.
45. A. Graves, M. Liwicki, H. Bunke, J. Schmidhuber, S. Fernández, "Unconstrained on-line handwriting recognition with recurrent neural networks, " in Proc. Adv. Neural Inf. Process. Syst., 2008, pp. 577-584.
46. G. Hooker, "Generalized functional ANOVA diagnostics for highdimensional functions of dependent variables, " J. Comput. Graph. Statist., vol. 16, no. 3, pp. 709-732, Sep. 2007. [Online]. Available: Http://www.tandfonline.com/doi/abs/10.1198/106186007X237892