A clockwork RNN

31st International Conference on Machine Learning, ICML 2014

Volumn 5, Issue , 2014, Pages 3881-3889

  Koutník, Jan ^a   Greff, Klaus ^a   Gomez, Faustino ^a   Schmidhuber, Jürgen ^a  

^a DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CHARACTER RECOGNITION; CLOCKS; LEARNING SYSTEMS; RECURRENT NEURAL NETWORKS;

HIDDEN LAYERS; LONG TERM MEMORY; NETWORK EVALUATION; ONLINE HANDWRITING RECOGNITION; RECURRENT NEURAL NETWORK (RNNS); SEQUENCE PREDICTION; SHORT TERM MEMORY; TEMPORAL GRANULARITY;

COMPLEX NETWORKS;

EID: 84919782249     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (39)

1. Elman, J. L. Finding structure in time. CRL Technical Report 8801, Center for Research in Language, University of California, San Diego, 1988.
2. Fernandez, S., Graves, A., and Schmidhuber, J. Sequence labelling in structured domains with hierarchical recurrent neural networks. In Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI), 2007.
3. Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., and Dahlgren, N. L. DARPA TIMIT acoustic phonetic continuous speech corpus CD-ROM, 1993.
4. Graves, A. Supervised sequence labelling with recurrent neural networks. PhD thesis, Technical University Munich, 2008.
5. Graves, A. and Schmidhuber, J. Offline handwriting recognition with multidimensional recurrent neural networks. In Advances in Neural Information Processing Systems 21, pp. 545-552. MIT Press, Cambridge, MA, 2009.
6. Graves, A., Beringer, N., and Schmidhuber, J. Rapid retraining on speech data with LSTM recurrent networks. Technical Report IDSIA-09-05, IDSIA, 2005. URL http://www.idsia.ch/idsiareport/IDSIA-09-05.pdf.
7. Graves, A., Fernandez, S., Gomez, F. J., and Schmidhuber, J. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural nets. In ICML'06: Proceedings of the International Conference on Machine Learning. 2006.
8. Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Bunke, H., and Schmidhuber, J. A novel connectionist system for improved unconstrained handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5), 2009.
9. Graves, A., Mohamed, A., and Hinton, G. E. Speech recognition with deep recurrent neural networks. In ICASSP, pp. 6645-6649. IEEE, 2013.
10. Hihi, S. E. and Bengio, Y. Hierarchical recurrent neural networks for long-term dependencies. In Touretzky, D. S., Mozer, M. C, and Hasselmo, M. E. (eds.), Advances in Neural Information Processing Systems 8, pp. 493-199. MIT Press, 1996.
11. Hochreiter, S. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München, 1991. Advisor: J. Schmidhuber.
12. Hochreiter, S. and Schmidhuber, J. Long Short-Term Memory. Neural Computation, 9(8): 1735-1780, 1997.
13. Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In Kremer, S. C. and Kolen, J. F. (eds.), A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001.
14. Laeger, H. Short term memory in echo state networks. GMD-Report 152, GMD - German National Research Institute for Computer Science, 2002. URL http://www.faculty.jacobs-university.de/hjaeger/pubs/STMEchoStatesTechRep.pdf
15. Lin, T., Home, B.G., Tino, P., and Giles, C.L. Learning long-term dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks, 7(6): 1329-1338, 1996.
16. Liwicki, M. and Bunke, H. IAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard. In Proc. 8th Int. Conf. on Document Analysis and Recognition, volume 2, pp. 956-961, 2005.
17. Liwicki, M., Graves, A., Bunke, H., and Schmidhuber, J. A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks. In Proceedings of the 9th International Conference on Document Analysis and Recognition, September 2007.
18. Martens, J. and Sutskever, I. Learning recurrent neural networks with Hessian-free optimization. In Getoor, L. and Scheffer, T. (eds.), Proceedings of the 28th International Conference on Machine Learning (ICML-1I), pp. 1033-1040, New York, NY, USA. lune 2011. ACM. ISBN 978-1-4503-0619-5.
19. Mayer, H., Gomez, F., Wierstra, D, Nagy, I., Knoll, A., and Schmidhuber, J. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. In Proceedings of the International Conference on Intelligent Robotics and Systems (IROS-06, Beijing), 2006.
20. Mermelstein, P. Distance measures for speech recognition: Psychological and instrumental. In Chen, C. H. (ed.), Pattern Recognition and Artificial Intelligence, pp. 374-388. Academic Press, New York, 1976.
21. Mozer, M. C. Induction of multiscale temporal structure. In Lippman, D. S., Moody, I. E., and Touretzky, D. S. (eds.), Advances in Neural Information Processing Systems 4, pp. 275-282. Morgan Kaufmann, 1992.
22. Mozer, M. C. Neural network music composition by prediction: Exploring the benefits of psychoacoustic constraints and multi-scale processing. Connection Science, 6(2-3):247-280, 1994.
23. Perez-Ortiz, I. A., Gers, F. A., Eck, D., and Schmidhuber, J. Kaiman filters improve LSTM network performance in problems unsolvable by traditional recurrent nets. Neural Networks, (16):241-250, 2003.
24. Ring, M. Learning sequential tasks by incrementally adding higher orders. In S. I. Hanson, J. D. Cowan and Giles, C. L. (eds.), Advances in Neural Information Processing Systems 5, pp. 115-122, Morgan Kaufmann, 1993.
25. Ring, M. Recurrent transition hierarchies for continual learning: A general overview. In Lifelong Learning, volume WS-11-15 of AM/ Workshops. A A AI, 2011.
26. Robinson, A. J. and Fallside, F. The utility driven dynamic error propagation network. Technical Report CUED/F-INFENG/TR.l, Cambridge University Engineering Department, 1987.
27. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning internal representations by error propagation. In Rumelhart, D. E. and McClelland, J. L. (eds.), Parallel Distributed Processing, volume 1, pp. 318-362. MIT Press, 1986.
28. Sak, H., Senior, A., and Beaufays, F. Long Short-Term Memory based recurrent neural network architectures for large vocabulary speech recognition. Technical report, Google, February 2014. arXiv:1402.1128 [cs.NE].
29. Schmidhuber, I. Neural sequence chunkers. Technical Report FKI-148-91, Institut für Informatik, Technische Universität München, April 1991.
30. Schmidhuber, J. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2): 234-242, 1992.
31. Schmidhuber, J., Wierstra, D., and Gomez, F. J. Evolino: Hybrid neuroevolution / optimal linear search for sequence prediction. In Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI), pp. 853-858, 2005.
32. Schmidhuber, J., Wierstra, D., Gagliolo, M., and Gomez, F. J. Training recurrent networks by Evolino. Neural Computation, 19(3):757-779, 2007.
33. Schuster, M. and Paliwal, K. K. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45:2673-2681, November 1997.
34. Sutskever, I. and Hinton, G. E. Temporal kernel recurrent neural networks. Neural Networks, 23(2):239-243, March 2010.
35. Sutskever, I., Martens, J., and Hinton, G. Generating text with recurrent neural networks. In Getoor, L. and Scheffer, T. (eds.), Proceedings of the 28th International Conference on Machine Learning (ICML-ll), pp. 1017-1024, New York, NY, USA, June 2011. ACM. ISBN 978-1-4503-0619-5.
36. Sutskever, I., Martens, J., Dahl, G. E., and Hinton, G. E. On the importance of initialization and momentum in deep learning. In Dasgupta, Sanjoy and Mcallester, David (eds.), Proceedings of the 30th International Conference on Machine Learning (ICML-13), volume 28, pp. 1139-1147. IMLR Workshop and Conference Proceedings, May 2013. URL http://jmlr.org/proceedings/papers/v28/sutskever13.pdf.
37. Werbos, P. J. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1, 1988.
38. Williams, R. J. Complexity of exact gradient computation algorithms for recurrent neural networks. Technical Report NU-CCS-89-27, Boston: Northeastern University, College of Computer Science, 1989.
39. Williams, R. J. Training recurrent networks using the extended kalman filter. In Neural Networks, 1992. IJCNN, International Joint Conference on, volume 4, pp. 241-246. IEEE, 1992.