The statistical recurrent unit

34th International Conference on Machine Learning, ICML 2017

Volumn 6, Issue , 2017, Pages 4098-4107

  Oliva, Junier B ^a   Poczos, Barnabas ^a   Schneider, Jeff ^a  

^a Carnegie Mellon University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING SYSTEMS; LONG SHORT-TERM MEMORY; SULFUR DETERMINATION;

HYPERPARAMETERS; LONG-TERM DEPENDENCIES; MOVING AVERAGES; REAL-WORLD TASK;

NETWORK ARCHITECTURE;

EID: 85041927817     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. NCEP North American Regional Reanalysis. https://data.noaa.gov/dataset/ncep-north-american-regional-reanalysis-narr-for-1979-to-present. Accesscd: 2016-10-17.
2. Nba movement data. https://github.com/sealneaward/nba-movement-data. Accessed: 2016-10-17.
3. Abadi, Martin, Barham, Paul, Chen, Jianmin, Chen, Zhifeng, Davis, Andy, Dean, Jeffrey, Devin, Matthieu, Ghemawat, Sanjay, Irving, Geoffrey, Isard, Michael, et al. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). Savannah, Georgia, USA, 2016.
4. Bengio, Yoshua, Boulangcr-Lcwandowski, Nicolas, and Pascanu, Razvan. Advances in optimizing recurrent networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 8624-8628. IEEE, 2013.
5. Bergstra, James and Bengio, Yoshua. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-305, 2012.
6. Bergstra, James, Komer, Brent, Eliasmith, Chris, Yamins, Dan, and Cox, David D. Hyperopt: A python library for model selection and hyperparameter optimization. Computational Science & Discovery, 8(1):014008, 2015.
7. Boulangcr-Lewandowski, Nicolas, Bengio, Yoshua, and Vincent, Pascal. Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. arXiv preprint arXiv:1206.6392, 2012.
8. Cho, Kyunghyun, Van Merrienboer, Bart, Bahdanau, Dzmitry, and Bengio, Yoshua. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
9. Chung, Junyoung, Gulcehre, Caglar, Cho, KyungHyun, and Bengio, Yoshua. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
10. Donahue, Jeffrey, Anne Hendricks, Lisa, Guadarrama, Sergio, Rohrbach, Marcus, Venugopalan, Subhashini, Saenko, Kate, and Darrell, Trevor. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2625- 2634, 2015.
11. Edwards, Harrison and Storkey, Amos. Towards a neural statistician. arXiv preprint arXiv:1606.02185, 2016.
12. Elman, Jeffrey L. Finding structure in time. Cognitive science, 14(2): 179-211, 1990.
13. Graves, Alex and Jaitly, Navdeep. Towards end-to-end speech recognition with recurrent neural networks. In ICML, volume 14, pp. 1764-1772, 2014.
14. Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In Acoustics, speech and signal processing (icassp), 2013 IEEE international conference on, pp. 6645-6649. IEEE, 2013.
15. Hochreiter, Sepp and Schmidhuber, Jiirgen. Long short- term memory. Neural computation, 9(8): 1735-1780, 1997.
16. Hochreiter, Sepp, Bengio, Yoshua, Frasconi, Paolo, and Schmidhuber, Jiirgen. Gradient flow in recurrent nets: The difficulty of learning long-term dependencies, 2001.
17. Iandola, Forrest N, Han, Song, Moskewicz, Matthew W, Ashraf, Khalid, Dally, William J, and Keutzer, Kurt. Squeezenet: Alexnet-level accuracy with 50x fewer parameters andj 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016.
18. Jaeger, Herbert, Luko Sevifiius, Mantas, Popovici, Dan, and Siewert, Udo. Optimization and applications of echo state networks with leaky-integrator neurons. Neural networks, 20(3):335-352, 2007.
19. Jarrett, Kevin, Kavukcuoglu, Koray, Le Cun, Yann, et al. What is the best multi-stage architecture for object recognition? In Computer Vision, 2009 IEEE 12th International Conference on, pp. 2146-2153. IEEE, 2009.
20. Koutnik, Jan, Greff, Klaus, Gomez, Faustino, and Schmidhuber, Juergen. A clockwork rnn. pp. 1863-1871, 2014.
21. Le, Quoc V, Jaitly, Navdeep, and Hinton, Geoffrey E. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00941, 2015.
22. Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. arXiv preprint arXiv:1312.4400, 2013.
23. Mikolov, Tomas, Joulin, Armand, Chopra, Sumit, Mathieu, Michael, and Ranzato, Marc'Aurelio. Learning longer memory in recurrent neural networks. arXiv preprint arXiv:1412.7753, 2015.
24. Muandet, Krikamol, Fukumizu, Kenji, Sriperumbudur, Bharath, and Scholkopf, Bernhard. Kernel mean embedding of distributions: A review and beyonds. arXiv preprint arXiv:1605.09522, 2016.
25. Miiller, Meinard. Information retrieval for music and motion, volume 2. Springer, 2007.
26. Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
27. Pascanu, Razvan, Mikolov, Tomas, and Bcngio, Yoshua. On the difficulty of training recurrent neural networks. ICML(3), 28:1310-1318, 2013.
28. Ramsay, J.O. and Silverman, B.W. Applied functional data analysis: methods and case studies, volume 77. Springer New York:, 2002.
29. Smola, Alex, Gretton, Arthur, Song, Le, and Scholkopf, Bernhard. A hilbert space embedding for distributions. In International Conference on Algorithmic Learning Theory, pp. 13-31. Springer, 2007.
30. Vinyals, Oriol, Kaiser, tukasz, Koo, Terry, Petrov, Slav, Sutskever, Ilya, and Hinton, Geoffrey. Grammar as a foreign language. In Advances in Neural Information Processing Systems, pp. 2773-2781, 2015.
31. Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014.