Deep Bayesian bandits showdown: An empirical comparison of Bayesian deep networks for Thompson sampling

6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings

Volumn , Issue , 2018, Pages

  Riquelme, Carlos ^a   Tucker, George ^a   Snoek, Jasper ^a  

^a GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BAYESIAN NETWORKS; DECISION MAKING; MACHINE LEARNING; NEURAL NETWORKS; REINFORCEMENT LEARNING; UNCERTAINTY ANALYSIS;

APPROXIMATE BAYESIAN; BALANCE EXPLORATION; CONTEXTUAL BANDITS; EMPIRICAL - COMPARISONS; NEURAL NETWORK MODEL; SEQUENTIAL DECISION MAKING; THOMPSON SAMPLINGS; UNCERTAINTY ESTIMATES;

DEEP LEARNING;

EID: 85083952304     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (51)

1. Agrawal, Shipra and Goyal, Navin. Analysis of thompson sampling for the multi-armed bandit problem. In International Conference on Learning Theory, 2012.
2. Ahn, Sungjin, Balan, Anoop Korattikara, and Welling, Max. Bayesian posterior sampling via stochastic gradient fisher scoring. In International Conference on Machine Learning, 2012.
3. Asuncion, Arthur and Newman, David. UCI machine learning repository, 2007.
4. Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
5. Bertin-Mahieux, Thierry, Ellis, Daniel P.W., Whitman, Brian, and Lamere, Paul. The million song dataset. In International Conference on Music Information Retrieval, 2011.
6. Bishop, Christopher M. Pattern recognition and machine learning. springer, 2006.
7. Blundell, Charles, Cornebise, Julien, Kavukcuoglu, Koray, and Wierstra, Daan. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613–1622, 2015.
8. Bonilla, Edwin V, Chai, Kian M., and Williams, Christopher. Multi-task gaussian process prediction. In Advances in Neural Information Processing Systems, 2008.
9. Bubeck, Sébastien, Munos, Rémi, and Stoltz, Gilles. Pure exploration in multi-armed bandits problems. In International conference on Algorithmic learning theory, pp. 23–37. Springer, 2009.
10. Chapelle, Olivier and Li, Lihong. An empirical evaluation of Thompson sampling. In Advances in Neural Information Processing Systems 24, 2011.
11. Efron, Bradley. The jackknife, the bootstrap and other resampling plans. SIAM, 1982.
12. Elmachtoub, Adam N, McNellis, Ryan, Oh, Sechan, and Petrik, Marek. A practical method for solving contextual bandit problems using decision trees. arXiv preprint arXiv:1706.04687, 2017.
13. Fortunato, Meire, Azar, Mohammad Gheshlaghi, Piot, Bilal, Menick, Jacob, Osband, Ian, Graves, Alex, Mnih, Vlad, Munos, Remi, Hassabis, Demis, Pietquin, Olivier, Blundell, Charles, and Legg, Shane. Noisy networks for exploration. arXiv:1706.10295, 2017.
14. Gal, Yarin and Ghahramani, Zoubin. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International conference on machine learning, pp. 1050–1059, 2016.
15. Gelman, Andrew, Vehtari, Aki, Jylänki, Pasi, Robert, Christian, Chopin, Nicolas, and Cunningham, John P. Expectation propagation as a way of life. arXiv preprint arXiv:1412.4869, 2014.
16. Goldberg, Ken, Roeder, Theresa, Gupta, Dhruv, and Perkins, Chris. Eigentaste: A constant time collaborative filtering algorithm. Information Retrieval, 4(2):133–151, 2001.
17. Granmo, OleChristoffer. Solving twoarmed Bernoulli bandit problems using a Bayesian learning automaton. International Journal of Intelligent Computing and Cybernetics, 3(2):207–234, 2010.
18. Graves, Alex. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pp. 2348–2356, 2011.
19. Hensman, James, Matthews, Alexander G. de G., and Ghahramani, Zoubin. Scalable variational gaussian process classification. In Proceedings of AISTATS, 2015.
20. Hernández-Lobato, José Miguel, Li, Yingzhen, Rowland, Mark, Bui, Thang D., Hernández-Lobato, Daniel, and Turner, Richard E. Black-box alpha divergence minimization. In International Conference on Machine Learning, 2016.
21. Hinton, Geoffrey E and Van Camp, Drew. Keeping the neural networks simple by minimizing the description length of the weights. In Computational learning theory, pp. 5–13. ACM, 1993.
22. Hron, Jiri, Matthews, Alexander G de G, and Ghahramani, Zoubin. Variational gaussian dropout is not bayesian. arXiv preprint arXiv:1711.02989, 2017.
23. Jordan, Michael I, Ghahramani, Zoubin, Jaakkola, Tommi S, and Saul, Lawrence K. An introduction to variational methods for graphical models. Machine learning, 37(2):183–233, 1999.
24. Kingma, Diederik P, Salimans, Tim, and Welling, Max. Variational dropout and the local repa-rameterization trick. In Advances in Neural Information Processing Systems, pp. 2575–2583, 2015.
25. Kohavi, Ron. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In International Conference On Knowledge Discovery and Data Mining, 1996.
26. Li, Chunyuan, Chen, Changyou, Carlson, David, and Carin, Lawrence. Preconditioned stochastic gradient langevin dynamics for deep neural networks. In AAAI Conference on Artificial Intelligence, 2016.
27. Mandt, Stephan, Hoffman, Matthew D., and Blei, David M. A variational analysis of stochastic gradient algorithms. In International Conference on Machine Learning, 2016.
28. Matthews, Alexander G. de G., van der Wilk, Mark, Nickson, Tom, Fujii, Keisuke., Boukouvalas, Alexis, León-Villagrá, Pablo, Ghahramani, Zoubin, and Hensman, James. GPflow: A Gaussian process library using TensorFlow. Journal of Machine Learning Research, 2017.
29. Minka, Thomas. Power ep. 2004.
30. Minka, Thomas P. Expectation propagation for approximate bayesian inference. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pp. 362–369. Morgan Kaufmann Publishers Inc., 2001a.
31. Minka, Thomas Peter. A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, 2001b.
32. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, February 2015.
33. Neal, Radford M. Bayesian learning for neural networks. Dept. of Computer Science, University of Toronto, 1994.
34. Opper, Manfred and Winther, Ole. Gaussian processes for classification: Mean-field algorithms. Neural computation, 12(11):2655–2684, 2000.
35. Osband, Ian, Blundell, Charles, Pritzel, Alexander, and Van Roy, Benjamin. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, pp. 4026–4034, 2016.
36. Patterson, Sam and Teh, Yee Whye. Stochastic gradient riemannian langevin dynamics on the probability simplex. In Advances in Neural Information Processing Systems, 2013.
37. Pearl, Judea. Fusion, propagation, and structuring in belief networks. Artificial intelligence, 29(3): 241–288, 1986.
38. Plappert, Matthias, Houthooft, Rein, Dhariwal, Prafulla, Sidor, Szymon, Chen, Richard Y., Chen, Xi, Asfour, Tamim, Abbeel, Pieter, and Andrychowicz, Marcin. Parameter space noise for exploration. arXiv:1706.01905, 2017.
39. Rasmussen, Carl Edward and Williams, Christopher K. I. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
40. Riquelme, Carlos, Ghavamzadeh, Mohammad, and Lazaric, Alessandro. Active learning for accurate estimation of linear models. In International Conference on Machine Learning, 2017.
41. Russo, Dan and Van Roy, Benjamin. Learning to optimize via information-directed sampling. In Advances in Neural Information Processing Systems, pp. 1583–1591, 2014.
42. Russo, Daniel, Tse, David, and Van Roy, Benjamin. Time-sensitive bandit learning and satisficing thompson sampling. arXiv preprint arXiv:1704.09028, 2017.
43. Schaul, Tom, Quan, John, Antonoglou, Ioannis, and Silver, David. Prioritized experience replay. In International Conference on Learning Representations, 2016.
44. Schlimmer, Jeff. Mushroom records drawn from the audubon society field guide to north american mushrooms. GH Lincoff (Pres), New York, 1981.
45. Snelson, Edward and Ghahramani, Zoubin. Sparse gaussian processes using pseudo-inputs. In Weiss, Y., Schölkopf, P. B., and Platt, J. C. (eds.), Advances in Neural Information Processing Systems, 2006.
46. Snoek, Jasper, Rippel, Oren, Swersky, Kevin, Kiros, Ryan, Satish, Nadathur, Sundaram, Narayanan, Patwary, Mostofa, Prabhat, Mr, and Adams, Ryan. Scalable bayesian optimization using deep neural networks. In International Conference on Machine Learning, 2015.
47. Srivastava, Nitish, Hinton, Geoffrey E, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929–1958, 2014.
48. Thompson, William R. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285–294, 1933.
49. Titsias, Michalis K. Variational learning of inducing variables in sparse gaussian processes. In International Conference on Artificial Intelligence and Statistics, 2009.
50. Wainwright, Martin J, Jordan, Michael I, et al. Graphical models, exponential families, and variational inference. Foundations and TrendsR in Machine Learning, 1(1–2):1–305, 2008.
51. Welling, Max and Teh, Yee Whye. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning, 2011.