Simple and scalable predictive uncertainty estimation using deep ensembles

Advances in Neural Information Processing Systems

Volumn 2017-December, Issue , 2017, Pages 6403-6414

  Lakshminarayanan, Balaji ^a   Pritzel, Alexander ^a   Blundell, Charles ^a  

^a DEEPMIND   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

DEEP NEURAL NETWORKS; REACTOR CORES; STATISTICAL TESTS;

APPROXIMATE BAYESIAN; HYPER-PARAMETER; NEURAL NETWORKS (NNS); PREDICTIVE UNCERTAINTY; STATE OF THE ART; TRAINING PROCEDURES; UNCERTAINTY ESTIMATES; UNSOLVED PROBLEMS;

UNCERTAINTY ANALYSIS;

EID: 85046740096     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (61)

1. M. Abbasi and C. Gagné. Robustness to adversarial examples through an ensemble of specialists. arXiv preprint arXiv: 1702.06856, 2017.
2. B. Alipanahi, A. Delong, M. T. Weirauch, and B. J. Frey. Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning. Nature biotechnology, 33(8): 831-838, 2015.
3. D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety. arXiv preprint arXiv: 1606.06565, 2016.
4. J. M. Bernardo and A. F. Smith. Bayesian Theory, Volume 405. John Wiley & Sons, 2009.
5. C. M. Bishop. Mixture density networks. 1994.
6. C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural networks. In ICML, 2015.
7. L. Breiman. Bagging predictors. Machine learning, 24(2): 123-140, 1996.
8. L. Breiman. Random forests. Machine learning, 45(1): 5-32, 2001.
9. G. W. Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 1950.
10. C. Bucila, R. Caruana, and A. Niculescu-Mizil. Model compression. In KDD. ACM, 2006.
11. B. Clarke. Comparing Bayes model averaging and stacking when model approximation error cannot be ignored. J. Mach. Learn. Res. (JMLR), 4: 683-712, 2003.
12. A. P. Dawid. The well-calibrated Bayesian. Journal of the American Statistical Association, 1982.
13. M. H. DeGroot and S. E. Fienberg. The comparison and evaluation of forecasters. The statistician, 1983.
14. T. G. Dietterich. Ensemble methods in machine learning. In Multiple classifier systems. 2000.
15. Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
16. P. Geurts, D. Ernst, and L. Wehenkel. Extremely randomized trees. Machine learning, 63(1): 3-42, 2006.
17. T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477): 359-378, 2007.
18. I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
19. A. Graves. Practical variational inference for neural networks. In NIPS, 2011.
20. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. arXiv preprint arXiv: 1706.04599, 2017.
21. L. Hasenclever, S. Webb, T. Lienart, S. Vollmer, B. Lakshminarayanan, C. Blundell, and Y. W. Teh. Distributed Bayesian learning with stochastic natural-gradient expectation propagation and the posterior server. arXiv preprint arXiv: 1512.09327, 2015.
22. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
23. D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv: 1610.02136, 2016.
24. J. M. Hernández-Lobato and R. P. Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In ICML, 2015.
25. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 29(6): 82-97, 2012.
26. G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.02531, 2015.
27. G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger. Snapshot ensembles: Train 1, get M for free. ICLR submission, 2017.
28. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1): 79-87, 1991.
29. D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterization trick. In NIPS, 2015.
30. A. Korattikara, V. Rathod, K. Murphy, and M. Welling. Bayesian dark knowledge. In NIPS, 2015.
31. A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
32. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
33. A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial machine learning at scale. arXiv preprint arXiv: 1611.01236, 2016.
34. B. Lakshminarayanan. Decision trees and forests: a probabilistic perspective. PhD thesis, UCL (University College London), 2016.
35. Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553): 436-444, 2015.
36. S. Lee, S. Purushwalkam, M. Cogswell, D. Crandall, and D. Batra. Why M heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv: 1511.06314, 2015.
37. S. Lee, S. P. S. Prakash, M. Cogswell, V. Ranjan, D. Crandall, and D. Batra. Stochastic multiple choice learning for training diverse deep ensembles. In NIPS, 2016.
38. Y. Li, J. M. Hernández-Lobato, and R. E. Turner. Stochastic expectation propagation. In NIPS, 2015.
39. C. Louizos and M. Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. arXiv preprint arXiv: 1603.04733, 2016.
40. D. J. MacKay. Bayesian methods for adaptive models. PhD thesis, California Institute of Technology, 1992.
41. S.-i. Maeda. A Bayesian encourages dropout. arXiv preprint arXiv: 1412.7003, 2014.
42. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv: 1301.3781, 2013.
43. T. P. Minka. Bayesian model averaging is not model combination. 2000.
44. T. Miyato, S.-i. Maeda, M. Koyama, K. Nakae, and S. Ishii. Distributional smoothing by virtual adversarial examples. In ICLR, 2016.
45. V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010.
46. R. M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag New York, Inc., 1996.
47. D. A. Nix and A. S. Weigend. Estimating the mean and variance of the target probability distribution. In IEEE International Conference on Neural Networks, 1994.
48. I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. In NIPS, 2016.
49. J. Quinonero-Candela, C. E. Rasmussen, F. Sinz, O. Bousquet, and B. Schölkopf. Evaluating predictive uncertainty challenge. In Machine Learning Challenges. Springer, 2006.
50. C. E. Rasmussen and J. Quinonero-Candela. Healing the relevance vector machine through augmentation. In ICML, 2005.
51. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3): 211-252, 2015.
52. S. Singh, D. Hoiem, and D. Forsyth. Swapout: Learning an ensemble of deep architectures. In NIPS, 2016.
53. J. T. Springenberg, A. Klein, S. Falkner, and F. Hutter. Bayesian optimization with robust Bayesian neural networks. In Advances in Neural Information Processing Systems, pages 4134-4142, 2016.
54. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 2014.
55. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In ICLR, 2014.
56. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2818-2826, 2016.
57. F. Tramèr, A. Kurakin, N. Papernot, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv: 1705.07204, 2017.
58. O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In NIPS, 2016.
59. M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In ICML, 2011.
60. D. H. Wolpert. Stacked generalization. Neural networks, 5(2): 241-259, 1992.
61. J. Zhou and O. G. Troyanskaya. Predicting effects of noncoding variants with deep learning-based sequence model. Nature methods, 12(10): 931-934, 2015.