On calibration of modern neural networks

34th International Conference on Machine Learning, ICML 2017

Volumn 3, Issue , 2017, Pages 2130-2143

  Guo, Chuan ^a   Pleiss, Geoff ^a   Sun, Yu ^a   Weinberger, Kilian Q ^a  

^a Department of Obstetrics Gynecology   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; INFORMATION RETRIEVAL SYSTEMS; LEARNING SYSTEMS;

CALIBRATION METHOD; CLASSIFICATION MODELS; DOCUMENT CLASSIFICATION; POST PROCESSING; PROBABILITY ESTIMATE; SINGLE PARAMETER; TEMPERATURE SCALING; WEIGHT DECAY;

CALIBRATION;

EID: 85047020009     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (47)

1. Bengio, Yoshua, Goodfellow, Ian J, and Courville, Aaron. Deep learning. Nature, 521:436-444,2015
2. Bojarski, Mariusz, Del Testa, Davide, Dworakowski, Daniel, Firner, Bernhard, Flepp, Beat, Goyal, Prasoon, Jackel, Lawrence D, Monfort, Mathew, Muller, Urs, Zhang, Jiakai, et al. End to end learning for self-driving cars. ArXiv preprint arXiv:1604.07316, 2016
3. Caruana, Rich, Lou, Yin, Gehrke, Johannes, Koch, Paul, Sturm, Marc, and Elhadad, Noemie. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day rcadmission. In KDD, 2015
4. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clément. Torch7: A matlab-like environment for machine learning. In BigLearn Workshop, NIPS, 2011
5. Cosmides, Leda and Tooby, John. Are humans good intuitive statisticians after all? rethinking some conclusions from the literature on judgment under uncertainty, cog-w7vi,58(l):l-73, 1996
6. DeGroot, Morris H and Fienberg, Stephen E. The comparison and evaluation of forecasters. The statistician, pp. 12-22, 1983
7. Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009
8. Denker, John S and Lecun, Yann. Transforming neural-net output levels to probability distributions. In NIPS, pp. 853-859, 1990
9. Friedman, Jerome, Hastie, Trevor, and Tibshirani, Robert. The elements of statistical learning, volume 1. Springer series in statistics Springer, Berlin, 2001
10. Gal, Yarin and Ghahramani, Zoubin. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016
11. Girshick, Ross. Fastr-cnn. In ICCV, pp. 1440-1448, 2015
12. Hannun, Awni, Case, Carl, Casper, Jared, Catanzaro, Bryan, Diamos, Greg, Elsen, Erich, Prenger, Ryan, Satheesh, Sanjeev, Sengupta, Shubho, Coates, Adam, et al. Deep speech: Scaling up end-to-end speech recognition. ArXiv preprint arXiv:I4I2.5567, 2014
13. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016
14. Hendrycks, Dan and Gimpel, Kevin. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017
15. Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the knowledge in a neural network. 2015
16. Huang, Gao, Sun, Yu, Liu, Zhuang, Sedra, Daniel, and Weinberger, Kilian. Deep networks with stochastic depth. In ECCV, 2016
17. Huang, Gao, Liu, Zhuang, Weinberger, Kilian Q, and van der Maaten, Laurens. Densely connected convolutional networks. In CVPR, 2017
18. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariatc shift. 2015
19. Iyycr, Mohit, Manjunatha, Varun, Boyd-Grabcr, Jordan, and Daumé III, Hal. Deep unordered composition rivals syntactic methods for text classification. In ACL, 2015
20. Jaynes, Edwin T. Information theory and statistical mechanics. Physical review, 106(4):620, 1957
21. Jiang, Xiaoqian, Osi, Melanie, Kim, Jihoon, and Ohno-Machado, Lucila. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 19(2):263-274, 2012
22. Kendall, Alex and Cipolla, Roberto. Modelling uncertainty in deep learning for camera relocalization. 2016
23. Kendall, Alex and Gal, Yarin. What uncertainties do we need in Bayesian deep learning for computer vision? arXiv preprint arXiv: 1703.04977, 2017
24. Krause, Jonathan, Stark, Michael, Deng, Jia, and Fei-Fei, Li. 3d object representations for fine-grained categorization. In IEEE Workshop on 3D Representation and Recognition (3dRR), Sydney, Australia, 2013
25. Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple layers of features from tiny images, 2009
26. Kuleshov, Volodymyr and Ermon, Stefano. Reliable confidence estimation via online learning. ArXiv preprint arXiv: 1607.03594, 2016
27. Kuleshov, Volodymyr and Liang, Percy. Calibrated structured prediction. In NIPS, pp. 3474-3482, 2015
28. Lakshminarayanan, Balaji, Pritzel, Alexander, and Blundell, Charles. Simple and scalable predictive uncertainty estimation using deep ensembles. ArXiv preprint arXiv:1612.01474, 2016
29. LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998
30. MacKay, David JC. A practical Bayesian framework for backpropagation networks. Neural computation, 4(3): 448-472, 1992
31. Naeini, Mahdi Pakdaman, Cooper, Gregory F, and Hauskrecht, Milos. Obtaining well calibrated probabilities using Bayesian binning. In AAAI, pp. 2901, 2015
32. Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In Deep Learning and Unsupervised Feature Learning Workshop, NIPS, 2011
33. Niculescu-Mizil, Alcxandru and Caruana, Rich. Predicting good probabilities with supervised learning. In ICML, pp. 625-632,2005
34. Pereyra, Gabriel, Tucker, George, Chorowski, Jan, Kaiser, Lukasz, and Hinton, Geoffrey. Regularizing neural networks by penalizing confident output distributions. ArXiv preprint arXiv:1701.06548, 2017
35. Piatt, John et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3): 61-74, 1999
36. Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015
37. Sochcr, Richard, Pcrelygin, Alex, Wu, Jean, Chuang, Jason, Manning, Christopher D., Ng, Andrew, and Potts, Christopher. Recursive deep models for semantic com-positionality over a sentiment treebank. In EMNLP, pp. 1631-1642,2013
38. Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014
39. Srivastava, Rupesh Kumar, Greff, Klaus, and Schmid-huber, Jürgen. Highway networks. ArXiv preprint arXiv:l505.00387,2015
40. Tai, Kai Sheng, Socher, Richard, and Manning, Christopher D. Improved semantic representations from tree-structured long short-term memory networks. 2015
41. Vapnik, Vladimir N. Statistical Learning Theory. Wiley-Interscience, 1998
42. Welinder, P., Branson, S., Mita, T., Wah, C, Schroff, F., Belongie, S., and Perona, P. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010
43. Xiong, Wayne, Droppo, Jasha, Huang, Xuedong, Seide, Frank, Seltzer, Mike, Stolcke, Andreas, Yu, Dong, and Zweig, Geoffrey. Achieving human parity in conversational speech recognition. ArXiv preprint arXiv:1610.05256,20\6
44. Zadrozny, Bianca and Elkan, Charles. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In ICML, pp. 609-616, 2001
45. Zadrozny, Bianca and Elkan, Charles. Transforming classifier scores into accurate multiclass probability estimates. In KDD, pp. 694-699, 2002
46. Zagoruyko, Sergey and Komodakis, Nikos. Wide residual networks. InβMVC, 2016
47. Zhang, Chiyuan, Bengio, Samy, Hardt, Moritz, Recht, Benjamin, and Vinyals, Oriol. Understanding deep learning requires rethinking generalization. In ICLR, 2017.