To trust or not to trust a classifier

Advances in Neural Information Processing Systems

Volumn 2018-December, Issue , 2018, Pages 5541-5552

  Jiang, Heinrich ^a   Kim, Been ^a   Gupta, Maya ^a   Guan, Melody Y ^b  

^a GOOGLE INC   (United States)

^b STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLASSIFIER PERFORMANCE; CONFIDENCE SCORE; HIGH-PRECISION; MACHINE LEARNING RESEARCH; NEAREST NEIGHBOR CLASSIFIER; STATISTICAL CONSISTENCIES; TESTING EXAMPLES; TOPOLOGICAL DATA ANALYSIS;

LEARNING SYSTEMS;

EID: 85063928915     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (58)

1. Kush R Varshney and Homa Alemzadeh. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big data, 5(3):246-255, 2017.
2. John D Lee and Katrina A See. Trust in automation: Designing for appropriate reliance. Human factors, 46(1):50-80, 2004.
3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. CoRR, abs/1606.06565, 2016.
4. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. arXiv preprint arXiv:1706.04599, 2017.
5. Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction. In Advances in Neural Information Processing Systems, pages 3474-3482, 2015.
6. Foster J Provost, Tom Fawcett, and Ron Kohavi. The case against accuracy estimation for comparing induction algorithms. In ICML, volume 98, pages 445-453, 1998.
7. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
8. Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 427-436, 2015.
9. John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61-74, 1999.
10. Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694-699. ACM, 2002.
11. Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning, pages 625-632. ACM, 2005.
12. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pages 6405-6416, 2017.
13. Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016.
14. Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pages 1050-1059, 2016.
15. Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, pages 5580-5590, 2017.
16. Peter L Bartlett and Marten H Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9(Aug):1823-1840, 2008.
17. Ming Yuan and Marten Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11(Jan):111-130, 2010.
18. Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory, pages 67-82. Springer, 2016.
19. Yves Grandvalet, Alain Rakotomamonjy, Joseph Keshet, and Stéphane Canu. Support vector machines with a reject option. In Advances in Neural Information Processing Systems, pages 537-544, 2009.
20. Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. In Advances in Neural Information Processing Systems, pages 1660-1668, 2016.
21. Radu Herbei and Marten H Wegkamp. Classification with reject option. Canadian Journal of Statistics, 34(4):709-721, 2006.
22. Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, and Scott Yang. Online learning with abstention. arXiv preprint arXiv:1703.03478, 2017.
23. C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46, 1970.
24. Bernard Dubuisson and Mylene Masson. A statistical decision rule with incomplete knowledge about classes. Pattern Recognition, 26(1):155-165, 1993.
25. Giorgio Fumera, Fabio Roli, and Giorgio Giacinto. Multiple reject thresholds for improving classification reliability. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pages 863-871. Springer, 2000.
26. Carla M Santos-Pereira and Ana M Pires. On optimal reject rules and ROC curves. Pattern Recognition Letters, 26(7):943-952, 2005.
27. Francesco Tortorella. An optimal reject rule for binary classifiers. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pages 611-620. Springer, 2000.
28. Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In Pattern Recognition with Support Vector Machines, pages 68-82. Springer, 2002.
29. Thomas CW Landgrebe, David MJ Tax, Pavel Paclík, and Robert PW Duin. The interaction between classification and reject performance for distance-based reject-option classifiers. Pattern Recognition Letters, 27(8):908-917, 2006.
30. Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(May):1605-1641, 2010.
31. Yair Wiener and Ran El-Yaniv. Agnostic selective classification. In Advances in Neural Information Processing Systems, pages 1665-1673, 2011.
32. David MJ Tax and Robert PW Duin. Growing a multi-class classifier with a reject option. Pattern Recognition Letters, 29(10):1565-1570, 2008.
33. Joseph Wang, Kirill Trapeznikov, and Venkatesh Saligrama. Efficient learning by directed acyclic graph for resource constrained prediction. Advances in Neural Information Processing Systems (NIPS), 2015.
34. Nathan Parrish, Hyrum S. Anderson, Maya R. Gupta, and Dun Yu Hsaio. Classifying with confidence from incomplete information. Journal of Machine Learning Research, 14(December):3561-3589, 2013.
35. Wei Fan, Fang Chu, Haixun Wang, and Philip S. Yu. Pruning and dynamic scheduling of cost-sensitive ensembles. AAAI, 2002.
36. Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765, 2018.
37. John A Hartigan. Clustering algorithms. 1975.
38. Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd, pages 226-231, 1996.
39. Alexandre B Tsybakov et al. On nonparametric estimation of density level sets. The Annals of Statistics, 25(3):948-969, 1997.
40. Aarti Singh, Clayton Scott, Robert Nowak, et al. Adaptive Hausdorff estimation of density level sets. The Annals of Statistics, 37(5B):2760-2782, 2009.
41. Philippe Rigollet, Régis Vert, et al. Optimal rates for plug-in estimators of density level sets. Bernoulli, 15(4):1154-1178, 2009.
42. Heinrich Jiang. Density level set estimation on manifolds with DBSCAN. In International Conference on Machine Learning, pages 1684-1693, 2017.
43. Alessandro Rinaldo and Larry Wasserman. Generalized density clustering. The Annals of Statistics, 38(5):2678-2722, 2010.
44. Partha Niyogi, Stephen Smale, and Shmuel Weinberger. Finding the homology of submanifolds with high confidence from random samples. Discrete & Computational Geometry, 39(1-3):419-441, 2008.
45. Christopher Genovese, Marco Perone-Pacifico, Isabella Verdinelli, and Larry Wasserman. Minimax manifold estimation. Journal of Machine Learning Research, 13(May):1263-1291, 2012.
46. Sivaraman Balakrishnan, Srivatsan Narayanan, Alessandro Rinaldo, Aarti Singh, and Larry Wasserman. Cluster trees on manifolds. In Advances in Neural Information Processing Systems, pages 2679-2687, 2013.
47. Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning. 2001.
48. Yann LeCun. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
49. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
50. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
51. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
52. Shuying Liu and Weihong Deng. Very deep convolutional neural network based image classification using small training sample size. 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pages 730-734, 2015.
53. François Chollet et al. Keras. https://github.com/fchollet/keras, 2015.
54. Kamalika Chaudhuri and Sanjoy Dasgupta. Rates of convergence for the cluster tree. In Advances in Neural Information Processing Systems, pages 343-351, 2010.
55. Luc Devroye, Laszlo Gyorfi, Adam Krzyzak, and Gábor Lugosi. On the strong universal consistency of nearest neighbor regression function estimates. The Annals of Statistics, pages 1371-1385, 1994.
56. Sanjoy Dasgupta and Samory Kpotufe. Optimal rates for k-NN density and mode estimation. In Advances in Neural Information Processing Systems, pages 2555-2563, 2014.
57. Frédéric Chazal. An upper bound for the volume of geodesic balls in submanifolds of Euclidean spaces. https://geometrica.saclay.inria.fr/team/Fred.Chazal/BallVolumeJan2013.pdf, 2013.
58. Heinrich Jiang. Uniform convergence rates for kernel density estimation. In International Conference on Machine Learning, pages 1694-1703, 2017.