Learning with differential privacy: Stability, learnability and the sufficiency and necessity of ERM principle

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages

  Wang, Yu Xiang ^a,b   Lei, Jing ^b   Fienberg, Stephen E ^a,b  

^a Carnegie Mellon University ^*  (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Characterization; Differential privacy; Learnability; Privacy preserving machine learning; Stability

Indexed keywords

ARTIFICIAL INTELLIGENCE; CHARACTERIZATION; CONVERGENCE OF NUMERICAL METHODS; DATA PRIVACY; LEARNING SYSTEMS;

CONSISTENT ALGORITHM; DIFFERENTIAL PRIVACIES; GENERAL LEARNING; LEARNABILITY; LEARNING ACCURACY; PRIVACY CONSTRAINTS; PRIVACY PRESERVING; REAL-LIFE PROBLEMS;

LEARNING ALGORITHMS;

EID: 84995390313     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (39)

1. David Applegate and Ravi Kannan. Sampling and integration of near log-concave functions. In ACM Symposium on Theory of Computing (STOC-91), pages 156-163, 1991.
2. Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization, revisited. arXiv preprint arXiv:1405.7085, 2014.
3. Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. arXiv preprint arXiv:1511.02513, 2015.
4. Amos Beimel, Kobbi Nissim, and Uri Stemmer. Characterizing the sample complexity of private learners. In Conference on Innovations in Theoretical Computer Science (ITCS-13), pages 97-110. ACM, 2013a.
5. Amos Beimel, Kobbi Nissim, and Uri Stemmer. Private learning and sanitization: Pure vs. approximate differential privacy. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, pages 363-378. Springer, 2013b.
6. Amos Beimel, Hai Brenner, Shiva Prasad Kasiviswanathan, and Kobbi Nissim. Bounds on the sample complexity for private learning and private data release. Machine learning, 94 (3):401-437, 2014.
7. Avrim Blum and Moritz Hardt. The ladder: A reliable leaderboard for machine learning competitions. In International Conference on Machine Learning (ICML-15), pages 1006-1014, 2015.
8. Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.
9. Mark Bun, Kobbi Nissim, Uri Stemmer, and Salil Vadhan. Differentially private release and learning of threshold functions. In IEEE Symposium on Foundations of Computer Science (FOCS-15), pages 634-649. IEEE, 2015.
10. Mark Mar Bun. New Separations in the Complexity of Differential Privacy. PhD thesis, Harvard University Cambridge, Massachusetts, 2016.
11. Kamalika Chaudhuri and Daniel Hsu. Sample complexity bounds for differentially private learning. In Conference on Learning Theory (COLT-11), volume 19, pages 155-186, 2011.
12. Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12:1069-1109, 2011.
13. Cynthia Dwork. Differential privacy. In Automata, Languages and Programming, pages 1-12. Springer, 2006.
14. Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In ACM Symposium on Theory of Computing (STOC-09), pages 371-380. ACM, 2009.
15. Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology-EUROCRYPT 2006, pages 486-503. Springer, 2006a.
16. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography, pages 265-284. Springer, 2006b.
17. Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. The reusable holdout: Preserving validity in adaptive data analysis. Science, 349 (6248):636-638, 2015a.
18. Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Leon Roth. Preserving statistical validity in adaptive data analysis. In ACM Symposium on Theory of Computing (STOC-15), pages 117-126. ACM, 2015b.
19. Moritz Hardt and Jonathan Ullman. Preventing false discovery in interactive data analysis is hard. In IEEE Symposium on Foundations of Computer Science (FOCS-14), pages 454-463. IEEE, 2014.
20. David Haussler. Decision theoretic generalizations of the pac model for neural net and other learning applications. Information and computation, 100(1):78-150, 1992.
21. Prateek Jain and Abhradeep Thakurta. Differentially private learning with kernels. In International Conference on Machine Learning (ICML-13), pages 118-126, 2013.
22. Shiva Prasad Kasiviswanathan, Homin K Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately? SIAM Journal on Computing, 40(3):793-826, 2011.
23. Michael Kearns and Dana Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Neural Computation, 11(6):1427-1453, 1999.
24. Michael J Kearns, Robert E Schapire, and Linda M Sellie. Toward efficient agnostic learning. In Workshop on Computational learning theory (COLT-92), pages 341-352. ACM, 1992.
25. Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression. Journal of Machine Learning Research, 1:41, 2012.
26. Jing Lei. Differentially private m-estimators. In Advances in Neural Information Processing Systems (NIPS-11), pages 361-369, 2011.
27. Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In IEEE Symposium on Foundations of Computer Science, 2007 (FOCS-07), pages 94-103, 2007.
28. Sayan Mukherjee, Partha Niyogi, Tomaso Poggio, and Ryan Rifkin. Learning theory: Stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Mathematics, 25(1-3):161-193, 2006.
29. Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems (NIPS-07), pages 1177-1184, 2007.
30. Robert E Schapire. The strength of weak learnability. Machine Learning, 5(2):197-227, 1990.
31. Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research, 11:2635-2670, 2010.
32. Adam Smith. Privacy-preserving statistical estimation with optimal convergence rates. In ACM Symposium on Theory of Computing (STOC-11), pages 813-822, 2011.
33. Abhradeep Guha Thakurta and Adam Smith. Differentially private feature selection via stability arguments, and the robustness of the lasso. In Conference on Learning Theory (COLT-13), pages 819-850, 2013.
34. Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134-1142, 1984.
35. Vladimir Vapnik. Statistical Learning Theory. Wiley, 1998.
36. Vladimir N Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
37. Yu-Xiang Wang, Stephen E Fienberg, and Alex Smola. Privacy for free: Posterior sampling and stochastic gradient monte carlo. In International Conference on Machine Learning (ICML-15), 2015.
38. Larry Wasserman and Shuheng Zhou. A statistical framework for differential privacy. Journal of the American Statistical Association, 105(489):375-389, 2010.
39. Fei Yu, Stephen E Fienberg, Aleksandra B Slavković, and Caroline Uhler. Scalable privacy-preserving data sharing methodology for genome-wide association studies. Journal of biomedical informatics, 50:133-141, 2014.