Privacy for free: Posterior sampling and stochastic gradient Monte Carlo

32nd International Conference on Machine Learning, ICML 2015

Volumn 3, Issue , 2015, Pages 2483-2492

  Wang, Yu Xiang ^a   Fienberg, Stephen E ^a,b   Smola, Alexander J ^a,c  

^a Carnegie Mellon University ^*  (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

^c Marianas Labs   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; MONTE CARLO METHODS; STOCHASTIC SYSTEMS;

ALGORITHMIC PROCEDURE; DIFFERENTIAL PRIVACIES; HYBRID MONTE CARLO; POSTERIOR DISTRIBUTIONS; PRIVACY CONSTRAINTS; STANDARD ASSUMPTIONS; STATISTICAL ESTIMATORS; STOCHASTIC GRADIENT;

LEARNING SYSTEMS;

EID: 84969939834     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (49)

1. Ahn, Sungjin, Korattikara, Anoop, and Welling, Max. Bayesian posterior sampling via stochastic gradient fisher scoring. In Proceedings of the 29th International Conference on Machine Learning (ICML-12), 2012.
2. Airoldi, Edoardo M, Blei, David M, Fienberg, Stephen E, and Xing, Eric P. Mixed membership stochastic block-models. In Advances in Neural Information Processing Systems, pp. 33-40, 2009.
3. Applegate, David and Kannan, Ravi. Sampling and integration of near log-concave functions. In Proceedings of the twenty-third annual ACM symposium on Theory of computing, pp. 156-163. ACM, 1991.
4. Bassily, Raef, Smith, Adam, and Thakurta, Abhradeep. Private empirical risk minimization, revisited. arXiv preprint arXiv: 1405.7085, 2014.
5. Beimel, Amos, Brenner, Hai, Kasiviswanathan, Shiva Prasad, and Nissim, Kobbi. Bounds on the sample complexity for private learning and private data release. Machine learning, 94(3):401-437, 2014.
6. Bialek, William, Nemenman, Ilya, and Tishby, Naftali. Predictability, complexity, and learning. Neural computation, 13(11):2409-2463, 2001.
7. Blei, David M, Ng, Andrew Y, and Jordan, Michael I. Latent dirichlet allocation, the Journal of machine Learning research, 3:993-1022, 2003.
8. Chaudhuri, Kamalika, Monteleoni, Claire, and Sarwate, Anand D. Differentially private empirical risk minimization. The Journal of Machine Learning Research, 12: 1069-1109, 2011.
9. Chen, Tianqi, Fox, Emily B., and Guestrin, Carlos. Stochastic Gradient Hamiltonian Monte Carlo. In Proceeding of 31th International Conference on Machine Learning (ICML'14), 2014.
10. De Blasi, Pierpaolo and Walker, Stephen G. Bayesian asymptotics with misspecified models. Statistica Sinica, 23:169-187, 2013.
11. Dimitrakakis, Christos, Nelson, Blaine, Mitrokotsa, Aikaterini, and Rubinstein, Benjamin IP. Robust and private bayesian inference. In Algorithmic Learning Theory, pp. 291-305. Springer, 2014.
12. Ding, Nan, Fang, Youhan, Babbush, Ryan, Chen, Changyou, Skeel, Robert D, and Neven, Hartmut. Bayesian sampling using stochastic gradient thermostats. In Advances in Neural Information Processing Systems, pp. 3203-3211, 2014.
13. Dwork, Cynthia. Differential privacy. In Proceedings of the 33rd international conference on Automata, Languages and Programming-Volume Part II, pp. 1-12. Springer-Verlag, 2006.
14. Dwork, Cynthia and Lei, Jing. Differential privacy and robust statistics. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pp. 371-380. ACM, 2009.
15. Dwork, Cynthia and Roth, Aaron. The algorithmic foundations of differential privacy. Theoretical Computer Science, 9(3-4):211-407, 2013.
16. Dwork, Cynthia, McSherry, Frank, Nissim, Kobbi, and Smith, Adam. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography, pp. 265-284. Springer, 2006.
17. Fei-Fei, Li and Perona, Pietro. A bayesian hierarchical model for learning natural scene categories. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 2, pp. 524-531. IEEE, 2005.
18. Gelman, Andrew, Carlin, John B, and Stern, Hal S. Bayesian data analysis, volume 2. Taylor & Francis, 2014.
19. Geman, Stuart and Geman, Donald. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. Pattern Analysis and Machine Intelligence, IEEE Transactions on, (6):721-741, 1984.
20. Ghosal, Subhashis. The Dirichlet process, related priors and posterior asymptotics, volume 2. Chapter, 2010.
21. Girolami, Mark and Calderhead, Ben. Riemann manifold langevin and hamiltonian monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(2): 123-214, 2011.
22. Hastings, W Keith. Monte carlo sampling methods using markov chains and their applications. Biometrika, 57(1): 97-109, 1970.
23. Hofmann, Thomas, Schölkopf, Bernhard, and Smola, Alexander J. Kernel methods in machine learning. The annals of statistics, pp. 1171-1220, 2008.
24. Kasiviswanathan, Shiva P and Smith, Adam. On the 'semantics' of differential privacy: A bayesian formulation. Journal of Privacy and Confidentiality, 6(1): 1,2014.
25. Kifer, Daniel, Smith, Adam, and Thakurta, Abhradeep. Private convex empirical risk minimization and high-dimensional regression. Journal of Machine Learning Research, 1:41, 2012.
26. Kleijn, BJK, van der Vaart, AW, et al. The bernstein-von-mises theorem under misspecification. Electronic Journal of Statistics, 6:354-381, 2012.
27. Le Cam, Lucien Marie. On the Bernstein-von Mises theorem. Department of Statistics, University of California, 1986.
28. Lehmann, Erich L and Romano, Joseph P. Testing statistical hypotheses. Springer Science & Business Media, 2006.
29. McSherry, Frank and Talwar, Kunal. Mechanism design via differential privacy. In Foundations of Computer Science, 2007. FOCS'07. 48th Annual IEEE Symposium on, pp. 94-103. IEEE, 2007.
30. Mir, Darakhshan J. Differential privacy: an exploration of the privacy-utility landscape. PhD thesis, Rutgers University, 2013.
31. Neal, Radford. Mcmc using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2, 2011.
32. Orbanz, Peter. Lecture notes on bayesian nonparametrics. Journal of Mathematical Psychology, 56:1-12, 2012.
33. Penny, William D, Friston, Karl J, Ashburner, John T, Kiebel, Stefan J, and Nichols, Thomas E. Statistical parametric mapping: the analysis of functional brain images: the analysis of functional brain images. Academic press, 2011.
34. Propp, James and Wilson, David. Coupling from the past: a users guide. Microsurveys in Discrete Probability, 41: 181-192, 1998.
35. Rabiner, Lawrence. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257-286, 1989.
36. Rajkumar, Arun and Agarwal, Shivani. A differentially private stochastic gradient descent algorithm for multiparty classification. In International Conference on Artificial Intelligence and Statistics, pp. 933-941, 2012.
37. Robbins, Herbert and Monro, Sutton. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
38. Rosenthal, Jeffrey S. Minorization conditions and convergence rates for markov chain monte carlo. Journal of the American Statistical Association, 90(430):558-566, 1995.
39. Sato, Issei and Nakagawa, Hiroshi. Approximation analysis of stochastic gradient langevin dynamics by using fokker-planck equation and ito process. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 982-990, 2014.
40. Smith, Adam. Efficient, differentially private point estimators. arXiv preprint arXiv:0809.4794, 2008.
41. Song, Shuang, Chaudhuri, Kamalika, and Sarwate, Anand D. Stochastic gradient descent with differentially private updates. In IEEE Global Conference on Signal and Information Processing, 2013.
42. Sontag, David and Roy, Dan. Complexity of inference in latent dirichlet allocation. In Advances in neural information processing systems, pp. 1008-1016, 2011.
43. Van der Vaart, Aad W. Asymptotic statistics, volume 3. Cambridge university press, 2000.
44. Vollmer, Sebastian J, Zygalakis, Konstantinos C, et al. (non-) asymptotic properties of stochastic gradient langevin dynamics. arXiv preprint arXiv:1501.00438, 2015.
45. Wainwright, Martin J and Jordan, Michael I. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning, 1(1-2): 1-305, 2008.
46. Wang, Yu-Xiang, Fienberg, Stephen E, and Smola, Alex. Privacy for free: Posterior sampling and stochastic gradient monte carlo. arXiv preprint arXiv: 1502.07645, 2015.
47. Welling, Max and Teh, Yee W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681-688, 2011.
48. Williams, Oliver and McSherry, Frank. Probabilistic inference and differential privacy. In Advances in Neural Information Processing Systems, pp. 2451-2459, 2010.
49. Xiao, Yonghui and Xiong, Li. Bayesian inference under differential privacy. arXiv preprint arXiv: 1203.0617, 2012.