Efficient private empirical risk minimization for high-dimensional learning

33rd International Conference on Machine Learning, ICML 2016

Volumn 1, Issue , 2016, Pages 750-761

  Kasiviswanathan, Shiva Prasad ^a   Jin, Hongxia ^a  

^a SAMSUNG RESEARCH AMERICA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CLUSTERING ALGORITHMS; DATA PRIVACY; LEARNING SYSTEMS;

COMPUTATIONAL SAVINGS; DIFFERENTIAL PRIVACIES; DIMENSIONALITY REDUCTION; EMPIRICAL RISK MINIMIZATION; HIGH DIMENSIONAL DATA; HIGH-DIMENSIONAL LEARNING; RANDOM PROJECTIONS; THEORETICAL GUARANTEES;

C (PROGRAMMING LANGUAGE);

EID: 84997824414     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (48)

1. Arriaga, Rosa I and Vempala, Santosh. An algorithmic theory of learning: Robust concepts and random projection. Machine Learning, 63(2):161-182, 2006.
2. Bassily, Raef, Smith, Adam, and Thakurta, Abhradeep. Differentially private empirical risk minimization: Efficient algorithms and tight error bounds. In FOCS. IEEE, 2014.
3. Blocki, Jeremiah, Blum, Avrim, Datta, Anupam, and Shef-fet, Or. The johnson-lindenstrauss transform itself preserves differential privacy. In Foundations of Computer Science (FOCS), 2012 IEEE 53rd Annual Symposium on, pp. 410-419. IEEE, 2012.
4. Bourgain, Jean, Sjoerd, Dirksen, and Nelson, Jelani. Toward a unified theory of sparse dimensionality reduction in euclidean space. In Proceedings of the 47th ACM Symposium on Theory of Computing. Association for Computing Machinery, 2015.
5. Boyd, Stephen and Vandenberghe, Lieven. Convex optimization. Cambridge university press, 2004.
6. Calderbank, Robert, Jafarpour, Sina, and Schapire, Robert. Compressed learning: Universal sparse dimensionality reduction and learning in the measurement domain. preprint, 2009.
7. Chaudhuri, Kamalika and Monteleoni, Claire. Privacy-preserving logistic regression. In Advances in Neural Information Processing Systems, pp. 289-296, 2009.
8. Chaudhuri, Kamalika, Monteleoni, Claire, and Sarwate, Anand D. Differentially private empirical risk minimization. The Journal of Machine Learning Research, 12:1069-1109, 2011.
9. Davenport, Mark A, Wakin, Michael B, and Baraniuk, Richard G. Detection and estimation with compressive measurements. Dept. of ECE, Rice University, Tech. Rep, 2006.
10. Dirksen, Sjoerd. Dimensionality reduction with sub-gaussian matrices: a unified theory. arXiv preprint arXiv: 1402.3973, 2014.
11. Duchi, John C, Jordan, Michael, Wainwright, Martin J, et al. Local privacy and statistical minimax rates. In Foundations of Computer Science (FOCS), 2013 IEEE 54th Annual Symposium on, pp. 429-438. IEEE, 2013.
12. Duncan, George T, Pearson, Robert W, et al. Enhancing access to microdata while protecting confidentiality: Prospects for the future. Statistical Science, 6(3):219-232, 1991.
13. Dwork, Cynthia and Roth, Aaron. The algorithmic foundations of differential privacy. Theoretical Computer Science, 9(3-4):211-407, 2013.
14. Dwork, Cynthia, Kenthapadi, Krishnaram, McSherry, Frank, Mironov, Ilya, and Naor, Moni. Our data, ourselves: Privacy via distributed noise generation. In EU-ROCRYPT, LNCS, pp. 486-503. Springer, 2006a.
15. Dwork, Cynthia, McSherry, Frank, Nissim, Kobbi, and Smith, Adam. Calibrating noise to sensitivity in private data analysis. In TCC, volume 3876 of LNCS, pp. 265284. Springer, 2006b.
16. Eldar, Yonina C and Kutyniok, Gitta. Compressed sensing: theory and applications. Cambridge University Press, 2012.
17. Fard, Mahdi Milani, Grinberg, Yuri, Pineau, Joelle, and Precup, Doina. Compressed least-squares regression on sparse spaces. In AAAI, 2012.
18. n. Springer, 1988.
19. Jain, Prateek and Thakurta, Abhradeep. Differentially private learning with kernels. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 118-126, 2013.
20. Jain, Prateek and Thakurta, Abhradeep Guha. (near) dimension independent risk bounds for differentially private learning. In Proceedings of The 31st International Conference on Machine Learning, pp. 476-484, 2014.
21. Jain, Prateek, Kothari, Pravesh, and Thakurta, Abhradeep. Differentially private online learning. In COLT 2012, pp. 24.1-24.34, 2012.
22. Kabán, Ata. New bounds on compressive linear least squares regression. In The 17-th International Conference on Artificial Intelligence and Statistics (AISTATS 2014), volume 33, pp. 448-456, 2014.
23. Kasiviswanathan, Shiva, Nissim, Kobbi, and Jin, Hongxia. Private incremental regression, 2016.
24. Kenthapadi, Krishnaram, Korolova, Aleksandra, Mironov, Ilya, and Mishra, Nina. Privacy via the johnson-lindenstrauss transform. Journal of Privacy and Confidentiality, 5(1):39-71, 2013.
25. Kifer, Daniel, Smith, Adam, and Thakurta, Abhradeep. Private convex empirical risk minimization and highdimensional regression. Journal of Machine Learning Research, 1:41, 2012.
26. Ledoux, Michel and Talagrand, Michel. Probability in Ba-nach Spaces: isoperimetry and processes, volume 23. Springer Science and Business Media, 2013.
27. Mahoney, Michael W. Randomized algorithms for matrices and data. Foundations and Trends in Machine Learning, 3(2):123-224, 2011.
28. Maillard, Odalric and Munos, Rémi. Compressed least-squares regression. In Advances in Neural Information Processing Systems, pp. 1213-1221, 2009.
29. McSherry, Frank and Talwar, Kunal. Mechanism design via differential privacy. In FOCS, pp. 94-103. IEEE, 2007.
30. Mendelson, Shahar. Geometric parameters in learning theory. In Geometric aspects of functional analysis, pp. 193-235. Springer, 2004.
31. Mendelson, Shahar, Pajor, Alain, and Tomczak-Jaegermann, Nicole. Reconstruction and subgaussian operators in asymptotic geometric analysis. Geometric and Functional Analysis, 17(4):1248-1282, 2007.
32. Mishra, Nikita and Thakurta, Abhradeep. (nearly) optimal differentially private stochastic multi-arm bandits. In UAI, pp. 592-601, 2015.
33. Rubinstein, Benjamin IP, Bartlett, Peter L, Huang, Ling, and Taft, Nina. Learning in a large function space: Privacy-preserving mechanisms for svm learning. arXiv preprint arXiv:0911.5708, 2009.
34. Sarwate, Anand D and Chaudhuri, Kamalika. Signal processing and machine learning with differential privacy: Algorithms and challenges for continuous data. Signal Processing Magazine, IEEE, 30(5):86-94, 2013.
35. Shalev-Shwartz, Shai, Shamir, Ohad, Srebro, Nathan, and Sridharan, Karthik. Stochastic convex optimization. In COLT, 2009.
36. Sheffet, Or. Private approximations of the 2nd-moment matrix using existing techniques in linear regression. arXiv preprint arXiv: 1507.00056, 2015.
37. Smith, Adam and Thakurta, Abhradeep Guha. Differentially private feature selection via stability arguments, and the robustness of the lasso. In Conference on Learning Theory, pp. 819-850, 2013.
38. Talwar, Kunal, Thakurta, Abhradeep, and Zhang, Li. Private empirical risk minimization beyond the worst case: The effect of the constraint set geometry. arXiv preprint arXiv: 1411.5417, 2015a.
39. Talwar, Kunal, Thakurta, Abhradeep, and 31Zhang, Li. Nearly optimal private lasso. In Advances in Neural Information Processing Systems, pp. 3007-3015, 2015b.
40. Thakurta, Abhradeep. Personal communication, 2016.
41. Thakurta, Abhradeep Guha and Smith, Adam, (nearly) optimal algorithms for private online learning in full-information and bandit settings. In Advances in Neural Information Processing Systems, pp. 2733-2741, 2013.
42. Ullman, Jonathan. Private multiplicative weights beyond linear queries. In Proceedings of the 34th ACM Symposium on Principles of Database Systems, pp. 303-312. ACM, 2015.
43. Vempala, Santosh S. The random projection method, volume 65. American Mathematical Soc., 2005.
44. Vershynin, Roman. Estimation in high dimensions: a geometric perspective. arXiv preprint arXiv:1405.5103, 2014.
45. Wang, Yining, Wang, Yu-Xiang, and Singh, Aarti. A deterministic analysis of noisy sparse subspace clustering for dimensionality-reduced data. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1422-1431, 2015.
46. Wright, John, Yang, Allen Y, Ganesh, Arvind, Sastry, Shankar S, and Ma, Yi. Robust face recognition via sparse representation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 31(2):210-227, 2009.
47. Zhou, Shuheng, Lafferty, John, and Wasserman, Larry. Compressed and privacy-sensitive sparse regression. Information Theory, IEEE Transactions on, 55(2):846-866, 2009a.
48. Zhou, Shuheng, Ligett, Katrina, and Wasserman, Larry. Differential privacy with compression. In Information Theory, 2009. ISIT 2009. IEEE International Symposium on, pp. 2718-2722. IEEE, 2009b.