Algorithmic stability for adaptive data analysis?

Proceedings of the Annual ACM Symposium on Theory of Computing

Volumn 19-21-June-2016, Issue , 2016, Pages 1046-1059

  Bassily, Raef ^a   Steinke, Thomas ^b   Nissim, Kobbi ^b,c   Stemmer, Uri ^c   Smith, Adam ^d   Ullman, Jonathan ^e  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b HARVARD UNIVERSITY   (United States)

^c BEN GURION UNIVERSITY OF THE NEGEV   (Israel)

^d PENNSYLVANIA STATE UNIVERSITY   (United States)

^e NORTHEASTERN UNIVERSITY   (United States)

Author keywords

Adaptivity; Data analysis; Differential privacy; Generalization; Stability; Statistics

Indexed keywords

ALGORITHMS; ALUMINUM; COMPUTATION THEORY; CONVERGENCE OF NUMERICAL METHODS; DATA HANDLING; DATA REDUCTION; INFORMATION ANALYSIS; OPTIMIZATION; STABILITY; STATISTICS;

ADAPTIVITY; ALGORITHMIC STABILITY; DIFFERENTIAL PRIVACIES; GENERALIZATION; GENERALIZATION ERROR; INDEPENDENT SAMPLES; STATISTICAL VALIDITY; UPPER AND LOWER BOUNDS;

QUERY PROCESSING;

EID: 84979223173     PISSN: 07378017     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2897518.2897566     Document Type: Conference Paper

References (34)

1. Olivier Bousquet and André Elisseeff. Stability and generalization. JMLR, 2:499-526, 2002.
2. Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289-300, 1995.
3. Avrim Blum and Moritz Hardt. The ladder: A reliable leaderboard for machine learning competitions. CoRR, abs/1502.04585, 2015.
4. Carlo Emilio Bonferroni. Teoria statistica delle classi e calcolo delle probabilita. Pubbl. d. R. Ist. Super. di Sci. Econom. e Commerciali di Firenze., 8, 1936.
5. Raef Bassily, Adam Smith, Thomas Steinke, and Jonathan Ullman. More general queries and less generalization error in adaptive data analysis. CoRR, abs/1503.04843, 2015.
6. Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In FOCS, pages 464-473, 2014.
7. Mark Bun, Jonathan Ullman, and Salil P. Vadhan. Fingerprinting codes and the price of approximate differential privacy. In STOC, pages 1-10. ACM, May 31 - June 3 2014.
8. Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. In NIPS, 2015.
9. Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. Preserving statistical validity in adaptive data analysis. In STOC, 2015.
10. Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248):636-638, June 2015.
11. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265-284. Springer, March 4-7 2006.
12. Irit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In PODS, pages 202-210. ACM, June 9-12 2003.
13. Cynthia Dwork, Guy N. Rothblum, and Salil P. Vadhan. Boosting and differential privacy. In FOCS, pages 51-60, 2010.
14. Olive Jean Dunn. Multiple comparisons among means. JASA, 56:52-64, 1961.
15. Luc Devroye and Terry J. Wagner. Distribution-free inequalities for the deleted and holdout error estimates. IEEE Transactions on Information Theory, 25(2):202-207, 1979.
16. Luc Devroye and Terry J. Wagner. Distribution-free performance bounds for potential function rules. IEEE Transactions on Information Theory, 25(5):601-604, 1979.
17. Cynthia Dwork. Differential privacy. In ICALP, pages 1-12, 2006.
18. Andrew Gelman and Eric Loken. The statistical crisis in science. American Scientist, 102(6):460, 2014.
19. Moritz Hardt and Guy Rothblum. A multiplicative weights mechanism for privacy-preserving data analysis. In FOCS, pages 61-70. IEEE, 2010.
20. Moritz Hardt and Jonathan Ullman. Preventing false discovery in interactive data analysis is hard. In FOCS, 2014.
21. John P. A. Ioannidis. Why most published research findings are false. PLoS Medicine, 2(8):124, August 2005.
22. Michael J. Kearns. Efficient noise-tolerant learning from statistical queries. In STOC, pages 392-401. ACM, May 16-18 1993.
23. Michael J. Kearns and Dana Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Neural Computation, 11(6):1427-1453, 1999.
24. Frank McSherry. Differential privacy for measure concentration. Blog post. February 4, 2014. http://windowsontheory.org/.
25. Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In FOCS, pages 94-103. IEEE, Oct. 20-23 2007.
26. Kobbi Nissim and Uri Stemmer. On the generalization properties of differential privacy. CoRR, abs/1504.05800, 2015.
27. Aaron Roth and Tim Roughgarden. Interactive privacy via the median mechanism. In STOC, pages 765-774. ACM, June 5-8 2010.
28. Daniel Russo and James Zou. Controlling bias in adaptive data analysis using information theory. In AISTATS, 2016.
29. Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. JMLR, 11:2635-2670, 2010.
30. Thomas Steinke and Jonathan Ullman. Between pure and approximate differential privacy. CoRR, abs/1501.06095, 2015.
31. Thomas Steinke and Jonathan Ullman. Interactive fingerprinting codes and the hardness of preventing false discovery. In COLT, pages 1588-1628, 2015.
32. 2+o (1) counting queries with differential privacy is hard. In STOC, pages 361-370. ACM, June 1-4 2013.
33. Jonathan Ullman. Private multiplicative weights beyond linear queries. In PODS. ACM, May 31-June 4 2015.
34. Yu-Xiang Wang, Jing Lei, and Stephen E. Fienberg. A minimax theory for adaptive data analysis. arXiv:1602.04287 [stat. ML], 2016.