Generalization in adaptive data analysis and holdout reuse

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 2350-2358

  Dwork, Cynthia ^a   Pitassi, Toniann ^b   Feldman, Vitaly ^c   Reingold, Omer ^d   Hardt, Moritz ^e   Roth, Aaron ^f  

^a MICROSOFT RESEARCH   (United States)

^b UNIVERSITY OF TORONTO   (Canada)

^c IBM ALMADEN RESEARCH CENTER   (United States)

^d SAMSUNG RESEARCH AMERICA   (United States)

^e GOOGLE INC   (United States)

^f UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; DATA HANDLING; INFORMATION SCIENCE;

ADAPTIVE PROCESS; ADAPTIVE SETTING; DIFFERENTIAL PRIVACIES; FORMAL APPROACH; PRACTICAL METHOD; STATISTICAL INFERENCE; STATISTICAL VALIDITY; SYNTHETIC EXPERIMENTS;

INFORMATION ANALYSIS;

EID: 84965181547     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. Raef Bassily, Adam Smith, Thomas Steinke, and Jonathan Ullman. More general queries and less generalization error in adaptive data analysis. CoRR, abs/1503.04843, 2015.
2. Avrim Blum and Moritz Hardt. The ladder: A reliable leaderboard for machine learning competitions. CoRR, abs/1502.04585, 2015.
3. Olivier Bousquet and André Elisseeff. Stability and generalization. JMLR, 2:499-526, 2002.
4. Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11:2079-2107, 2010.
5. Chuong B. Do, Chuan-Sheng Foo, and Andrew Y. Ng. Efficient multiple hyperparameter learning for log-linear models. In NIPS, pages 377-384, 2007.
6. Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. CoRR, abs/1506.
7. Extended abstract to appear in NIPS 2015.
8. Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. Preserving statistical validity in adaptive data analysis. CoRR, abs/1411.2664, 2014.
9. Extended abstract in STOC 2015.
10. Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, pages 486-503, 2006.
11. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography, pages 265-284. Springer, 2006.
12. David A. Freedman. A note on screening regression equations. The American Statistician, 37(2):152-155, 1983.
13. Moritz Hardt and Jonathan Ullman. Preventing false discovery in interactive data analysis is hard. In FOCS, pages 454-463, 2014.
14. Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer series in statistics. Springer, 2009.
15. John Langford. Clever methods of overfitting. http://hunch.net/?p=22, 2005.
16. Kobbi Nissim and Uri Stemmer. On the generalization properties of differential privacy. CoRR, abs/1504.05800, 2015.
17. Tomaso Poggio, Ryan Rifkin, Sayan Mukherjee, and Partha Niyogi. General conditions for predictivity in learning theory. Nature, 428(6981):419-422, 2004.
18. R. Bharat Rao and Glenn Fung. On the dangers of cross-validation. an experimental evaluation. In International Conference on Data Mining, pages 588-596. SIAM, 2008.
19. Juha Reunanen. Overfitting in making comparisons between variable selection methods. Journal of Machine Learning Research, 3:1371-1382, 2003.
20. Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. The Journal of Machine Learning Research, 11:2635-2670, 2010.
21. Thomas Steinke and Jonathan Ullman. Interactive fingerprinting codes and the hardness of preventing false discovery. arXiv preprint arXiv:1410.1228, 2014.
22. Jonathan Taylor and Robert J. Tibshirani. Statistical learning and selective inference. Proceedings of the National Academy of Sciences, 112(25):7629-7634, 2015.
23. Yu-Xiang Wang, Jing Lei, and Stephen E. Fienberg. Learning with differential privacy: Stability, learnability and the sufficiency and necessity of ERM principle. CoRR, abs/1502.06309, 2015.