Statistical significance of the netflix challenge

Statistical Science

Volumn 27, Issue 2, 2012, Pages 202-231

  Feuerverger, Andrey ^a   He, Yu ^b   Khatri, Shashi ^c  

^a UNIVERSITY OF TORONTO   (Canada)

^b NANJING UNIVERSITY   (China)

^c Speak Empirics   (Canada)

Author keywords

Collaborative filtering; Cross validation; Effective number of degrees of freedom; Empirical bayes; Ensemble methods; Gradient descent; Latent factors; Nearest neighbors; Netflix contest; Neural networks; Penalization; Prediction error; Recommender systems; Restricted Boltzmann machines; Shrinkage; Singular value decomposition

Indexed keywords

EID: 84871531231     PISSN: 08834237     EISSN: None     Source Type: Journal    
DOI: 10.1214/11-STS368     Document Type: Article

References (130)

1. ACM SIGKDD (2007). KDD Cup and Workshop 2007. Available at www.cs.uic.edu/~liub/Netflix-KDD-Cup-2007.html.
2. ADOMAVICIUS, G. and TUZHILIN, A. (2005). Towards the next generation of recommender systems: A survey of the state-ofthe-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering 17 634-749.
3. BARBIERI, M. M. and BERGER, J. O. (2004). Optimal predictive model selection. Ann. Statist. 32 870-897.
4. BARON, A. (1984). Predicted squared error: A criterion for automatic model selection. In Self-Organizing Methods in Modeling (S. Farrow, ed.). Marcel Dekker, New York.
5. BELL, R. and KOREN, Y. (2007). Lessons from the Netflix Prize challenge. ACM SIGKDD Explorations Newsletter 9 75-79.
6. BELL, R. and KOREN, Y. (2007). Improved neighborhood-based collaborative filtering. In Proc. KDD Cup and Workshop 2007 7-14. ACM, New York.
7. BELL, R. and KOREN, Y. (2007). Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proc. Seventh IEEE Int. Conf. on Data Mining 43-52. IEEE Computer Society, Los Alamitos, CA.
8. BELL, R., KOREN, Y. and VOLINSKY, C. (2007). Modeling relationships at multiple scales to improve accuracy of large recommender systems. In Proc. 13th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining 95-104. ACM, New York.
9. BELL, R., KOREN, Y. and VOLINSKY, C. (2007). The BellKor solution to the Netflix Prize. Available at http://www.research.att.com/~volinsky/netflix/ ProgressPrizes2007BellKorSolution.pdf.
10. BELL, R., KOREN, Y. and VOLINSKY, C. (2007). Chasing $1,000,000: How we won the Netflix Progress Prize. ASA Statistical and Computing Graphics Newsletter 18 4-12.
11. BELL, R., KOREN, Y. and VOLINSKY, C. (2008). The BellKor 2008 solution to the Netflix Prize. Available at http://www. netflixprize.com/assets/ProgressPrize2008_BellKor.pdf.
12. BELL, R. M., BENNETT, J., KOREN, Y. and VOLINSKY, C. (2009). The million dollar programming prize. IEEE Spectrum 46 28-33.
13. BENNETT, J. and LANNING, S. (2007). The Netflix Prize. In Proc. KDD Cup and Workshop 2007 3-6. ACM, New York.
14. BERGER, J. (1982). Bayesian robustness and the Stein effect. J. Amer. Statist. Assoc. 77 358-368.
15. BISHOP, C. M. (1995). Neural Networks for Pattern Recognition. Clarendon Press, New York.
16. BISHOP, C. M. (2006). Pattern Recognition and Machine Learning. Springer, New York.
17. BREIMAN, L. (1996). Bagging predictors. Machine Learning 26 123-140.
18. BREIMAN, L. and FRIEDMAN, J. H. (1997). Predicting multivariate responses in multiple linear regression (with discussion). J. Roy. Statist. Soc. Ser. B 59 3-54.
19. BURGES, C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2 121-167.
20. CANDES, E. and PLAN, Y. (2009). Matrix completion with noise. Technical report, Caltech.
21. CANDES, E. and TAO, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313-2351.
22. CANNY, J. F. (2002). Collaborative filtering with privacy via factor analysis. In Proc. 25th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval 238-245. ACM, New York.
23. CARLIN, B. P. and LOUIS, T. A. (1996). Bayes and Empirical Bayes Methods for Data Analysis. Monogr. Statist. Appl. Probab. 69. Chapman & Hall, London.
24. CASELLA, G. (1985). An introduction to empirical Bayes data analysis. Amer. Statist. 39 83-87.
25. CHIEN, Y. H. and GEORGE, E. (1999). A Bayesian model for collaborative filtering. In Online Proc. 7th Int. Workshop on Artificial Intelligence and Statistics. Fort Lauderdale, FL.
26. CHRISTIANINI, N. and SHAWE-TAYLOR, J. (2000). An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge Univ. Press, Cambridge.
27. COHEN, W. W., SCHAPIRE, R. E. and SINGER, Y. (1999). Learning to order things. J. Artificial Intelligence Res. 10 243-270 (electronic).
28. COPAS, J. B. (1983). Regression, prediction and shrinkage. J. Roy. Statist. Soc. Ser. B 45 311-354.
29. DECOSTE, D. (2006). Collaborative prediction using ensembles of maximum margin matrix factorizations. In Proc. 23rd Int. Conf. on Machine Learning 249-256. ACM, New York.
30. DEERWESTER, S. C., DUMAIS, S. T., LANDAUER, T. K., FURNAS, G. W. and HARSHMAN, R. A. (1990). Indexing by latent semantic analysis. Journal of the Amercan Society of Information Science 41 391-407.
31. DEMPSTER, A. P., LAIRD, N. M. and RUBIN, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm (with discussion). J. Roy. Statist. Soc. Ser. B 39 1-38.
32. EFRON, B. (1975). Biased versus unbiased estimation. Advances in Math. 16 259-277.
33. EFRON, B. (1983). Estimating the error rate of a prediction rule: Improvement on cross-validation. J. Amer. Statist. Assoc. 78 316-331.
34. EFRON, B. (1986). How biased is the apparent error rate of a prediction rule. J. Amer. Statist. Assoc. 81 461-470.
35. EFRON, B. (1996). Empirical Bayes methods for combining likelihoods (with discussion). J. Amer. Statist. Assoc. 91 538-565.
36. EFRON, B. (2004). The estimation of prediction error: Covariance penalties and cross-validation (with discussion). J. Amer. Statist. Assoc. 99 619-642.
37. EFRON, B. andMORRIS, C. (1971). Limiting the risk of Bayes and empirical Bayes estimators. I. The Bayes case. J. Amer. Statist. Assoc. 66 807-815.
38. EFRON, B. and MORRIS, C. (1972). Limiting the risk of Bayes and empirical Bayes estimators. II. The empirical Bayes case. J. Amer. Statist. Assoc. 67 130-139.
39. EFRON, B. and MORRIS, C. (1972). Empirical Bayes on vector observations: An extension of Stein's method. Biometrika 59 335-347.
40. EFRON, B. and MORRIS, C. (1973). Stein's estimation rule and its competitors-an empirical Bayes approach. J. Amer. Statist. Assoc. 68 117-130.
41. EFRON, B. andMORRIS, C. (1973). Combining possibly related estimation problems (with discussion). J. Roy. Statist. Soc. Ser. B 35 379-421.
42. EFRON, B. and MORRIS, C. (1975). Data analysis using Stein's estimator and its generalization. J. Amer. Statist. Assoc. 70 311-319.
43. EFRON, B. and MORRIS, C. (1977). Stein's paradox in statistics. Scientific American 236 119-127.
44. EFRON, B., HASTIE, T., JOHNSTONE, I. and TIBSHIRANI, R. (2004). Least angle regression (with discussion). Ann. Statist. 32 407-499.
45. FAN, J. and LI, R. (2006). Statistical challenges with high dimensionality: Feature selection in knowledge discovery. In International Congress of Mathematicians III 595-622. Eur. Math. Soc., Zürich.
46. FRIEDMAN, J. (1994). An overview of predictive learning and function approximation. In From Statistics to Neural Networks (V. Cherkassky, J. Friedman and H. Wechsler, eds.). NATO ISI Series F 136. Springer, New York.
47. FUNK, S. (2006/2007). See Webb, B. (2006/2007).
48. GORRELL, G. and WEBB, B. (2006). Generalized Hebbian algorithm for incremental latent semantic analysis. Technical report, Linköping Univ., Sweden.
49. GREENSHTEIN, E. and RITOV, Y. (2004). Persistence in highdimensional linear predictor selection and the virtue of overparametrization. Bernoulli 10 971-988.
50. HASTIE, T., TIBSHIRANI, R. and FRIEDMAN, J. (2009). The Elements of Statistical Learning, 2nd ed. Springer, New York.
51. HERLOCKER, J. L., KONSTAN, J. A., BORCHERS, A. and RIEDL, J. (1999). An algorithmic framework for performing collaborative filtering. In Proc. 22nd ACM SIGIR Conf. on Information Retrieval 230-237.
52. HERLOCKER, J. L.,KONSTAN, J. A. andRIEDL, J. T. (2000). Explaining collaborative filtering recommendations. In Proc. 2000 ACM Conf. on Computer Supported Cooperative Work 241-250. ACM, New York.
53. HERLOCKER, J. L., KONSTAN, J. A., TERVEEN, L. G. and RIEDL, J. T. (2004). Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems 22 5-53.
54. HERTZ, J., KROGH, A. and PALMER, R. G. (1991). Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City, CA.
55. HILL, W., STEAD, L., ROSENSTEIN, M. and FURNAS, G. (1995). Recommending and evaluating choices in a virtual community of use. In Proc. SIGCHI Conf. on Human Factors in Computing Systems 194-201. ACM, New York.
56. HINTON, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Comput. 14 1771-1800.
57. HOFMANN, T. (2001). Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn. J 42 177-196.
58. HOFMANN, T. (2001). Learning what people (don't) want. In Proc. European Conf. on Machine Learning. Lect. Notes Comput. Sci. Eng. 2167 214-225. Springer, Berlin.
59. HOFMANN, T. (2004). Latent semantic models for collaborative filtering. ACM Transactions on Information Systems 22 89-115.
60. HOFMANN, T. and PUZICHA, J. (1999). Latent class models for collaborative filtering. In Proc. Int. Joint Conf. on Artificial Intelligence 2 688-693. Morgan Kaufmann, San Francisco, CA.
61. HU, Y., KOREN, Y. and VOLINSKY, C. (2008). Collaborative filtering for implicit feedback datasets. Technical report, AT&T Labs-Research, Florham Park, NJ.
62. IZENMAN, A. J. (2008). Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning. Springer, New York.
63. JAMES, W. and STEIN, C. (1961). Estimation with quadratic loss. In Proc. 4th Berkeley Sympos. Math. Statist. Probab. I 361-379. Univ. California Press, Berkeley, CA.
64. KIM, D. and YUM, B. (2005). Collaborative filtering based on iterative principal component analysis. Expert Systems with Applications 28 823-830.
65. KOREN, Y. (2008). Factorization meets the neighborhood: A multifaceted collaborative filtering model. In Proc. 14th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining 426-434. ACM, New York.
66. KOREN, Y. (2009). Collaborative filtering with temporal dynamics. In Proc. 15th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining 447-456. ACM, New York.
67. KOREN, Y. (2010). Factor in the neighbors: Scalable and accurate collaborative filtering. ACM Transactions on Knowledge Discovery from Data 4 Article 1.
68. KOREN, Y., BELL, R. and VOLINSKY, C. (2009). Matrix factorization techniques for recommender systems. Computer 42 (8) 30-37.
69. LI, K.-C. (1985). From Stein's unbiased risk estimates to the method of generalized cross validation. Ann. Statist. 13 1352-1377.
70. LIM, Y. J. and TEH, Y. W. (2007). Variational Bayesian approach to movie rating predictions. In Proc. KDD Cup and Workshop 2007 15-21. ACM, New York.
71. LITTLE, R. J. A. and RUBIN, D. B. (1987). Statistical Analysis with Missing Data. Wiley, New York. MR0890519 MALLOWS, C. (1973). Some comments on Cp. Technometrics 15 661-675.
72. MARITZ, J. S. and LWIN, T. (1989). Empirical Bayes Methods, 2nd ed. Monogr. Statist. Appl. Probab. 35. Chapman & Hall, London.
73. MARLIN, B. (2004). Collaborative filtering: A machine learning perspective. M.Sc. thesis, Computer Science Dept., Univ. Toronto.
74. MARLIN, B. and ZEMEL, R. S. (2004). The multiple multiplicative factor model for collaborative filtering. In Proc. 21st Int. Conf. on Machine Learning. ACM, New York.
75. MARLIN, B., ZEMEL, R. S.,ROWEIS, S. and SLANEY, M. (2007). Collaborative filtering and the missing at random assumption. In Proc. 23rd Conf. on Uncertainty in Artificial Intelligence. AMC, New York.
76. MOGUERZA, J. M. and MUÑOZ, A. (2006). Support vector machines with applications. Statist. Sci. 21 322-336.
77. MOODY, J. E. (1992). The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems. In Advances in Neural Information Processing Systems 4. Morgan Kaufmann, San Francisco, CA.
78. MORRIS, C. N. (1983). Parametric empirical Bayes inference: Theory and applications (with discussion). J. Amer. Statist. Assoc. 78 47-65.
79. NARAYANAN, A. and SHMATIKOV, V. (2008). Robust deanonymization of large datasets (How to break anonymity of the Netflix Prize dataset). Preprint.
80. NEAL, R. M. and HINTON, G. E. (1998). A view of the EM algorithm that justifies incremental, sparse and other variants. In Learning in Graphical Models (M. I. Jordan, ed.) 355-368. Kluwer.
81. NETFLIX INC. (2006/2010). Netflix Prize webpage: http://www. netflixprize.com/. Netflix Prize Leaderboard: http://www. netflixprize.com/leaderboard/. Netflix Prize Forum: www. netflixprize.com/community/.
82. OARD, D. and KIM, J. (1998). Implicit feedback for recommender systems. In Proc. AAAI Workshop on Recommender Systems 31-36. AAAI, Menlo Park, CA.
83. PARK, S. T. and PENNOCK, D. M. (2007). Applying collaborative filtering techniques to movie search for better ranking and browsing. In Proc. 13th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining 550-559. ACM, New York.
84. PATEREK, A. (2007). Improving regularized singular value decomposition for collaborative filtering. In Proc. KDD Cup and Workshop 2007 39-42. ACM, New York.
85. PIATETSKY, G. (2007). Interview with Simon Funk. SIGKDD Explorations Newsletter 9 38-40.
86. POPESCUL, A., UNGAR, L., PENNOCK, D. and LAWRENCE, S. (2001). Probabilistic models for unified collaborative and content-based recommendation in sparse-data environments. In Proc. 17th Conf. on Uncertainty Artificial Intelligence. Morgan Kaufmann, San Francisco, CA. 437-444.
87. PU, P., BRIDGE, D. G.,MOBASHER, B. and RICCI, F. (2008). In Proc. ACM Conf. on Recommender Systems 2008.
88. RAIKO, T., ILIN, A. and KARHUNEN, J. (2007). Principal component analysis for large scale problems with lots of missing values. In ECML 2007. Lecture Notes in Artificiant Intelligence 4701 (J. N. Kok et al. eds.) 691-698. Springer, Berlin.
89. RENNIE, J. D. M. and SREBRO, N. (2005). Fast maximum margin matrix factorization for collaborative prediction. In Proc. 22nd Int. Conf. on Machine Learning 713-719. ACM, New York.
90. RESNICK, P. and VARIAN, H. R. (1997). Recommender systems. Communications of the ACM 40 56-58.
91. RESNICK, P., IACOCOU, N., SUCHAK, M., BERSTROM, P. and RIEDL, J. (1994). Grouplens: An open architecture for collaborative filtering of netnews. In Proc. ACM Conf. on Computer Support Cooperative Work 175-186.
92. RIPLEY, B. D. (1996). Pattern Recognition and Neural Networks. Cambridge Univ. Press, Cambridge.
93. ROBBINS, H. (1956). An empirical Bayes approach to statistics. In Proc. 3rd Berkeley Sympos. Math. Statist. Probab. I 157-163. Univ. California Press, Berkeley.
94. ROBBINS, H. (1964). The empirical Bayes approach to statistical decision problems. Ann. Math. Statist. 35 1-20.
95. ROBBINS, H. (1983). Some thoughts on empirical Bayes estimation. Ann. Statist. 11 713-723.
96. ROWEIS, S. (1997). EM algorithms for PCA and SPCA. In Advances in Neural Information Processing Systems 10 626-632. MIT Press, Cambridge, MA.
97. SALAKHUTDINOV, R. and MNIH, A. (2008). Probabilistic matrix factorization. In Advances in Neural Information Processing Systems 20 1257-1264. MIT Press, Cambridge, MA.
98. SALAKHUTDINOV, R. and MNIH, A. (2008). Bayesian probabilistic matrix factorization using MCMC. In Proc. 25th Int. Conf. on Machine Learning.
99. SALAKHUTDINOV, R., MNIH, A. and HINTON, G. (2007). Restricted Boltzmann machines for collaborative filtering. In Proc. 24th Int. Conf. on Machine Learning. ACM Inetrnational Conference Proceeding Series 227 791-798. ACM, New York.
100. SALI, S. (2008). Movie rating prediction using singular value decomposition. Technical report, Univ. California, Santa Cruz.
101. SARWAR, B., KARYPIS, G., KONSTAN, J. and RIEDL, J. T. (2000). Application of dimensionality reduction in recommender system-a case study. In Proc. ACM WebKDD Workshop. ACM, New York.
102. SARWAR, B., KARYPIS, G., KONSTAN, J. and RIEDL, J. T. (2001). Item-based collaborative filtering recommendation algorithms. In Proc. 10th Int. Conf. on the World Wide Web 285-295. ACM, New York.
103. SREBRO, N. and JAAKKOLA, T. (2003). Weighted low-rank approximations. In Proc. Twentieth Int. Conf. on Machine Learning (T. Fawcett and N. Mishra, eds.) 720-727. ACM, New York.
104. SREBRO, N., RENNIE, J. D. M. and JAAKKOLA, T. S. (2005). Maximum-margin matrix factorization. In Advances in Neural Information Processing Systems 17 1329-1336.
105. STEIN, C. (1974). Estimation of the mean of a multivariate normal distribution. In Proceedings of the Prague Symposium on Asymptotic Statistics (Charles Univ., Prague, 1973) II 345-381. Charles Univ., Prague.
106. STEIN, C. M. (1981). Estimation of the mean of a multivariate normal distribution. Ann. Statist. 9 1135-1151.
107. STONE, M. (1974). Cross-validatory choice and assessment of statistical predictions (with discussion). J. Roy. Statist. Soc. Ser. B 36 111-147.
108. TAKACS, G., PILASZY, I., NEMETH, B. and TIKK, D. (2007). On the Gravity recommendation system. In Proc. KDD Cup and Workshop 2007 22-30. ACM, New York.
109. TAKACS, G., PILASZY, I., NEMETH, B. and TIKK, D. (2008). Major components of the Gravity recommendation system. SIGKDD Explorations 9 80-83.
110. TAKACS, G., PILASZY, I., NEMETH, B. and TIKK, D. (2008). Investigation of various matrix factorization methods for large recommender systems. In Proc. 2nd Netflix-KDD Workshop. ACM, New York.
111. TAKACS, G., PILASZY, I., NEMETH, B. and TIKK, D. (2008). Matrix factorization and neighbor based algorithms for the Netflix Prize problem. In Proc. ACM Conf. on Recommender Systems 267-274. ACM, New York.
112. TIBSHIRANI, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267-288.
113. TINTAREV, N. and MASTHOFF, J. (2007). A survey of explanations in recommender systems. In Proc. 23rd Int. Conf. on Data Engineering Workshops 801-810. IEEE, New York.
114. TOSCHER, A. and JAHRER, M. (2008). The BigChaos solution to the Netflix Prize 2008. Technical report, commendo research and consulting, Köflach, Austria.
115. TOSCHER, A., JAHRER, M. and BELL, R. M. (2009). The BigChaos solution to the Netflix Grand Prize. Technical report, commendo research and consulting, Koflach, Austria.
116. TOSCHER, A., JAHRER, M. and LEGENSTEIN, R. (2008). Improved neighbourhood-based algorithms for large-scale recommender systems. In Proc. 2nd Netflix-KDD Workshop 2008. ACM, New York.
117. TOSCHER, A., JAHRER, M. and LEGENSTEIN, R. (2010). Combining predictions for accurate recommender systems. In Proc. 16th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining 693-701. ACM, Washington, DC.
118. TUZHILIN, A., KOREN, Y., BENNETT, C., ELKAN, C. and LEMIRE, D. (2008). Proc. 2nd KDD Workshop on Large Scale Recommender Systems and the Netflix Prize Competition. ACM, New York.
119. UNGAR, L. and FOSTER, D. (1998). Clustering methods for collaborative filtering. In Proc. Workshop on Recommendation Systems. AAAI Press, Menlo Park.
120. VAN HOUWELINGEN, J. C. (2001). Shrinkage and penalized likelihood as methods to improve predictive accuracy. Statist. Neerlandica 55 17-34.
121. VAPNIK, V. N. (2000). The Nature of Statistical Learning Theory, 2nd ed. Springer, New York. MR1719582
122. WANG, J., DE VRIES, A. P. and REINDERS, M. J. T. (2006). Unifying user-based and item-based collaborative filtering approaches by similarity fusion. In Proc. 29th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval 501-508. ACM, New York.
123. WEBB, B. (aka Funk, S.) (2006/2007). 'Blog' entries, 27 October 2006, 2 November 2006, 11 December 2007 and 17 August 2007. Available at http://sifter.org/~simon/journal/.
124. WU, M. (2007). Collaborative filtering via ensembles of matrix factorizations. In Proc. KDD Cup and Workshop 2007 43-47. ACM, New York.
125. YE, J. (1998). On measuring and correcting the effects of data mining and model selection. J. Amer. Statist. Assoc. 93 120-131.
126. YUAN, M. and LIN, Y. (2005). Efficient empirical Bayes variable selection and estimation in linear models. J. Amer. Statist. Assoc. 100 1215-1225.
127. ZHANG, Y. and KOREN, J. (2007). Efficient Bayesian hierarchical user modeling for recommendation systems. In Proc. 30th Int. ACM SIGIR Conf. on Research and Developments in Information Retrieval. ACM, New York.
128. ZHOU, Y.,WILKINSON, D., SCHREIBER, R. and PAN, R. (2008). Large scale parallel collaborative filtering for the Netlix Prize. In Proc. 4th Int. Conf. Algorithmic Aspects in Information and Management. Lecture Notes in Comput. Sci. 5031 337-348. Springer, Berlin.
129. ZOU, H., HASTIE, T. and TIBSHIRANI, R. (2006). Sparse principal component analysis. J. Comput. Graph. Statist. 15 265-286.
130. ZOU, H., HASTIE, T. and TIBSHIRANI, R. (2007). On the "degrees of freedom" of the lasso. Ann. Statist. 35 2173-2192.