A survey of accuracy evaluation metrics of recommendation tasks

Journal of Machine Learning Research

Volumn 10, Issue , 2009, Pages 2935-2962

  Gunawardana, Asela ^a   Shani, Guy ^b  

^a MICROSOFT RESEARCH   (United States)

^b BEN GURION UNIVERSITY OF THE NEGEV   (Israel)

Author keywords

Collaborative filtering; Comparative studies; Recommender systems; Statistical analysis

Indexed keywords

ACCURACY EVALUATION; COLLABORATIVE FILTERING; COMPARATIVE STUDIES; EVALUATION METRICS; OFFLINE; RECOMMENDATION ALGORITHMS; RECOMMENDER SYSTEMS; RESEARCH COMMUNITIES; STATISTICAL ANALYSIS;

EXPERIMENTS; SIGNAL FILTERING AND PREDICTION; STATISTICAL METHODS;

ALGORITHMS;

EID: 75249088154     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (41)

1. D. Bamber. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology, 12:387-415, 1975.
2. Y. Bengio and Y. Grandvalet. No unbiased estimator of the variance of k-fold cross-validation. Journal of Machine Learning Research, 5, 2004.
3. F. Bodon. A fast APRIORI implementation. In The IEEE ICDM Workshop on Frequent Itemset Mining Implementations, 2003.
4. D. Braziunas and C. Boutilier. Local utility elicitation in GAI models. In Proceedings of the Twenty-first Conference on Uncertainty in Artificial Intelligence, pages 42-49, Edinburgh, 2005.
5. J. S. Bréese, D. Heckerman, and C. M. Kadie. Empirical analysis of predictive algorithms for collaborative filtering. In UAI: Uncertainty in Artificial Intelligence, pages 43-52, 1998.
6. Ò. Celma and P. Herrera. A new approach to evaluating novel recommendations. In RecSys '08: Proceedings of the 2008 ACM Conference on Recommender Systems, 2008.
7. M. Claypool, P Le, M. Waseda, and D. Brown. Implicit interest indicators. In Intelligent User Interfaces, pages 33-40. ACM Press, 2001.
8. F. Hernández del Olmo and E. Gaudioso. Evaluation of recommender systems: A new approach. Expert Systems Applications, 35(3), 2008.
9. J. Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 2006.
10. R. O. Duda and P. E. Hart. Pattern Classification and Scene Analysis. Wiley, 1973.
11. C. Goutte and E. Gaussier. A probabilistic interpretation of precision, recall, and F-score, with implication for evaluation. In ECIR '05: Proceedings of the 27th European Conference on Information Retrieval, pages 345-359, 2005.
12. A. Gunawardana and C. Meek. Aggregators and contextual effects in search ad markets. In WWW Workshop on Targeting and Ranking for Online Advertising, 2008.
13. J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems, 22(1), 2004.
14. C. N. Hsu, H. H. Chung, and H. S. Huang. Mining skewed and sparse transaction data for personalized shopping recommendation. Machine Learning, 57(1-2), 2004.
15. R. Hu and P. Pu. A comparative user study on rating vs. personality quiz based preference elicitation methods. In IUI '09: Proceedings of the 13th International Conference on Intelligent User Interfaces, 2009.
16. S. L. Huang. Comparision of utility-based recommendation methods. In The Pacific Asia Conference on Information Systems, 2008.
17. C. Kadie, C. Meek, and D. Heckerman. CFW: A collaborative filtering system using posteriors over weights of evidence. In Proceedings of the 18th Annual Conference on Uncertainty in Artificial Intelligence (UAI-02), pages 242-250, San Francisco, CA, 2002. Morgan Kaufmann.
18. R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 2009.
19. J. A. Konstan, S. M. McNee, C. N. Ziegler, R. Torres, N. Kapoor, and J. Riedl. Lessons on applying automated recommender systems to information-seeking tasks. In Proceedings of the Twenty-First National Conference on Artifical Intelligence (AAAI), 2006.
20. R. Kumar, P Raghavan, S. Rajagopalan, and A. Tomkins. Recommendation systems: A probabilistic analysis. In FOCS '98: Proceedings of the 39th Annual Symposium on Foundations of Computer Science, 1998.
21. D. Larocque, J. Nevalainen, and H. Oja. A weighted multivariate sign test for cluster-correlated data. Biometrika, 94:267-283, 2007. (Pubitemid 47351560)
22. G. Linden, B. Smith, and J. York. Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet Computing, 7(1), 2003.
23. M. R. McLaughlin and J. L. Herlocker. A collaborative filtering algorithm and evaluation metric that accurately model the user experience. In SIGIR '04: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2004.
24. S. Menee, S. K. Lam, C. Guetzlaff, J. A. Konstan, and J. Riedl. Confidence displays and training in recommender systems. In Proceedings of the 9th IFIP TC 13 International Conference on Human Computer Interaction INTERACT, pages 176-183. IOS Press, 2003.
25. S. M. McNee, J. Riedl, and J. K. Konstan. Making recommendations better: an analytic model for human-recommender interaction. In CHI '06 Extended Abstracts on Human Factors in Computing Systems, 2006.
26. M. Montaner, B. López, and J. L. De La Rosa. A taxonomy of recommender agents on the internet. Artificial Intelligence Review, 19(4), 2003.
27. D. Oard and J. Kim. Implicit feedback for recommender systems. In The AAAI Workshop on Recommender Systems, pages 81-83, 1998.
28. B. Price and P Messinger. Optimal recommendation sets: Covering uncertainty over user preferences. In National Conference on Artificial Intelligence (AAAI), pages 541-548. AAAI Press AAAI Press / The MIT Press, 2005.
29. P. Pu and L. Chen. Trust building with explanation interfaces. In IUI '06: Proceedings of the 11th International Conference on Intelligent User Interfaces, 2006.
30. P. Resnick and H. R. Varian. Recommender systems. Communications of the ACM, 40(3), 1997.
31. C. J. Van Rijsbergen. Information Retrieval. Butterworth-Heinemann, 1979.
32. G. Salton. The SMART Retrieval System-Experiments in Automatic Document Processing. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1971.
33. S. L. Salzberg. On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery, 1(3), 1997.
34. B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Analysis of recommendation algorithms for ecommerce. In. EC '00: Proceedings of the 2nd ACM conference on Electronic commerce, 2000.
35. J. B. Schafer, J. Konstan, and J. Riedi. Recommender systems in e-commerce. In EC '99: Proceedings of the 1st ACM conference on Electronic commerce, 1999.
36. A. I. Schein, A. Popescul, L. H. Ungar, and D. M. Pennock. Methods and metrics for cold-start recommendations. In SIGIR '02: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2002.
37. G. Shani, D. Heckerman, and R. I. Brafman. An mdp-based recommender system. Journal of Machine Learning Research, 6:1265-1295, 2005.
38. M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society B, 36(1): 111-147, 1974.
39. E. M. Voorhees. The philosophy of information retrieval evaluation. In CLEF '01: Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, 2002a.
40. E. M. Voorhees. Overview of tree 2002. In The 11th Text Retrieval Conference (TREC 2002), NIST Special Publication 500-251, pages 1-15, 2002b.
41. C. N. Ziegler, S. M. McNee, J. A. Konstan, and G. Lausen. Improving recommendation lists through topic diversification. In WWW '05: Proceedings of the 14th International Conference on the World Wide Web, 2005.