Evaluating collaborative filtering recommender systems

ACM Transactions on Information Systems

Volumn 22, Issue 1, 2004, Pages 5-53

  Herlocker, Jonathan L ^a   Konstan, Joseph A ^b,c   Terveen, Loren G ^b,c   Riedl, John T ^b,c  

^a OREGON STATE UNIVERSITY   (United States)

^b UNIVERSITY OF MINNESOTA   (United States)

^c University of Minnesota   (United States)

Author keywords

Collaborative filtering; Evaluation; Metrics; Recommender systems

Indexed keywords

COLLABORATIVE FILTERING; DATASETS; RATINGS DENSITY; RECOMMENDER SYSTEMS;

ALGORITHMS; CORRELATION METHODS; DATA REDUCTION; DECISION MAKING; DECISION SUPPORT SYSTEMS; EQUIVALENCE CLASSES; METRIC SYSTEM; PROBLEM SOLVING; SIGNAL PROCESSING; STANDARDIZATION; STANDARDS;

SIGNAL FILTERING AND PREDICTION;

EID: 3042697346     PISSN: 10468188     EISSN: None     Source Type: Journal    
DOI: 10.1145/963770.963772     Document Type: Review

References (62)

1. AGGARWAL, C. C., WOLF, J. L., WU, K.-L., AND YU, P. S. 1999. Horting hatches an egg: A new graph-theoretic approach to collaborative filtering. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, New York.
2. AMENTO, B., TERVEEN, L., HILL, W., HIX, D., AND SCHULMAN, R. 2003. Experiments in social data mining: The TopicShop System. ACM Trans. Computer-Human Interact. 10, 1 (Mar.), 54-85.
3. AMENTO, B., TERVEEN, L., HIX, D., AND JU, P. 1999. An empirical evaluation of user interfaces for topic management of web sites. In Proceedings of the 1999 Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, 552-559.
4. BAEZA-YATES, R. AND RIBIERO-NETO, B. 1999. Modern Information Retrieval. Addison-Wesley Longman, Boston, Mass.
5. BAILEY, B. P., GURAK, L. J., AND KONSTAN, J. A. 2001. An examination of trust production in computer-mediated exchange. In Proceedings of the 7th Conference on Human Factors and the Web (July).
6. BALABANOVÍC, M. AND SHOHAM, Y. 1997. Fab: Content-based, collaborative recommendation. Commun. ACM 40, 66-72.
7. BASU, C., HIRSH, H., COHEN, W. W. 1998. Recommendation as classification: using social and content-based information in recommendation. In Proceedings of the 15th National Conference on Artificial Intelligence (AAAI-98). C. Rich, and J. Mostow, Eds. AAAI Press, Menlo Park, Calif., 714-720.
8. BILLSUS, D. AND PAZZANI, M. J. 1998. Learning collaborative information filters. In Proceedings of the 15th National Conference on Artificial Intelligence (AAAI-98). C. Rich, and J. Mostow, Eds. AAAI Press, Menlo Park, Calif., 46-53.
9. BREESE, J, S., HECKERMAN, D., AND KADIE, C. 1998. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI-98). G. F. Cooper, and S. Moral, Eds. Morgan-Kaufmann, San Francisco, Calif., 43-52.
10. CANNY, J. 2002. Collaborative filtering with privacy via factor analysis. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information retrieval. ACM, New York, 238-245.
11. CLAYPOOL, M., BROWN, D., LE, P., AND WASEDA, M. 2001. Inferring user Interest. IEEE Internet Comput. 5, 32-39.
12. CLEVERDON, C. AND KEAN, M. 1968. Factors Determining the Performance of Indexing Systems. Aslib Cranfield Research Project, Cranfield, England.
13. COSLEY, D., LAM, S. K., ALBERT, I., KONSTAN, J. A., AND RIEDL, J. 2003. Is seeing believing? How recommender interfaces affect users' opinions. CHI Lett. 5.
14. DAHLEN, B. J., KONSTAN, J. A., HERLOCKER, J. L., GOOD, N., BORCHERS, A., AND RIEDL, J. 1998. Jump-starting movielens: User benefits of starting a collaborative filtering system with "dead data". TR 98-017. University of Minnesota.
15. DOMINGOS, P. AND RICHARDSON, M. 2003. Mining the network value of customers. In Proceedings of the 7th International Conference on Knowledge Discovery and Data Mining. ACM, New York, 57-66.
16. GOLDBERG, D., NICHOLS, D., OKI, B. M., AND TERRY, D. 1992. Using collaborative filtering to weave an information tapestry. Commun. ACM 35, 61-70.
17. GOLDBERG, K., ROEDER, T., GUPTRA, D., AND PERKINS, C. 2001. Eigentaste: A constant-time collaborative filtering algorithm. Inf. Retr. 4, 133-151.
18. GOOD, N., SCHAFER, J. B., KONSTAN, J. A., BORCHERS, A., SARWAR, B. M., HERLOCKER, J. L., AND RIEDL, J. 1999. Combining collaborative filtering with personal agents for better recommendations. In Proceedings of the 16th National Conference on Artificial Intelligence (AAAI-99), J. Hendler, and D. Subramanian, Eds. AAAI Press, Menlo Park, Calif., 439-446.
19. HANLEY, J. A. AND MCNEIL, B. J. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29-36.
20. HARMAN, D. 1995. The TREC conferences. Hypertext - Information Retrieval - Multimedia: Synergieeffekte Elektronisher Informationssysteme. In Proceedings of HIM '95.
21. HARTER, S. P. 1996. Variations in relevance assessments and the measurement of retrieval effectiveness. J. ASIS 47, 37-49.
22. HECKERMAN, D., CHICKERING, D. M., MEEK, C., ROUNTHWAITE, R., AND KADIE, C. 2000. Dependency networks for inference, collaborative filtering, and data visualization. J. Mach. Learn. Res. 1, 49-75.
23. HELANDER, M. 1988. Handbook of Human-Computer Interaction. North Holland, Amsterdam.
24. HERLOCKER, J. L., KONSTAN, J. A., BORCHERS, A., AND RIEDL, J. 1999. An algorithmic framework for performing collaborative filtering. In Proceedings of the 22nd International Conference on Research and Development in Information Retrieval (SIGIR '99) (Aug). M. A. Hearst, F. F. Gey, and R. Tong, Eds. ACM, New York. 230-237.
25. HERLOCKER, J. L., KONSTAN, J. A., AND RIEDL, J. 2000. Explaining collaborative filtering recommendations. In Proceedings of the 2000 Conference on Computer Supported Cooperative Work, 241-250.
26. HERLOCKER, J. L., KONSTAN, J. A., AND RIEDL, J. 2002. An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Inf. Retr. 5, 287-310.
27. HILL, W., STEAD, L., ROSENSTEIN, M., AND FURNAS, G. W. 1995. Recommending and evaluating choices in a virtual community of use. In Proceedings of ACM CHI'95 Conference on Human Factors in Computing Systems. ACM, New York, 194-201.
28. KONSTAN, J. A., MILLER, B. N., MALTZ, D., HERLOCKER, J. L., GORDON, L. R., AND RIEDL, J. 1997. GroupLens: Applying collaborative filtering to Usenet news. Commun. ACM 40, 77-87.
29. LE, C. T., LINDREN, B. R. 1995. Construction and comparison of two receiver operating characteristics curves derived from the same samples. Biom. J. 37, 869-877.
30. LINTON, F., CHARRON, A., AND JOY, D. 1998. OWL: A recommender system for organziation-wide learning. In Proceedings of the 1998 Workshop on Recommender Systems 65-69.
31. MCDONALD, D. W. 2001. Evaluating Expertise Recommendations. In. Proceedings of the ACM 2001 International Conference on Supporting Group Work (GROUP'01). ACM, New York.
32. MCNEE, S., ALBERT, I., COSLEY, D., GOPALKRISHNAN, P, RASHID, A. M., KONSTAN, J. A., AND RIEDL, J. 2002. On the recommending of citations for research papers. In Proceedings of ACM CSCW 2002. ACM, New York.
33. MILLER, B. N., ALBERT, I., LAM, S. K., KONSTAN, J. A., AND RIEDL, J. 2003. MovieLens unplugged: Experiences with a recommender systems on four mobile devices. In Proceedings of the 2003 Conference on Intelligent User Interfaces.
34. MILLER, B. N., RIEDL, J., AND KONSTAN, J. A. 1997. Experiences with GroupLens: Making Usenet useful again. In Proceedings of the 1997 USENIX Technical Conference.
35. MOBASHER, B., DAI, H., LUO, T., AND NAKAGAWA, M. 2001. Effective personalization based on association rule discovery from web usage data. In Proceedings of the 3rd ACM Workshop on Web Information and Data Management (WIDM01), held in conjunction with the International Conference on Information and Knowledge Management (CIKM 2001). ACM, New York.
36. MORITA, M. AND SHINODA, Y. 1994. Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of SIGIR '94, ACM, New York. 272-281.
37. MUI, L, ANG, C., AND MOHTASHEMI, M. 2001. A Probabilistic Model for Collaborative Sanctioning. Technical Memorandum 617. MIT LCS.
38. NEWMAN, W. 1997. Better or just different? On the benefits of designing interactive systems in terms of critical parameters. In Proceedings of the Designing Interactive Systems (DIS97). ACM, New York, 239-246.
39. NIELSEN, J. 1994. Usability Engineering. Academic Press, San Diego, Calif.
40. PENNOCK, D. M., HORVITZ, E., LAWRENCE, S., AND GILES, C. L. 2000. Collaborative filtering by personality diagnosis: A hybrid memory- and model-based approach. In Proceedings of the 16th Annual Conference on Uncertainty in Artificial Intelligence (UAI-2000). Morgan Kaufmann, San Francisco, Calif., 473-480.
41. RASHID, A. M., ALBERT, I., COSLEY, D., LAM, S. K., MCNEE, S., KONSTAN, J. A., AND RIEDL, J. 2002. Getting to know you: Learning new user preferences in recommender systems. In Proceedings of the 2002 Conference on Intelligent User Interfaces (IUI 2002). 127-134.
42. REDDY, P. K., KITSUREGAWA, P., SREEKANTH, P., AND RAO, S. S. 2002. A graph based approach to extract a neighborhood customer community for collaborative filtering. In Databases in Networked Information Systems, Second International Workshop. Lecture Notes in Computer Science Springer-Verlag, New York, 188-200.
43. RESNICK, P., IACOVOU, N., SUCHAK, M., BERGSTROM, P., AND RIEDL, J. 1994. GroupLens: An open architecture for collaborative filtering of netnews. In Proceedings of the 1994 Conference on Computer Supported Collaborative Work. R. Furuta and C. Neuwirth, Eds. ACM, New York. 175-186.
44. RESNICK, P. AND VARIAN, H. R. 1997. Recommender systems. Commun. ACM 40, 56-58.
45. ROGERS, S. C. 2001. Marketing Strategies, Tactics, and Techniques: A handbook for practitioners. Quorum Books, Westport, Conn.
46. SARWAR, B. M., KARYPIS, G., KONSTAN, J. A., AND RIEDL, J. 2000a. Analysis of recommendation algorithms for E-commerce. In Proceedings of the 2nd ACM Conference on Electronic Commerce (EC'00). ACM, New York. 285-295.
47. SARWAR, B. M., KARYPIS, G., KONSTAN, J. A., AND RIEDL, J. 2000b. Application of dimensionality reduction in recommender system-A case study. In Proceedings of the ACM WebKDD 2000 Web Mining for E-Commerce Workshop.
48. SARWAR, B. M., KARYPIS, G., KONSTAN, J. A., AND RIEDL, J. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International World Wide Web Conference (WWW10).
49. SARWAR, B. M., KONSTAN, J. A., BORCHERS, A., HERLOCKER, J. L., MILLER, B. N, AND RIEDL, J. 1998. Using filtering agents to improve prediction quality in the grouplens research collaborative filtering system. In Proceedings of the ACM 1998 Conference on Computer Supported Cooperative Work (CSCW '98), ACM, New York.
50. SCHAFER, J. B., KONSTAN, J. A., AND RIEDL, J. 2002. Meta-recommendation systems:User-controlled integration of diverse recommendations. In Proceedings of the 11th International Conference on Information and Knowledge Management, Nov. 2002, 43-51.
51. SCHEIN, A. I., POPESCUL, A., UNGAR, L. H., AND PENNOCK, D. M. 2001. Generate models for cold-start recommendations. Proceedings of the 2001 ACM SIGIR Workshop on Recommender Systems. ACM, New York.
52. SCHEIN, A. I., POPESCUL, A., UNGAR, L. H., AND PENNOCK, D. M. 2002. Methods and metrics for cold-start collaborative filtering. In Proceedings of the 25th Annual international ACM SIGIR Conference on Research and Development in Information Retrieval (Aug.). ACM, New York.
53. SHARDANAND, U. AND MAES, P. 1995. Social information filtering: Algorithms for automating "word of mouth". In Proceedings of ACM CHI'95 Conference on Human Factors in Computing Systems. ACM, New York. 210-217.
54. SINHA, R. AND SWEARINGEN, K. 2002. The role of transparency in recommender systems. In CHI 2002 Conference Companion.
55. SWEARINGEN, K. AND SINHA, R. 2001. Beyond algorithms: An HCI perspective on recommender systems. In Proceedings of the SIGIR 2001 Workshop on Recommender Systems.
56. SWETS, J. A. 1963. Information retrieval systems. Science 141, 245-250.
57. SWETS, J. A. 1969. Effectiveness of information retrieval methods. Amer. Doc. 20, 72-89.
58. TURPIN, A. AND HERSH, W. 2001. Why batch and user evaluations do not give the same results. In Proceedings of the 24th Annual ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, 17-24.
59. VOORHEES, E. M. AND HARMAN, D. K. 1999. Overview of the seventh Text REtrieval Conference (TREC-7). In NIST Special Publication 500-242 (July), B. M. Voorhees, and D. K. Harman, Eds. NIST, 1-24.
60. WEXELBLAT, A. AND MAES, P. 1999. Footprints: History-rich tools for information foraging. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. M. G. Williams, and M. W. Altom, Eds. ACM, New York, 270-277.
61. WHITTAKER, S., TERVEEN, L. G., AND NARDI, B, 2000. Let's stop pushing the envelope and start addressing it: A reference task agenda for HCI. Human-Computer Interact. 15, 2-3 (Sept.), 75-106.
62. YAO, Y. Y. 1995. Measuring retrieval effectiveness based on user preference of documents. J. ASIS. 46, 133-145.