Evaluating recommender systems from the user's perspective: Survey of the state of the art

User Modeling and User-Adapted Interaction

Volumn 22, Issue 4-5, 2012, Pages 317-355

  Pu, Pearl ^a   Chen, Li ^b   Hu, Rong ^a  

^a EPFL   (Switzerland)

^b HONG KONG BAPTIST UNIVERSITY   (Hong Kong)

Author keywords

Design guidelines; Explanation interface; Recommender systems; Research survey; User experience research

Indexed keywords

BEHAVIORAL INTENTION; DESIGN AND DEVELOPMENT; DESIGN FEATURES; DESIGN METHOD; EVALUATION CRITERIA; EVALUATION FRAMEWORK; PERFORMANCE OF ALGORITHM; PREFERENCE ELICITATION; PRIVACY PRESERVING; REFINEMENT PROCESS; RESEARCH SURVEY; STATE OF THE ART; STATED PREFERENCES; SYSTEM EFFECTIVENESS; USER EXPERIENCE RESEARCH; WEB TECHNOLOGIES;

DESIGN; QUALITY CONTROL; RESEARCH; SURVEYS;

RECOMMENDER SYSTEMS;

EID: 84867333920     PISSN: 09241868     EISSN: 15731391     Source Type: Journal    
DOI: 10.1007/s11257-011-9115-7     Document Type: Article

References (82)

1. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734-749 (2005) (Pubitemid 40860454)
2. Adomavicius, G., Tuzhilin, A.: Context-aware recommender systems. In: The 2nd ACM Conference on Recommender Systems (RecSys '08), pp. 335-336. ACM, New York (2008)
3. Basartan, Y.: Amazon versus the shopbot: an experiment about how to improve the shopbots. Unpublished Ph.D. Summer Paper, Carnegie Mellon University, Pittsburgh, PA (2001)
4. Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., Kraut, R.E.: Using social psychology to motivate contributions to online communities. In: 2004 ACM Conference on Computer Supported Cooperative Work (CSCW '04), pp. 212-221. ACM, New York (2004) (Pubitemid 40928519)
5. Berger, H., Denk, M., Dittenbach, M., Merkl, D., Pesenhofer, A.: Quo Vadis Homo Turisticus? Towards a picture-based tourist profiler. Inf. Commun. Technol. Tour. 2, 87-96 (2007)
6. Bollen, D.G.F.M., Knijnenburg, B.P., Willemsen, M.C., Graus, M.P.: Understanding choice overload in recommender systems. In: The 4th ACM Conference on Recommender Systems (RecSys'10), pp. 63-70. ACM, New York (2010)
7. Brodie, C., Karat, C.M., Karat, J.: Creating an E-commerce environment where consumers are willing to share personal information. In: Karat, C., Blom, J.O., Karat, J. (eds.) Designing Personalized User Experiences in eCommerce, pp. 185-206. Springer, Netherlands (2004)
8. Burke, R.: Hybrid recommender systems: survey and experiments. User Model. User Adapt. Interact. 12(4), 331-370 (2002)
9. Burke, R., Hammond, K., Young, B.: The find me approach to assisted browsing. IEEE Expert Intell. Syst. Appl. 12(4), 32-40 (1997)
10. Chen, L., Pu, P.: Preference-based organization interface: aiding user critiques in recommender systems. In: International Conference on User Modeling (UM'07), Corfu, Greece, 25-29 June, pp. 77-86 (2007) (Pubitemid 350270970)
11. Chen, L., Pu, P.: Interaction design guidelines on critiquing-based recommender systems. User Model. User Adapt. Interact. 19(3), 167-206 (2009)
12. Chen, L., Pu, P.: Experiments on the preference-based organization interface in recommender systems.ACM Trans. Comput. Hum. Interact. 17(1), 1-33 (2010)
13. Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J.: Is seeing believing? How recommender system interfaces affect users' opinions. In: SIGCHI Conference on Human Factors in Computing Systems (CHI '03), pp. 585-592. ACM, New York (2003)
14. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 319-339 (1989)
15. Drenner, S., Sen, S., Terveen, L.: Crafting the initial user experience to achieve community goals. In: 2008 ACM Conference on Recommender Systems (RecSys '08), pp. 187-194. ACM, New York (2008)
16. Dunn, G.,Wiersema, J., Ham, J., Aroyo, L.: Evaluating interface variants on personality acquisition for recommender systems. In: Houben, G.-J.,McCalla, G., Pianesi, F., Zancanaro, M. (eds.) User Modeling, Adaptation, and Personalization, vol. 5535, pp. 259-270. Springer-Verlag, Berlin (2009)
17. Einhorn, H., Hogarth, R.: Confidence in judgment: persistence of the illusion of validity. Psychol. Rev. 85, 395-416 (1978)
18. Guttman, R.H.: Merchant differentiation through integrative negotiation in agent-mediated electronic commerce. Master's Thesis, School of Architecture and Planning, Program in Media Arts and Sciences, Massachusetts Institute of Technology (1998)
19. Haubl, G., Trifts, V.: Consumer decision making in online shopping environments: the effects of interactive decision aids. Mark. Sci. 19, 4-21 (2000)
20. Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining collaborative filtering recommendations. In: 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00), pp. 241-250. ACM, New York (2000) (Pubitemid 32400055)
21. Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5-53 (2004)
22. Hu, R., Pu, P.: A comparative user study on rating vs. personality quiz based preference elicitation methods. In: The 14th International Conference on Intelligent User Interfaces (IUI '09), 8-11 February, pp. 367-372. ACM, Sanibel Island (2009a)
23. Hu, R., Pu, P.: Acceptance issues of personality-based recommender systems. In: The 3rd ACM Conference on Recommender Systems (RecSys 2009), 22-25 October, pp. 221-224. ACM, New York (2009b)
24. Hu, R., Pu, P.: A study on user perception of personality-based recommender systems. In: De Bra, P., Kobsa, A., Chin, D. (eds.) User Modeling, Adaptation, and Personalization, LNCS 6075, pp. 291-302. Springer, Heidelberg (2010)
25. Hu, R., Pu, P.: Enhancing recommendation diversity with organization interfaces. In: the 16th International Conference on Intelligent user Interfaces (IUI '11), pp. 347-350. ACM, New York (2011)
26. Jones, N., Pu, P.: User technology adoption issues in recommender systems. In: Networking and Electronic Commerce Research Conference (NAEC '07), pp. 379-394 (2007)
27. Karau, S.J.,Williams, K.D.: Understanding individual motivation in groups: the collective effort model. In: Turner, M.E. (ed.) Groups at Work: Theory and Research, pp. 113-141. LEA, Mahwah (2001)
28. Kirakowski, J.: SUMI: the software usability measurement inventory. Br. J. Educ. Technol. 24(3), 210-214 (1993)
29. Kleinmuntz, D.N., Schkade, D.A.: Information displays and decision processes. Psychol. Sci. 4, 221-227 (1993)
30. Knijnenburg, B.P., Willemsen, M.C., Hirtbach, S.: Receiving recommendations and providing feedback: the user-experience of a recommender system. In: The 11th International Conference on Electronic Commerce and Web Technologies, pp. 207-216 (2010)
31. Knijnenburg, B.P.,Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model. User Adapt. Interact. 22 (2012). doi:10.1007/s11257-011-9118-4
32. Kobsa, A., Schreck, J.: Privacy through pseudonymity in user-adaptive systems. ACM Trans. Internet Technol. 3(2), 149-183 (2003)
33. Lam, S.K., Frankowski, D., Riedl, J.: Do you trust your recommendations? An exploration of security and privacy issues in recommender systems. In: 2006 International Conference on Emerging Trends in Information and Communication Security (ETRICS), Freiburg, Germany, pp. 14-29 (2006) (Pubitemid 44156237)
34. Linden, G., Hanks, S., Lesh, N.: Interactive assessment of user preference models: the automated travel assistant. In: The Sixth International Conference on User Modeling, pp. 67-78. Springer, Chia Laguna (1997)
35. Locke, E.A., Latham, G.P.: Building a practically useful theory of goal setting and task motivation: a 35 year odyssey. Am. Psychol. 57(9), 705-717 (2002)
36. Mahmood, T., Ricci, F., Venturini, A.: Improving recommendation effectiveness by adapting the dialogue strategy in online travel planning. Int. J. Inf. Technol. Tour. 11(4), 285-302 (2010)
37. McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Thinking positively - explanatory feedback for conversational recommender systems. In: European Conference on Case-Based Reasoning (ECCBR-04) Explanation Workshop, Madrid, Spain, pp. 115-124 (2004)
38. McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Experiments in dynamic critiquing. In: The 10th International Conference on Intelligent User Interfaces (IUI '05), pp. 175-182. ACM, New York (2005)
39. McGinty, L., Smyth, B.: On the role of diversity in conversational recommender systems. In: The Fifth International Conference on Case-Based Reasoning, pp. 276-290. Springer, Berlin (2003) (Pubitemid 36905838)
40. McNee, M.S., Albert, I., Cosley, D., Gopalkrishnan, P., Lam, S.K., Rashid, A.M., Konstan, J.A., Riedl, J.: On the recommending of citations for research papers. In: 2002 ACM Conference on Computer Supported Cooperative Work (CSCW '02), pp. 116-125. ACM, New York (2002)
41. McNee, S.M., Lam, S.K., Konstan, J.A., Riedal, J.: 2003. Interfaces for eliciting new user preferences in recommender systems. In: User Modeling 2003, pp. 178-187 (2003) (Pubitemid 36768563)
42. McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI Extended Abstracts, pp. 1097-1101 (2006a)
43. McNee, S.M., Riedl, J., Konstan, J.A.: Making recommendations better: an analytic model for human-recommender interaction. In: CHI Extended Abstracts, pp. 1103-1108 (2006b)
44. McSherry, D.: Explanation in recommender systems. Artif. Intell. Rev. 24(2), 179-197 (2005) (Pubitemid 41618682)
45. Ochi, P., Rao, S., Takayama, L., Nass, C.: Predictors of user perceptions of web recommender systems: How the basis for generating experience and search product recommendations affects user responses. Int. J. Hum. Comput. Stud. 68(8), 472-482 (2010)
46. Paramythis, A.,Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User Adapt. Interact. 20(5), 383-453 (2010)
47. Payne, J.W., Bettman, J.R., Schkade, D.A.: Measuring constructed preference: towards a building code. J. Risk Uncertain. 19(1-3), 243-270 (1999)
48. Pu, P., Chen, L.: Integrating tradeoff support in product search tools for e-commerce sites. In: The 6th ACM Conference on Electronic Commerce (EC '05), pp. 269-278. ACM, New York (2005) (Pubitemid 43048687)
49. Pu, P., Chen, L.: Trust building with explanation interfaces. In: The 11th International Conference on Intelligent User Interfaces (IUI '06), pp. 93-100. ACM, New York (2006) (Pubitemid 44074855)
50. Pu, P., Chen, L.: Trust-inspiring explanation interfaces for recommender systems. Knowl-Based. Syst. 20(6), 542-556 (2007) (Pubitemid 47068958)
51. Pu, P., Chen, L.: A user-centric evaluation framework of recommender systems. In: Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces (UCERSTI'10), ACM Conference on Recommender Systems (RecSys'10), Barcelona, Spain, pp. 14-21 (2010)
52. Pu, P., Faltings, B.: Decision tradeoff using example-critiquing and constraint programming. Constraints Int. J. 9(4), 289-310 (2004) (Pubitemid 40672187)
53. Pu, P., Kumar, P.: Evaluating example-based search tools. In: The 5th ACM Conference on Electronic Commerce (EC '04), pp. 208-217. ACM, New York (2004)
54. Pu, P., Viappiani, P., Faltings, B.: Increasing user decision accuracy using suggestions. In: Grinter, R., Rodden, T., Aoki, P., Cutrell, E., Jeffries, R., Olson, G. (eds.) The SIGCHI Conference on Human Factors in Computing Systems (CHI '06), ACM, New York (2006)
55. Pu, P., Zhou, M., Castagnos, S.: Critiquing recommenders for public taste products. In: The Third ACM Conference on Recommender Systems (RecSys '09), pp. 249-252. ACM, New York (2009)
56. Pu, P., Faltings, B., Chen, L., Zhang, J.Y., Viappiani, P.: Usability guidelines for product recommenders based on example critiquing research. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, Chapter 16, pp. 511-546. Springer (2010)
57. Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems. In: The 5th ACM Conference on Recommender Systems (RecSys'11), Chicago, IL, USA, 23-27 October (2011)
58. Rashid, A.M., Albert, I., Cosley, D., Lam, S.K., McNee, S.M., Konstan, J.A., Riedl, J.: Getting to know you: learning new user preferences in recommender systems. In: The 7th International Conference on Intelligent User Interfaces (IUI '02), pp. 127-134. ACM, New York (2002)
59. Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Dynamic critiquing. In: Funk, P., González Calero, P.A. (eds.) Advances in Case-Based Reasoning (ECCBR 2004), LNAI 3155, pp. 763-777. Springer, Heidelberg (2004)
60. Resnick, P., Varian, H.R.: Recommender systems. Commun. ACM 40, 56-58 (1997) (Pubitemid 127441130)
61. Riedl, J.: Personalization and privacy. IEEE Internet Comput. 5(6), 29-31 (2001)
62. Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Analysis of recommendation algorithms for e commerce. In: The 2nd ACM Conference on Electronic Commerce (EC '00), pp. 158-167. ACM, New York (2000)
63. Sarwar,B., Karypis, G.,Konstan, J., Riedl, J. Item-based collaborative filtering recommendation algorithms. In: WWW'01, pp. 285-295 (2001)
64. Shearin, S., Lieberman, H.: Intelligent profiling by example. In: The 6th International Conference on Intelligent User Interfaces (IUI '01), pp. 145-151. ACM, New York (2001) (Pubitemid 32470957)
65. Simonson, I.: Determinants of customers' responses to customized offers: conceptual framework and research propositions. J. Mark. 69, 32-45 (2005) (Pubitemid 40195117)
66. Sinha, R., Swearingen, K.: Comparing recommendations made by online systems and friends. In: DELOSNSF Workshop on Personalization and Recommender Systems in Digital Libraries (2001)
67. Sinha, R., Swearingen, K.: The role of transparency in recommender systems. In: CHI Extended Abstracts, pp. 830-831(2002)
68. Smyth, B.: Case-based recommendation. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web: Methods and Strategies ofWeb Personalization, Lecture Notes in Computer Science, vol. 4321. Springer-Verlag, Berlin (2007)
69. Smyth, B., McClave, P.: Similarity vs. diversity. In: Aha, D.W., Watson, I. (eds.) The 4th International Conference on Case-Based Reasoning: Case-Based Reasoning Research and Development (ICCBR '01), pp. 347-361. Springer-Verlag, London (2001) (Pubitemid 33312742)
70. Spiekermann, S., Parachiv, C.: Motivating human-agent interaction: transferring insights from behavioral marketing to interface design. J. Electron. Commer. Res. 2(3), 255-285 (2002)
71. Spiekermann, S., Grossklags, J., Berendt, B.: E-privacy in 2nd generation E-commerce: privacy preferences versus actual behavior. In: The 3rd ACMConference on Electronic Commerce, pp. 38-47. ACM, New York (2001)
72. Swearingen, K., Sinha, R.: Interaction design for recommender systems. In: Designing Interactive Systems (DIS'02), London, 25-28 June (2002)
73. Tintarev, N., Masthoff, J.: Effective explanations of recommendations: user-centered design. In: 2007 ACM Conference on Recommender Systems (RecSys '07), pp. 153-156. ACM, New York (2007a) (Pubitemid 351545646)
74. Tintarev, N., Masthoff, J.: A survey of explanations in recommender systems. In: The 23rd IEEE International Conference on Data Engineering Workshop, pp. 801-810 (2007b)
75. Viappiani, P., Faltings, B., Pu, P.: Evaluating preference-based search tools: a tale of two approaches. In: The Twenty-First National Conference on Artificial Intelligence (AAAI-06), Boston, USA, 16-20 July, pp. 205-210 (2006)
76. Viappiani, P., Faltings, B., Pu, P.: Preference-based search using example-critiquing with suggestions. J. Artif. Intell. Res. 27, 465-503 (2007)
77. Vig, J., Sen, S., Riedl, J.: Tagsplanations: explaining recommendations using tags. In: The 13th International Conference on Intelligent User Interfaces (IUI'09), pp. 47-56. ACM, New York (2009)
78. Williams, M.D., Tou, F.N.: RABBIT: an interface for database access. In: ACM '82 Conference (ACM '82), pp. 83-87. ACM, New York (1982)
79. Xiao, B., Benbasat, I.: Ecommerce product recommendation agents: use, characteristics, and impact. MIS Q. 31(1), 137-209 (2007) (Pubitemid 46382521)
80. Zanker,M., Jessenitschnig, M.: Case-studies on exploiting explicit customer requirements in recommender systems. User Model. User Adapt. Interact. 19(1-2), 133-166 (2009)
81. Ziegler, C., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: The 14th International Conference on World Wide Web, pp. 22-32 (2005)
82. Zukerman, I., Albrecht, D.W.: Predictive statistical models for user modeling. User Model. User Adapt. Interact. 11(1-2), 5-18 (2001) (Pubitemid 32409021)