In situ evaluation of recommender systems: Framework and instrumentation

International Journal of Human Computer Studies

Volumn 68, Issue 8, 2010, Pages 525-547

  Funk, M ^a   Rozinat, A ^a   Karapanos, E ^a   Alves De Medeiros, A K ^a   Koca, A ^a  

^a EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Experience sampling; Observation; Process mining; Recommender systems; Use experience

Indexed keywords

CONSUMER ELECTRONICS PRODUCTS; EXPERIENCE SAMPLING; IN-SITU; INTERACTIVE TV; PROCESS MINING; PRODUCT QUALITY; RECOMMENDER SYSTEMS; REMOTE DATA; USAGE SCENARIOS; USER BEHAVIORS; USER EXPERIENCE; USER TESTING;

BEHAVIORAL RESEARCH; CONSUMER ELECTRONICS; DATA MINING; DIGITAL TELEVISION;

DATA ACQUISITION;

EID: 77955194758     PISSN: 10715819     EISSN: 10959300     Source Type: Journal    
DOI: 10.1016/j.ijhcs.2010.01.002     Document Type: Article

References (49)

1. K. Ali, and W. van Stam Tivo: making show recommendations using a distributed collaborative filtering architecture KDD '04: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2004 ACM New York, NY, USA 394 401
2. Alves de Medeiros, A., Pedrinaci, C., van der Aalst, W., Domingue, J., Song, M., Rozinat, A., Norton, B., Cabral, L., 2007. An outlook on semantic business process mining and monitoring. In: Meersman, R., Tari, Z., Herrero, P. (Eds.), OTM Workshops (2). Lecture Notes in Computer Science, vol. 4806. Springer, Berlin, pp. 12441255.
3. Alves de Medeiros, A.K., van der Aalst, W.M.P., Pedrinaci, C., 2008. Semantic process mining tools: core building blocks. In: Proceedings of the 16th European Conference on Information Systems (ECIS).
4. L. Ardissono, and M. Maybury Preface: special issue on user modeling and personalization for television User Modeling and User-Adapted Interaction 14 1 2004 1 3
5. L. Ardissono, A. Kobsa, and M. Maybury Personalized Digital Television: Targeting Programs to Individual Viewers, Human-Computer Interaction Series, vol. 6 2004 Kluwer Academic Publishers Norwell, MA, USA
6. N. Bolger, A. Davis, and E. Rafaeli Diary methods: capturing life as it is lived Annual Reviews in Psychology 54 1 2003 579 616
7. M. Bradley, and P. Lang Measuring emotion: the self-assessment Manikin and the semantic differential Journal of Behavior Therapy and Experimental Psychiatry 25 1 1994 49 59
8. J. Bruijn, H. Lausen, A. Polleres, and D. Fensel The web service modeling language WSML: an overview Y. Sure, J. Domingue, ESWC. Lecture Notes in Computer Science vol. 4011 2006 Springer Berlin 590 604
9. F. Casati, and M. Shan Semantic analysis of business process executions Eighth International Conference on Extending Database Technology (EDBT '02) 2002 Springer London, UK 287 296
10. J. Chin, V. Diehl, and K. Norman Development of an instrument measuring user satisfaction of the human-computer interface Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 1988 ACM New York, NY, USA 213 218
11. Csikszentmihalyi, M., Larson, R., 1992. Validity and reliability of the experience sampling method. In: The Experience of Psychopathology: Investigating Mental Disorders in their Natural Settings, pp. 4357.
12. J. Froehlich, M. Chen, S. Consolvo, B. Harrison, and J. Landay MyExperience: a system for in situ tracing and capturing of user feedback on mobile phones Proceedings of the 5th International Conference on Mobile Systems, Applications and Services 2007 ACM Press New York, NY, USA 57 70
13. Funk, M., van der Putten, P.H.A., Corporaal, H., 2008a. Specification for user modeling with self-observing systems. In: Proceedings of the First International Conference on Advances in Computer-Human Interaction, pp. 243248.
14. Funk, M., van der Putten, P.H.A., Corporaal, H., 2008b. Model interpretation for executable observation specifications. In: Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering, Knowledge Systems Institute, pp. 785790.
15. Funk, M., Rozinat, A., Alves de Medeiros, A., van der Putten, P., Corporaal, H., van der Aalst, W., 2009a. Improving product usage monitoring and analysis with semantic concepts. In: ISTA '09: Proceedings of the 2009 International Conference on Information Systems Technology and its Applications, pp. 190201.
16. Funk, M., Hoyer, P., Link, S., 2009b. Model-driven instrumentation of graphical user interfaces. In: Proceedings of the 2nd International Conference on Advances in Computer-Human Interaction, pp. 1925.
17. T. Gruber A translation approach to portable ontology specifications Knowledge Acquisition 5 2 1993 199 220
18. Gnther, C., Aalst, W., 2007. Fuzzy mining: adaptive process simplification based on multi-perspective metrics. In: Alonso, G., Dadam, P., Rosemann, M. (Eds.), International Conference on Business Process Management (BPM 2007), vol. 4714, pp. 328343.
19. Gnther, C.W., van der Aalst, W.M.P., 2006. A generic import framework for process event logs. In: Eder, J., Dustdar, S. (Eds.), Business Process Management Workshops, vol. 4103, pp. 8192.
20. Hartson, H., Castillo, J., 1998. Remote evaluation for post-deployment usability improvement. In: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 2229.
21. M. Hassenzahl, and D. Ullrich To do or not to do: differences in user experience and retrospective judgments depending on the presence or absence of instrumental goals Interacting with Computers 19 4 2007 429 437
22. J.M. Hektner, J.A. Schmidt, and M. Csikszentmihalyi Experience Sampling Method: Measuring the Quality of Everyday Life 2007 Sage Publications Inc. Beverley Hills, CA
23. Hepp, M., Leymann, F., Domingue, J., Wahler, A., Fensel, D., 2005. Semantic business process management: a vision towards using semantic web services for business process management. In: IEEE International Conference on e-Business Engineering (ICEBE 2005), pp. 535540.
24. J.L. Herlocker, J.A. Konstan, L.G. Terveen, and J.T. Riedl Evaluating collaborative filtering recommender systems ACM Transactions on Information Systems 22 1 2004 5 53
25. Hilbert, D.M., Redmiles, D.F., 1998. An approach to large-scale collection of application usage data over the internet. In: Proceedings of the 20th International Conference on Software Engineering, Kyoto, Japan, pp. 136145, ISBN:0-8186-8368-6
26. Hofstra, P.P.H.J., 2009. Analysing the effect of consumer knowledge on product usability using process mining techniques. Master's Thesis, Eindhoven University of Technology, Department of Industrial Design, Eindhoven, The Netherlands.
27. Holtzblatt, K., Jones, S., 1993. Contextual inquiry: a participatory technique for system design. In: Participatory Design: Principles and Practice, pp. 180193.
28. S. Intille, J. Rondoni, C. Kukla, I. Ancona, and L. Bao A context-aware experience sampling tool Conference on Human Factors in Computing Systems 2003 ACM New York, NY, USA 972 973
29. Kabitzsch, K., Vasyutynskyy, V., 2004. Architecture and data model for monitoring of distributed automation systems. In: 1st IFAC Symposium on Telematics Applications in Automation and Robotics, Helsinki, pp. 1924.
30. Kahneman, D., Krueger, A., Schkade, D., Schwarz, N., Stone, A., 2004. A survey method for characterizing daily life experience: the day reconstruction method.
31. E. Karapanos, M. Hassenzahl, and J.-B. Martens User experience over time CHI '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems 2008 ACM New York, NY, USA 3561 3566
32. V. Khan, P. Markopoulos, B. Eggen, W. IJsselsteijn, and B. de Ruyter Reconexp: a way to reduce the data loss of the experiencing sampling method Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services 2008 ACM New York, NY, USA 471 476
33. A. Koca, and A.C. Brombacher Extracting "broken expectations" from call center records: Why and how M. Czerwinski, A.M. Lund, D.S. Tan, CHI '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems 2008 ACM Press New York, NY, USA 2985 2990
34. Koca, A., Brombacher, A.C., 2008b. User-centered analysis of feedback operations for quality improvement in new product development. In: van der Vaart, T., van Donk, D.P. (Eds.), Proceedings of the 15th International EurOMA Conference (EurOMA 2008): Tradition and Innovation, EIASM.
35. Koca, A., Schouwenaar, A.J., Brombacher, A.C., 2007. Field-feedback in innovative product development: a comparison of two industrial approaches. In: Fernandes, A., Teixeira, A., Jorge, R.N. (Eds.), Proceedings of the 14th International Product Development Management Conference, EIASM, pp. 657668.
36. A. Koca, M. Funk, E. Karapanos, A. Rozinat, and N. van der Gaarden Grasping product pragmatics: a case with internet on TV M. Darnell, J. Masthoff, S. Panabaker, M. Sullivan, A. Lugmayr, Proceedings of uxTV 2008 International Conference on Designing Interactive User Experiences for TV and Video, ACM International Conference Proceeding Series 2008 ACM Press New York, NY, USA 193 202
37. A. Koca, E. Karapanos, and A. Brombacher "Broken expectations" from a global business perspective CHI '09: Proceedings of the 27th International Conference on Human Factors in Computing Systems 2009 ACM New York, NY, USA 4267 4272
38. Konstan, J., Riedl, J., 1999. Research resources for recommender systems. In: CHI '99 Workshop Interacting with Recommender Systems.
39. R. Larson, and M. Csikszentmihalyi The experience sampling method New Directions for Methodology of Social & Behavioral Science 15 1983 41 56
40. T. Leong, S. Howard, and F. Vetere Choice: abdicating or exercising? CHI '08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems 2008 ACM New York, NY, USA 715 724
41. S.M. McNee, J. Riedl, and J.A. Konstan Being accurate is not enough: how accuracy metrics have hurt recommender systems CHI '06: CHI '06 Extended Abstracts on Human Factors in Computing Systems 2006 ACM New York, NY, USA 1097 1101 http://doi.acm.org/10.1145/1125451.1125659
42. O'Riain, S., Spyns, P., 2006. Enhancing the business analysis function with semantics. In: Meersman, R., Tari, z. (Eds.), OTM Conferences (1). Lecture Notes in Computer Science, vol. 4275, Springer, Berlin, pp. 818835.
43. M. Robinson, and G. Clore Belief and feeling: evidence for an accessibility model of emotional self-report Psychological Bulletin 128 6 2002 934 960
44. J. Russell Core affect and the psychological construction of emotion Psychological Review-New York 110 1 2003 145 172
45. E. Shifroni, and B. Shanon Interactive user modeling: an integrative explicit-implicit approach User Modeling and User-Adapted Interaction 2 4 1992 331 365
46. W. van der Aalst, H. Reijers, A. Weijters, B. van Dongen, A. Alves de Medeiros, M. Song, and H. Verbeek Business process mining: an industrial application Information Systems 32 5 2007 713 732
47. van der Aalst, W.M.P., van Dongen, B.F., Gnther, C.W., Mans, R.S., Alves de Medeiros, A.K., Rozinat, A., Rubin, V., Song, M., Verbeek, H.M.W., Weijters, A.J.M.M., 2007b. ProM 4.0: comprehensive support for real process analysis. In: Kleijn, J., Yakovlev, A. (Eds.), Application and Theory of Petri Nets and Other Models of Concurrency (ICATPN 2007), Lecture Notes in Computer Science, vol. 4546, Springer, Berlin, pp. 484494.
48. W3C: Web Ontology Language (OWL) 〈 http://www.w3.org/2004/OWL/ 〉.
49. WSMT: Web Service Modeling Toolkit 〈 http://sourceforge.net/ projects/wsmt 〉.