User-centered evaluation of adaptive and adaptable systems: A literature review

Knowledge Engineering Review

Volumn 23, Issue 3, 2008, Pages 261-281

  Van Velsen, Lex ^a   Van Der Geest, Thea ^a   Klaassen, Rob ^a   Steehouder, Michaël ^a  

^a UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER NETWORKS; INFORMATION RETRIEVAL; PROCESS DESIGN; PROCESS ENGINEERING; SURVEYS;

ADAPTABLE SYSTEMS; LITERATURE REVIEWS; USER-CENTERED;

QUALITY CONTROL;

EID: 51149108307     PISSN: 02698889     EISSN: 14698005     Source Type: Journal    
DOI: 10.1017/S0269888908001379     Document Type: Review

References (73)

1. Akoumianakis, D., Grammenos, D. & Stephanidis, C. 2001. User interface adaptation: Evaluation perspectives. In User interfaces for All, Stephanidis, C. (ed.). Erlbaum, 339-352.
2. Alpert, S. R. & Vergo, J. G. 2007. User-centered evaluation of personalized web sites: What's unique? In Human Computer Interaction Research in Web Design and Evaluation, Zaphiris, P. & Kurniawan, S. (eds). Idea Group, 257-272.
3. Ammenwerth, E., Iller, C. & Mansmann, U. 2003. Can evaluation studies benefit from triangulation? A case study. International Journal of Medical Informatics 70(2/3), 237-248.
4. Benbunan-Fich, R. 2001. Using protocol analysis to evaluate the usability of a commercial web site. Information & Management 39(2), 151-163.
5. Benyon, D. & Murray, D. 1993. Adaptive systems: From intelligent tutoring to autonomous agents. Knowledge-Based Systems 6(4), 197-219.
6. Bohnenberger, T., Jameson, A., Krüger, A. & Butz, A. 2002. Location-aware shopping assistance: Evaluation of a decision-theoretic approach. Lecture Notes in Computer Science 2411, 155-169.
7. BSI. 1998. ISO 9241-11: Ergonomic Requirements for Office Work with Visual Display Terminals. British Standards Institution.
8. Buchauer, A., Pohl, U., Kurzel, N. & Haux, R. 1999. Mobilizing a health professional's Workstation: Results of an evaluation study. International Journal of Medical Informatics 54, 105-114.
9. Cawsey, A. J., Jones, R. B. & Pearson, J. 2000. The evaluation of a personalised health information system for patients with cancer. User Modeling and User-Adapted Interaction 10, 47-72.
10. Chesnais, P. R., Mucklo, M. J. & Sheena, J. A. 1995. The Fishwrap personalized news system. Paper Presented at the Second International Workshop on Community Networking 'Integrated Multimedia Services to the Home', Princeton, USA.
11. Cheverst, K., Byun, H. E., Fitton, D., Sas, C., Kray, C. & Villar, N. 2005. Exploring issues of user model transparency and proactive behaviour in an office environment control system. User Modeling and User-Adapted Interaction 15, 235-273.
12. Chin, D. N. 2001. Empirical evaluation of user models and user-adapted systems. User Modeling and User-Adapted Interaction 11, 181-194.
13. Cronbach, L. J. 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16(3), 297-334.
14. Davis, F. D., Bagozzi, R. P. & Warshaw, P. R. 1989. User acceptance of computer technology: A comparison of two theoretical models. Management Science 35, 982-1003.
15. De Jong, M. D. T. & Schellens, P. J. 1997. Reader-focused text evaluation. An overview of goals and methods. Journal of Business and Technical Communication 11(4), 402-432.
16. Dicks, R. S. 2002. Mis-usability: On the uses and misuses of usability testing. Paper Presented at the 20th Annual International Conference on Computer Documentation, Toronto, Canada.
17. Ericsson, K. A. & Simon, H. A. 1993. Protocol Analysis. MIT Press.
18. Field, A., Hartel, P. & Mooij, W. 2001. Personal DJ, an open architecture for personalised content delivery. Paper Presented at the Tenth International Conference World Wide Web, Hong Kong.
19. Gates, K. F., Lawhead, P. B. & Wilkins, D. E. 1998. Toward an adaptive www: A case study in customised hypermedia. New Review of Hypermedia and Multimedia 4, 89-113.
20. Gena, C. & Torre, I. 2004. The importance of adaptivity to provide onboard services: A preliminary evaluation of an adaptive tourist information service onboard vehicles. Applied Artificial Intelligence 18(6), 549-580.
21. Gena, C. 2005. Methods and techniques for the evaluation of user-adaptive systems. Knowledge Engineering Review 20(1), 1-37.
22. Gena, C. & Weibelzahl, S. 2007. Usability engineering for the adaptive web. In The Adaptive Web, Brusilovsky, P., Kobsa, A. & Nejdl, W. (eds). Springer-Verlag, 720-762.
23. Goren-Bar, D., Graziola, I., Kuflik, T., Pianesi, F., Rocchi, C, Stock, O. & Zancanaro, M. 2005.I like it: An affective interface for a multimodal museum guide. Retrieved on 20 April 2006 from http://peach.itc.it/papers/ gorenbar2005.pdf
24. Gregor, P., Dickinson, A., Macaffer, A. & Andreasen, P. 2003. Seeword: A personal word processing environment for dyslexic computer users. British Journal of Educational Technology 34(3), 341-355.
25. Hartson, H. R., Andre, T. S. & Williges, R. C. 2003. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction 15(1), 145-181.
26. Henderson, R., Rickwood, D. & Roberts, P. 1998. The beta test of an electronic supermarket. Interacting with Computers 10, 385-399.
27. Herder, E. 2006. Forward, Back and Home Again. Analyzing User Behaviour on the Web. University of Twente.
28. Hertzum, M. & Jacobsen, N. E. 2003. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction 15(1), 183-204.
29. Höök, K. 1997. Evaluating the utility and usability of an adaptive hypermedia system. Paper Presented at IUI '97, Orlando, USA
30. Höök, K. 2000. Steps to take before intelligent user interfaces become real. Interacting with Computers 12(4), 409-426.
31. Hvannberg, E. T., Law, E. L., Lárusdóttir, M. K. 2007. Heuristic evaluation: Comparing ways of finding and reporting usability problems. Interacting with Computers 19, 225-240.
32. Hyldegaard, J. & Seiden, P. 2004. My E-journal: Exploring the usefulness of personalized access to scholarly articles and services. Information Research 9(3), paper 181. Available at http://InformationR.net/ir/9-3/paper181.html
33. ISO. 1999. ISO 13407: Human-Centered Design Processes for Interactive Systems, International Standard Organizations.
34. Jameson, A. 2003. Adaptive interfaces and agents. In Human-Computer Interaction Handbook, Jacko, J. A. & Sears A. (eds). Erlbaum, 305-330.
35. Jameson, A. 2006. Adaptive interfaces and agents. In Human-Computer Interaction Handbook (2nd edn), Jacko, J. A. & Sears, A. (eds). Erlbaum, 433-458.
36. Jensen, A. L., Boll, P. S., Thysen, I. & Pathak, B. K. 2000. Pl@nteInfo®: A web-based system for personalised decision support in crop management. Computers and Electronics in Agriculture 25, 271-293.
37. Kaasinen, E. 2003. User needs for location aware mobile services. Personal Ubiquitous Computing 7(1), 70-79.
38. Karat, C. M., Brodie, C, Karat, J., Vergo, J. & Alpert, S. R. 2003. Personalizing the user experience on ibm. Com. IBM Systems Journal 42(4), 686-701.
39. Ketamo, H. 2003. Xtask: An adaptable learning environment. Journal of Computer Assisted Learning 19(3), 360-370.
40. Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C, El Emam, K. & Rosenberg, J. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28(8), 721-734.
41. Kjeldskov, J., Graham, C, Pedell, S., Vetere, F., Howard, S., Balbo, S. & Davies, J. 2005. Evaluating the usability of a mobile guide: The influence of location, participants and resources. Behaviour & Information Technology 24(1), 51-65.
42. Kolari, J., Laakko, T., Hiltunen, T., Ikonen, V., Kulju, M., Suihkonen, R., Toivonen, S., & Virtanen, T. 2004. Context-Aware Services for Mobile Users (Technology and User Experiences), VTT publications 539.
43. Kramer, J., Noronha, S. & Vergo, J. 2000. A user-centered design approach to personalization. Communications of the ACM 43(8), 45-48.
44. Lentz, L. & De Jong, M. D. T. 1997. The evaluation of text quality: Expert-focused and reader-focused methods compared. IEEE Transactions on Professional Communication 40(3), 224-234.
45. Magoulas, G. D., Chen, S. Y. & Papanikolaou, K. A. 2003. Integrating layered and heuristic evaluation for adaptive learning environments. Paper Presented at the Second Workshop on Empirical Evaluation of Adaptive Systems Held in Conjunction with User Modelling 2003, Pittsburg, USA.
46. Maguire, M. 2001. Methods to support human-centred design. International Journal of Human-Computer Studies 55(4), 587-634.
47. Miles, M. B. & Huberman, A. M. 1994. Qualitative Data Analysis (2nd edn), Sage.
48. Morgan, D. L. 1996. Focus groups. Annual Review of Sociology, 22, 129-152.
49. Muntean, C. H. & McManis, J. 2006. The value of QoE-based adaptation approach in educational hypermedia: Empirical evaluation. Lecture Notes in Computer Science 4018, 121-130.
50. Nahl, D. 1998. Ethnography of novices' first use of web search engines: Affective control in cognitive processing. Internet Reference Services Quarterly 3(2), 51-72.
51. NHS Centre for Reviews and Dissemination. 2001. Undertaking Systematic Reviews of Research on Effectiveness, University of York.
52. Pateli, A. G., Giaglis, G. M. & Spinellis, D. D. 2005. Trial evaluation of wireless info-communication and indoor location-based services in exhibition shows. Advances in Informatics 3746, 199-210.
53. Patton, M. Q. 2002. Qualitative Research & Evaluation Methods, (3rd edn), Sage.
54. Savage, P. 1996. User interface evaluation in an iterative design process: A comparison of three techniques. Paper presented at CHI'96, Vancouver, Canada.
55. Schmidt-Belz, B. & Poslad, S. 2003. User validation of a mobile tourism service. Paper Presented at the Workshop on HCI in Mobile Guides, Udine, Italy.
56. Smith, H., Fitzpatrick, G. & Rogers, Y. 2004. Eliciting reactive and reflective feedback for a social communication tool: A multi-session approach. Paper Presented at Designing Interactive Systems: Across the Spectrum, Cambridge, USA.
57. Sodergard, C., Aaltonen, M., Hagman, S., Hiirsalmi, M., Jarvinen, T., Kaasinen, E., Kinnunen, T., Kolari, J., Kunnas, J. & Tammela, A. 1999. Integrated multimedia publishing: Combining TV and newspaper content on personal channels. Computer Networks: the International Journal of Computer and Telecommunications Networking 31, 1111-1128.
58. Spector, P. E. 1992. Summated Rating Scale Construction, Sage.
59. Stary, C. & Totter, A. 2003. Measuring the adaptability of universal accessible systems. Behaviour & Information Technology 22(2), 101-116.
60. Stein, A. 1997. Usability and assessments of multimodal interaction in the SPEAK! System: An experimental case study. The New Review of Hypermedia and Multimedia 3, 159-180.
61. Su, L. T. 1992. Evaluation measures for interactive information retrieval. Information Processing and Management 28(4), 503-516.
62. Van den Haak, M. J., De Jong, M. D. T. & Schellens, P. J. 2003. Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology 22(5), 339-351.
63. Van den Haak, M. J., De Jong, M. D. T. & Schellens, P. J. 2004. Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interacting with Computers 16, 1153-1170.
64. Van der Geest, T. 2004. Beyond accessibility: Comparing three website usability test methods for people with impairments. Paper Presented at HCI. 2004: Design for Life, Leeds, England.
65. Van Velsen, L., Van der Geest, T. & Klaassen, R. 2007. Testing the usability of a personalized system: Comparing the use of interviews, questionnaires and thinking-aloud. Paper Presented at the IEEE Professional Communication Conference, Seattle, USA.
66. Venkatesh, V., Morris, M. G., Davis, G. B. & Davis, F. D. 2003. User acceptance of information technology: Towards a unified view. MIS Quarterly 27(3), 425-478.
67. Virzi, R. A., Sokolov, J. L. & Karis, D. 1996. Usability problem identification using both low- and high-fidelity prototypes. Paper Presented at the SIGHI Conference on Human Factors in Computing Systems: Common Ground, Vancouver, Canada.
68. Vredenburg, K., Mao, J., Smith, P. W. & Carey, T. 2002. A survey of user-centered design practice. CHI Letters 4(1), 471-478.
69. Walker, M., Takayama, L. & Landay, J. A. 2002. High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. Paper Presented at the 46th Annual Meeting of the Human Factors and Ergonomics Society, Baltimore, USA.
70. Weibelzahl, S., Lippitsch, S. & Weber, G. 2002. Advantages, opportunities, and limits of empirical evaluations: Evaluating adaptive systems. Künstliche Intelligenz 3(2), 17-20.
71. Weibelzahl, S. 2003. Evaluation of Adaptive Systems. PhD thesis, University of Trier.
72. Weibelzahl, S. 2005. Problems and pitfalls in the evaluation of adaptive systems. In Adaptable and Adaptive Hypermedia Systems, Chen, S. Y. & Magoulas, G. D. (eds). IRM Press, 285-299.
73. Weibelzahl, S., Jedlitschka, A. & Ayari, B. 2006. Eliciting requirements for a adaptive decision support system through structured user interviews. Paper Presented at the User Centered Design and Evaluation Workshop Held in Conjunction with Adaptive Hypermedia '06, Dublin, Ireland.