Layered evaluation of interactive adaptive systems: Framework and formative methods

User Modeling and User-Adapted Interaction

Volumn 20, Issue 5, 2010, Pages 383-453

  Paramythis, Alexandros ^c   Weibelzahl, Stephan ^a   Masthoff, Judith ^b  

^a NATIONAL COLLEGE OF IRELAND   (Ireland)

^b UNIVERSITY OF ABERDEEN   (United Kingdom)

^c NONE

Author keywords

Design; Evaluation framework; Formative evaluation methods; Layered evaluation

Indexed keywords

ADAPTATION PROCESS; ADAPTIVITY; ALTERNATIVE APPROACH; EVALUATION FRAMEWORK; FORMATIVE EVALUATION; INTERACTIVE ADAPTIVE SYSTEMS; INTERACTIVE SYSTEM; LAYERED EVALUATION; PIECE-WISE; PRACTICAL ISSUES;

ADAPTIVE SYSTEMS;

EID: 78650729548     PISSN: 09241868     EISSN: 15731391     Source Type: Journal    
DOI: 10.1007/s11257-010-9082-4     Document Type: Article

References (181)

1. Ackerman, M.S., Cranor, L.F., Reagle, J.: Privacy in e-commerce: examining user scenarios and privacy preferences. In: 1st ACM Conference on Electronic Commerce, pp. 1-8. ACM, Denver, CO (1999)
2. D. Albrecht I. Zuckerman 2007 Statistical and probabilistic methods for user modeling Special issue User Model. User-Adap. Inter. 17 1-2 1 215
3. Anderson, J.R., Boyle, C.F., Yost, G.: The geometry tutor. In: 9th International Joint Conference on Artificial Intelligence, Los Angeles, CA, pp. 1-7. Morgan Kaufmann, San Francisco (1985)
4. Arruabarrena, R., Pérez, T., López-Cu adrado, J., Gutiérrez, J., Vadillo, J.: On evaluating adaptive systems for education. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Malaga, Spain. LNCS, vol. 2347, pp. 363-367. Springer, Berlin (2002)
5. A. Batliner S. Steidl C. Hacker E. Nöth 2008 Private emotions versus social interaction: a data-driven approach towards analysing emotion in speech User Model. User-Adap. Inter. 18 1-2 175 206
6. S. Berkovsky T. Kuflik F. Ricci 2008 Mediation of user models for enhanced personalization in recommender systems User Model. User-Adap. Inter. 18 3 245 286
7. Beyer H., Holtzblatt K.: Contextual Design: Defining Customer-Centred Systems. Morgan Kaufmann, San Francisco (1997)
8. Billsus, D., Pazzani, M.: A hybrid user model for news story classification. In: 7th International Conference on User Modeling, Banff, Canada, pp. 98-108. Springer, Vienna (1999)
9. K. Bontcheva Y. Wilks 2005 Tailoring automatically generated hypertext User Model. User-Adap. Inter. 15 1-2 135 168
10. C. Boyle A.O. Encarnacion 1994 MetaDoc: an adaptive hypertext reading system User Model. User-Adap. Inter. 4 1 1 19
11. E.J. Brown T.J. Brailsford T. Fisher A. Moore 2009 Evaluating learning style personalization in adaptive systems: quantitative methods and approaches IEEE Trans. Learn. Technol. 2 1 10 22
12. Browne D., Norman M., Riches D.: Why build adaptive systems. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 15-57. Academic Press, London (1990)
13. P. Brusilovsky J. Eklund 1998 A study of user model based link annotation in educational hypermedia J. Univ. Comput. Sci. 4 4 429 448
14. Brusilovsky, P., Karagiannidis, C., Sampson, D.: The benefits of layered evaluation of adaptive applications and services. In: 1st Workshop on Empirical Evaluation of Adaptive Systems at UM2001, pp. 1-8. Sonthofen, Germany (2001)
15. Brusilovsky, P., Farzan, R., Ahn, J.: Layered evaluation of adaptive search. In: Workshop on Evaluating Exploratory Search Systems at SIGIR06, pp. 11-13. Seattle, WA (2006)
16. S. Bull J. Kay 2007 Student models that invite the learner in: the SMILI Open learner modelling framework Int. J. Artif. Intell. Edu. 17 2 89 120
17. Card S.K., Thomas T.P., Newell A.: The Psychology of Human-Computer Interaction. Lawrence Erbaum Associates, London (1983)
18. F. Carmagnola F. Cena L. Console O. Cortassa C. Gena A. Goy I. Torre A. Toso F. Vernero 2008 Tag-based user modeling for social multi-device adaptive guides User Model. User-Adap. Inter. 18 5 497 538
19. D.J. Carmichael J. Kay B. Kummerfeld 2005 Consistent modelling of users, devices and sensors in a ubiquitous computing environment User Model. User-Adap. Inter. 15 3-4 197 234
20. M.J. Carroll 2000 Five reasons for scenario-based design Interact. Comput. 13 1 43 60
21. F. Cena L. Console C. Gena A. Goy G. Levi S. Modeo I. Torre 2006 Integrating heterogeneous adaptation techniques to build a flexible and usable mobile tourist guide AI Commun. 19 4 369 384 1127.68483 2312824
22. H.T. Chen 1996 A comprehensive typology for program evaluation Am. J. Eval. 17 2 121 130
23. K. Cheverst E. Byun H. D. Fitton C. Sas C. Kray N. Villar 2005 Exploring issues of user model transparency and proactive behaviour in an office environment control system User Model. User-Adap. Inter. 15 3-4 235 273
24. D.M. Chickering T. Paek 2007 Personalizing influence diagrams: applying online learning strategies to dialogue management User Model. User-Adap. Inter. 17 1-2 71 91
25. D. Chin 2001 Empirical evaluation of user models and user-adapted systems User Model. User-Adap. Inter. 11 1-2 181 194 1030.68623
26. Claypool, M., Le, P., Wased, M., Brown, D.: Implicit interest indicators. In: International Conference on Intelligent User Interfaces, pp. 33-40. ACM, Santa Fe, NM (2001)
27. C. Conati H. MacLaren 2009 Empirically building and evaluating a probabilistic model of user affect User Model. User-Adap. Inter. 19 3 267 303
28. Cooper A.: The Inmates are Running the Asylum. Macmillan, Indianapolis (1999)
29. Cooper, D., Arroyo, I., Woolf, B., Muldner, K., Burleson, W., Christopherson, R.: Sensors model student self concept in the classroom. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 30-41. Springer, Berlin (2009)
30. H. Cramer V. Evers S. Ramlal M. van Someren L. Rutledge N. Stash L. Aroyo B. Wielinga 2008 The effects of transparency on trust in and acceptance of a content-based art recommender User Model. User-Adap. Inter. 18 5 455 496
31. Cronbach L.J., Ambron S.R., Dornbusch S.M., Hess R.D., Hornik R.C., Philips D.C., Walker D.F., Weiner S.S.: Toward Reform of Program Evaluation. Jossey-Bass, San Francisco (1980)
32. Czarkowski, M.: A scrutable adaptive hypertext. PhD Thesis, University of Sydney (2006)
33. R. Damiano C. Gena V. Lombardo F. Nunnari A. Pizzo 2008 A stroll with Carletto: adaptation in drama-based tours with virtual characters User Model. User-Adap. Inter. 18 5 417 453
34. De Bra, P., Houben, G., Wu, H.: AHAM: a Dexter-based reference model for adaptive hypermedia. In: 10th ACM Conference on Hypertext and Hypermedia, pp. 147-156. ACM, Darmstadt, Germany (1999)
35. L. de Campos J. Fernández-Luna J. Huete M. Rueda-Morales 2009 Managing uncertainty in group recommending processes User Model. User-Adap. Inter. 19 3 207 242
36. de Rosis, F., Mazzotta, I., Miceli, M., Poggi, I.: Persuasion artifices to promote wellbeing. In: 1st International Conference on Persuasive Technology, Eindhoven, The Netherlands. LNCS, vol. 3962, pp. 84-95. Springer, Berlin (2006)
37. M. Degemmis P. Lops G. Semeraro 2007 A content-collaborative recommender that exploits WordNet-based user profiles for neighborhood formation User Model. User-Adap. Inter. 17 3 217 255
38. Dey, A.K., Abowd, G.D.: Towards a better understanding of context and context-awareness. In: Workshop on the What, Who, Where, When, and How of Context-Awareness at CHI 2000, pp. 304-307. The Hague, The Netherlands (2000)
39. Dix A., Finlay J., Abowd G., Beale R.: Human Computer Interaction. 2nd edn. Prentice-Hall, Englewood Cliffs (1998)
40. S.K. D'Mello S.D. Craig J. Sullins A.C. Graesser 2006 Predicting affective states through an emote-aloud procedure from AutoTutor's mixed-initiative dialogue Int. J. Artif. Intell. Edu. 16 1 3 28
41. S.K. D'Mello S.D. Craig A. Witherspoon B. McDaniel A.C. Graesser 2008 Automatic detection of learner's affect from conversational cues User Model. User-Adap. Inter. 18 1-2 45 80
42. C. Domshlak T. Joachims 2007 Efficient and non-parametric reasoning over user preferences User Model. User-Adap. Inter. 17 1-2 41 69
43. Dumas J.S., Loring B.A.: Moderating Usability Tests: Principles and Practices for Interacting. Morgan Kaufmann, San Francisco (2008)
44. Encarnação, L., Stoev, S.: Application-independent intelligent user support system exploiting action-sequence based user modeling. In: 7th International Conference on User Modeling, pp. 245-254. Banff, Canada (1999)
45. Ericsson, K.A., Simon, H.A.: Protocol analysis: verbal reports as data, revised edition. MIT Press, Cambridge, MA (1993)
46. K. Forbes-Riley M. Rotaru D.J. Litman 2008 The relative impact of student affect on performance models in a spoken dialogue tutoring system User Model. User-Adap. Inter. 18 1-2 11 43
47. C. Gena 2005 Methods and techniques for the evaluation of user-adaptive systems Knowl. Eng. Rev. 20 1 1 37
48. Gena, C., Ardissono, L.: Intelligent support to the retrieval of information about hydric resources. In: 3rd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems. LNCS, vol. 3137, pp. 126-135. Springer, Berlin (2004)
49. Gena C., Weibelzahl S.: Usability engineering for the adaptive web. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 720-762. Springer, Berlin (2007)
50. S. George I. Zukerman M. Niemann 2007 Inferences, suppositions and explanatory extensions in argument interpretation User Model. User-Adap. Inter. 17 5 439 474
51. Glahn, C., Specht, M., Koper, R.: Smart indicators on learning interactions. In: 2nd European Conference on Technology Enhanced Learning, Crete, Greece. LNCS, vol. 4753, pp. 56-70. Springer, Berlin (2007)
52. Goecks, J., Shavlik, J.: Learning users' interests by unobtrusively observing their normal behavior. In: 5th International Conference on Intelligent User Interfaces, pp. 129-132. ACM, New Orleans, LA (2000)
53. D. Goren-Bar I. Graziola F. Pianesi M. Zancanaro 2006 The influence of personality factors on visitor attitudes towards adaptivity dimensions for mobile museum guides User Model. User-Adap. Inter. 16 1 31 62
54. Goren-Bar, D., Graziola, I., Rocchi, C., Pianesi, F., Stock, O., Zancanaro, M.: Designing and redesigning an affective interface for an adaptive museum guide. In: 1st International Conference on Affective Computing and Intelligent Interaction, Beijing, China. LNCS, vol. 3784, pp. 939-946. Springer, Berlin (2005)
55. J.D. Gould C. Lewis 1985 Designing for usability: key principles and what designers think Commun. ACM 28 3 300 311
56. Gould, J., Conti, J., Hovanyecz, T.: Composing letters with a simulated listening typewriter. In: 1st ACM Conference on Human Factors in Computer Systems (CHI), pp. 367-370. ACM, Gaithersburg, MD (1982)
57. P.D. Green E.S. Ha G.J. Bullock 2010 Enough already about "black box" experiments: Studying mediation is more difficult than most scholars suppose Ann. Am. Acad. Political Soc. Sci. 628 1 200 208
58. Grudin, J., Pruitt, J.: Personas, participatory design, and product development: An infrastructure for engagement. In: Participatory Design Conference, pp. 144-161, ACM, Malmö, Sweden (2002)
59. E. Guzmán R. Conejo J.-L. Pérez de la Cruz 2007 Adaptive testing for hierarchical student models User Model. User-Adap. Inter. 17 1-2 119 157
60. Herder, E.: Utility-based evaluation of adaptive systems. In: 2nd Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 25-30. Johnstown, PA, USA (2003)
61. J.L. Herlocker J.A. Konstan L.G. Terveen J.T. Riedl 2004 Evaluating collaborative filtering recommender systems ACM Trans. Inf. Syst. 22 1 5 53
62. M. Hertzum K. Hansen H.H.K. Andersen 2009 Scrutinizing usability evaluation: does thinking aloud affect behaviour and mental workload? Behav. Inf. Technol. 28 2 165 181
63. V. Hollink M. van Someren B. Wielinga 2007 Discovering stages in web navigation for problem-oriented navigation support User Model. User-Adap. Inter. 17 1-2 183 214
64. K. Höök 2000 Steps to take before intelligent user interfaces become real Interact. Comput. 12 4 409 426
65. Hoppe, H., Tauber, M., Ziegler, J.: A survey of models and formal description methods in HCI with example applications. In: ESPRIT Project, vol. 385 (1986)
66. E. Horvitz T. Paek 2007 Complementary computing: Policies for transferring callers from dialog systems to human receptionists User Model. User-Adap. Inter. 17 1-2 159 182
67. Jameson A.: Systems that Adapt to their Users: An Integrative Perspective. Saarland University, Saarbrücken (2001)
68. Jameson A.: Adaptive interfaces and agents. In: Jacko, J.A., Sears, A. (eds) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 305-330. L. Erlbaum Associates, Hillsdale (2003)
69. Jameson, A.: User modeling meets usability goals. In: 10th International Conference on User Modeling, Edinburgh, UK. LNAI, vol. 3538, pp. 1-3. Springer, Berlin (2005)
70. Jameson A.: Adaptive user interfaces and agents. 2nd edn. In: Sears, A., Jacko, J. (eds) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 433-458. CRC Press, Boca Raton (2008)
71. A. Jameson 2009 Understanding and dealing with usability side effects of intelligent processing AI Mag. 30 4 23 40
72. Jameson A., Schwarzkopf, E.: Pros and cons of controllability: An empirical study. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Málaga, Spain. LNCS, vol. 2347, pp. 193-202. Springer, Berlin (2002)
73. C. Kaplan J. Fenwick J. Chen 1993 Adaptive hypertext navigation based on user goals and context User Model. User-Adap. Inter. 3 3 193 220
74. Karagiannidis, C., Sampson, D.: Layered evaluation of adaptive applications and services. In: 1st International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Trento, Italy. LNCS, vol. 1892, pp. 343-346. Springer, Berlin, (2000)
75. Kay, J.: Stereotypes, student models and scrutability. 5th International Conference on Intelligent Tutoring Systems, Montréal, Canada. LNCS, vol. 1839, pp. 19-30. Springer, Berlin, (2000)
76. J. Kay 2001 Learner control User Model. User-Adap. Inter. 11 1-2 111 127 1030.68724
77. E. Knutov P. De Bra M. Pechenizkiy 2009 AH-12 years later: a comprehensive survey of adaptive hypermedia methods and techniques New Rev. Hyperme'd. Multime'd. 15 1 5 38
78. Kobsa A.: Privacy-enhanced web personalization. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 628-670. Springer, Berlin (2007)
79. Koch, N., Wirsing, M.: The Munich reference model for adaptive hypermedia applications. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Málaga, Spain. LNCS, vol. 2347, pp. 213-222. Springer, Berlin (2002)
80. Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: 14th International Joint Conference on Artificial Intelligence, Montréal, Canada, pp. 1137-1145. Morgan Kaufmann, San Francisco (1995)
81. E. Kosba V. Dimitrova R. Boyle 2007 Adaptive feedback generation to support teachers in web-based distance education User Model. User-Adap. Inter. 17 4 379 413
82. Krogsæter M., Oppermann R., Thomas G.C.: A user interface integrating adaptability and adaptivity. In: Oppermann, R. (ed.) Adaptive User Support: Ergonomic Design of Manually and Automatically Adaptable Software, pp. 97-125. Lawrence Erlbaum, Hillsdale, NJ (1994)
83. Krueger R., Casey M.: Focus Groups: A Practical Guide for Applied Research. 4th edn. Sage Publications, Los Angeles (2009)
84. Kruppa, M., Aslan, I.: Parallel presentations for heterogeneous user groups-an initial user study. In: 4th International Conference on Intelligent Technologies for Interactive Entertainment, Madonna di Campiglio, Italy. LNAI. vol. 3814, pp. 54-63, Springer, Berlin (2005)
85. A. Law Y. Freer J. Hunter R. Logie N. McIntosh J. Quinn 2005 A comparison of graphical and textual presentations of time series data to support medical decision making in the neonatal intensive care unit J. Clin. Monitor. Comput. 19 3 183 194
86. G. Lekakos G. Giaglis 2007 A hybrid approach for improving predictive accuracy of collaborative filtering algorithms User Model. User-Adap. Inter. 17 1-2 5 40
87. Lewis C.: Using the 'thinking-aloud' method in cognitive interface design. In: Research Report RC9265. IBM T.J. Watson Research Center, Yorktown Heights, NY (1982)
88. Ley, T., Kump, B., Maas, A., Maiden, N., Albert, D.: Evaluating the adaptation of a learning system before the prototype is ready: A paper-based lab study. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 331-336, Springer, Berlin (2009)
89. Limongelli, C., Sciarrone, F., Vaste, G.: LS-Plan: An effective combination of dynamic courseware generation and learning styles in web-based education. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 133-142, Springer, Berlin (2008)
90. MacLaren, B., Koedinger, K.: When and why does mastery learning work: instructional experiments with ACT-R "SimStudents". In: 6th International Conference on Intelligent Tutoring Systems, Biarritz, France, pp. 355-366. Springer, Berlin (2002)
91. Magoulas, G.D., Chen, S.Y., Papanikolaou, K.A.: Integrating layered and heuristic evaluation for adaptive learning environments. In: 2nd Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 5-14, Johnstown, PA (2003)
92. M. Maguire 2001 Methods to support human-centred design Int. J. Hum. Comput. Stud. 55 4 587 634 0984.68616
93. Masthoff J.: The evaluation of adaptive systems. In: Patel, N. (ed.) Adaptive Evolutionary Information Systems, pp. 329-347. Idea Group Publishing, London (2002)
94. J. Masthoff 2004 Group modeling: slecting a sequence of television items to suit a group of viewers User Model. User-Adap. Inter. 14 1 37 85
95. Masthoff, J.: The user as wizard: A method for early involvement in the design and evaluation of adaptive systems. In: 5th Workshop on User-Centred Design and Evaluation of Adaptive Systems at AH06, pp. 460-469. Dublin, Ireland (2006)
96. Masthoff, J.: Automatically constructing good hierarchies: HCI meets AI, unpublished
97. J. Masthoff A. Gatt 2006 In pursuit of satisfaction and the prevention of embarrassment: affective state in group recommender systems User Model. User-Adap. Inter. 16 3-4 281 319
98. Masthoff, J., Vasconcelos, W.W., Aitken, C., Correa da Silva, F.S.: Agent-based group modelling for ambient intelligence. In: AISB Symposium on Affective Smart Environments, Newcastle, UK (2007)
99. Maulsby, D., Greenberg, S., Mander, R.: Prototyping an intelligent agent through wizard of Oz. In: 10th ACM Conference on Human Factors in Computing Systems, pp. 277-284. ACM, Amsterdam, The Netherlands (1993)
100. McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: How accuracy metrics have hurt recommender systems. In: CHI Work in Progress, Montréal, Canada (2006)
101. M. Miettinen A. Oulasvirta 2007 Predicting time-sharing in mobile interaction User Model. User-Adap. Inter. 17 5 475 510
102. E. Millán J.L. Pérez de la Cruz 2002 Diagnosis algorithm for student modeling diagnosis and its evaluation User Model. User-Adap. Inter. 12 2-3 281 330 1030.68781
103. Mobasher B.: Data mining for web personalization. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 90-135. Springer, Berlin (2007)
104. B. Mobasher A. Tuzhilin 2009 Special issue on data mining for personalization User Model. User-Adap. Inter. 19 1-2 1 166
105. Moncur, W., Masthoff, J., Reiter, E.: What do you want to know? Investigating the information requirements of patient supporters. In: 21th IEEE International Symposium on Computer-Based Medical Systems, pp. 443-448. IEEE, Jyväskylä, Finland (2008)
106. T. Murray 1993 Formative qualitative evaluation for 'exploratory' ITS research Int. J. Artif. Intell. Educ. 4 2-3 179 207
107. Nguyen, H., Santos, E. Jr.: An evaluation of the accuracy of capturing user intent for information retrieval. In: International Conference on Artificial Intelligence, pp. 341-350. CSREA Press, Las Vegas, NV (2007)
108. Nguyen, H., Masthoff, J., Edwards, P.: Modelling a receiver's position to persuasive arguments. In: 2nd International Conference on Persuasive Technology, Palo Alto, CA. LNCS, vol. 4744, pp. 271-282. Springer, Berlin (2007)
109. Nielsen J.: Evaluating the thinking-aloud technique for use by computer scientists. In: Hartson H.R., Hix D. (eds.) Advances in Human-Computer interaction, vol.3, pp. 69-82. Ablex, Norwood, NJ (1993)
110. Nielsen J.: Heuristic evaluation. In: Nielsen, J., Mack, R.L. (eds) Usability Inspection Methods, pp. 25-64. John Wiley & Sons, New York (1994a)
111. Nielsen J.: Usability Engineering. 2nd edn. Morgan Kaufmann, San Francisco (1994b)
112. A.D. Norman 1994 How might people interact with agents Commun. ACM 37 7 68 71
113. M. Nückles A. Winter J. Wittwer M. Herbert S. Hübner 2006 How do experts adapt their explanations to a layperson's knowledge in asynchronous communication? An experimental study User Model. User-Adap. Inter. 16 2 87 127
114. Ogata K.: Modern Control Engineering. 5th edn. Prentice Hall, Upper Saddle River, NJ (2009)
115. Ohene-Djan, J.: Ownership transfer via personalisation as a value-adding strategy for web-based education. In: Workshop on Adaptive Systems for Web-Based Education at AH2002, pp. 27-41. Málaga, Spain (2002)
116. Oliver, N., Horvitz, E.: A comparison of HMMs and dynamic bayesian networks for recognizing office activities. In: 10th International Conference on User Modeling, Edinburgh, UK. LNCS, vol. 3538, pp. 199-209. Springer, Berlin (2005)
117. O'Malley, C.E., Draper, S.W., Riley, M.S.: Constructive interaction: A method for studying Human-Computer-human interaction. In: 1st International Conference on Human-Computer Interaction, pp. 269-274. Honolulu, HI (1984)
118. R. Oppermann 1994 Adaptively supported adaptability Int. J. Hum. Comput. Stud. 40 3 455 472
119. Oppermann R.: Introduction. In: Oppermann, R. (ed.) Adaptive User Support: Ergonomic Design of Manually and Automatically Adaptable Software, pp. 1-13. Lawrence Erlbaum Associates, Hillsdale (1995)
120. Ortigosa, A., Carro, R. M.: The continuous empirical evaluation approach: Evaluating adaptive web-based courses. In: 9th International Conference on User Modeling, Johnstown, PA. LNCS, vol. 2702, pp. 163-167. Springer, Berlin (2003)
121. T. Paek D.M. Chickering 2007 Improving command and control speech recognition on mobile devices: using predictive user models for language modelling User Model. User-Adap. Inter. 17 1-2 93 117
122. Paramythis, A., Totter, A., Stephanidis, C.: A modular approach to the evaluation of adaptive user interfaces. In: 1st Workshop on Empirical Evaluation of Adaptive Systems at UM2001, pp. 9-24. Sonthofen, Germany (2001)
123. Paramythis, A., Weibelzahl, S.: A decomposition model for the layered evaluation of interactive adaptive systems. In: 10th International Conference on User Modeling, Edinburgh, UK. LNCS, vol. 3538, pp. 438-442 (2005)
124. Person, N.K., Graesser, A.C.: Tutoring research group human or computer? AutoTutor in a bystander Turing test. In: 6th International Conference on Intelligent Tutoring Systems, Biarritz, France. LNCS, vol. 2363, pp. 821-830. Springer, Berlin (2002)
125. D. Petrelli E. Not 2005 User-centred design of flexible hypermedia for a mobile guide: reflections on the HyperAudio experience User Model. User-Adap. Inter. 15 3-4 303 338
126. Pohl, W.: LaboUr-Machine learning for user modeling. In: 7th International Conference on Human-Computer Interaction, pp. 27-30. Elsevier, Amsterdam (1997)
127. W. Pohl 1999 Logic-based representation and reasoning for user modeling shell systems User Model. User-Adap. Inter. 9 3 217 282 366561
128. Popescu, E.: Evaluating the impact of adaptation to learning styles in a web-based educational system. In: 8th International Conference on Web-Based Learning, Aachen, Germany. LNCS, vol. 5686, pp. 343-352. Springer, Berlin (2009)
129. K. Porayska-Pomsta M. Mavrikis H. Pain 2008 Diagnosing and acting on student affect: the tutor's perspective User Model. User-Adap. Inter. 18 1-2 125 173
130. Preece J., Rogers Y., Sharp H., Benyon D.: Human-Computer Interaction. Addison-Wesley, Reading (1994)
131. Robson C.: Experiment, Design, and Statistics in Psychology. 3rd edn. Penguin, London (1994)
132. Santos, O., Boticario, J.: Guiding learners in learning management systems through recommendations. In: 4th European Conference on Technology Enhanced Learning, Nice, France. LNCS, vol. 5794, pp. 596-601. Springer, Berlin (2009)
133. Schein, A.I., Popescul, A., Ungar, L.H., Pennock, D.M.: Methods and metrics for cold-start collaborative filtering. In: 25th Annual international ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland, pp. 253-260. ACM, New York (2002)
134. Schmidt, D., Zukerman, I., Albrecht, D.: Assessing the impact of measurement uncertainty on user models in spatial domains. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 210-222. Springer, Berlin (2009)
135. Scriven M.: Produce evaluation. In: Smith, N.L. (ed.) New Techniques for Evaluation, pp. 121-126. Sage, Beverly Hills (1981)
136. Scriven M.: Beyond formative and summative evaluation. In: McLaughlin, G.W., Phillips, D.C. (eds) Evaluation and Education: At Quarter Century, pp. 19-64. University of Chicago Press, Chicago (1991)
137. M. Scriven 1996 Types of evaluation and types of evaluator Eval. Pract. 17 2 151 162
138. A. Serna H. Pigot V. Rialle 2007 Modeling the progression of Alzheimer's disease for cognitive assistance in smart homes User Model. User-Adap. Inter. 17 4 415 438
139. Shneiderman B.: Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison Wesley, Reading (1998)
140. A.J. Sixsmith 2000 An evaluation of an intelligent home monitoring system J. Telemed. Telecare 6 2 63 72
141. Spada, D., Sánchez-Montañés, M., Paredes, P., Carro, R.: Towards inferring sequential-global dimension of learning styles from mouse movement patterns. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 337-340. Springer, Berlin (2008)
142. Stary, C., Totter, A.: How to integrate concepts of the design and the evaluation of adaptable and adaptive user interfaces. In: 3rd ERCIM Workshop on User Interfaces for All, pp. 68-75. Obernai, France (1997)
143. S. Stamou A. Ntoulas 2009 Search personalization through query and page topical analysis User Model. User-Adap. Inter. 19 1 5 33
144. Stock O., Zancanaro M.: PEACH-Intelligent Interfaces for Museum Visits. Springer, Berlin (2007)
145. O. Stock M. Zancanaro P. Busetta C. Callaway A. Krüger M. Kruppa T. Kuflik, et al. 2007 Adaptive, intelligent presentation of information for the museum visitor in PEACH User Model. User-Adap. Inter. 17 3 257 304
146. S. Suebnukarn P. Haddawy 2006 Modeling individual and collaborative problem-solving in medical problem-based learning User Model. User-Adap. Inter. 16 3-4 211 248 (Pubitemid 44484492)
147. Tarpin-Bernard, F., Marfisi-Schottman, I., Habieb-Mammar, H.: AnAmeter: The first steps to evaluating adaptation. In: 6th Workshop on User-Centred Design and Evaluation of Adaptive Systems at UMAP2009, pp. 11-20. CEUR, Trento, Italy (2009)
148. Tintarev, N., Masthoff, J.: Effective explanations of recommendations: User-centered design. In: ACM conference on Recommender systems, pp. 153-156. ACM, Minneapolis, MN (2007)
149. Tintarev, N., Masthoff, J.: The effectiveness of personalized movie explanations: An experiment using commercial meta-data. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 204-213. Springer, Berlin (2008)
150. Tintarev, N., Masthoff, J.: Evaluating recommender explanations: problems experienced and lessons learned for the evaluation of adaptive systems. In: 6th Workshop on User Centered Design and Evaluation at UMAP09, pp. 54-63. CEUR, Trento, Italy (2009)
151. Tobar, C.M.: Yet another evaluation framework. In: Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 15-24. Johnstown, PA (2003)
152. Totterdell P., Boyle E.: The evaluation of adaptive systems. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 161-194. Academic Press, London (1990)
153. Totterdell P., Rautenbach P.: Adaptation as a problem of design. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 61-84. Academic Press, London (1990)
154. Totterdell P., Rautenbach P., Wilkinson A., Anderson S.: Adaptive interface techniques. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 131-160. Academic Press, London (1990)
155. Trewin, S.: Configuration agents, control and privacy. In: ACM Conference on Universal Usability, pp. 9-16. ACM, Arlington, VA (2000)
156. A. Turing 1950 Computing machinery and intelligence Mind 59 433 460 37064
157. van Barneveld, J., van Setten, M.: Involving users in the design of user interfaces for TV recommender systems. In: 3rd Workshop on Personalization in Future TV at UM03, Johnstown, PA (2003)
158. M.J. van den Haak M.D.T. de Jong P.J. Schellens 2003 Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue Behav. Inf. Technol. 22 5 339 351
159. M.J. van den Haak M.D.T. de Jong P.J. Schellens 2004 Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison Interact. Comput. 16 6 1153 1170
160. L. van Velsen T. van der Geest R. Klaassen M. Steehouder 2008 User-centered evaluation of adaptive and adaptable systems: a literature review Knowl. Eng. Rev. 23 3 261 281
161. VanLehn, K., Niu, Z., Siler, S., Gertner, A.S.: Student modeling from conventional test data: a Bayesian approach without priors. In: 5th International Conference on Intelligent Tutoring Systems, Montréal, Canada. LNCS, vol. 1452, pp. 434-443. Springer, Berlin (1998)
162. E. Walker N. Rummel K.R. Koedinger 2009 CTRL: A research framework for providing adaptive collaborative learning support User Model. User-Adap. Inter. 19 5 387 431
163. Wang, Y., Chen, Z., Kobsa, A.: A collection and systematization of international privacy laws, with special consideration of internationally operating personalized websites. http://www.ics.uci.edu/~kobsa/privacy (2006)
164. Weber, G., Specht, M.: User modeling and adaptive navigation support in WWW-based tutoring systems. In: 6th International Conference on User Modeling, Chia Laguna, Italy, pp. 289-300. Springer, Vienna (1997)
165. Weibelzahl, S.: Evaluation of adaptive systems. In: 8th International Conference on User Modeling. LNCS, vol. 2109, pp. 292-294. Springer, Berlin (2001)
166. Weibelzahl, S.: Evaluation of adaptive systems. PhD Thesis, University of Trier, Germany (2003)
167. Weibelzahl, S.: Problems and pitfalls in evaluating adaptive systems. In: 4th Workshop on the Evaluation of Adaptive Systems at UM'05, pp. 57-66. Edinburgh, UK (2005)
168. Weibelzahl, S., Weber, G.: Evaluating the inference mechanism of adaptive learning systems. In: 9th International Conference of User Modeling, Johnstown, PA. LNCS, vol. 2702, pp. 154-168. Springer, Berlin (2003)
169. Wharton C., Rieman J., Lewis C., Polson P.: The cognitive walkthrough method: A practitioner's guide. In: Nielsen, J., Mack, R.L. (eds) Usability Inspection Methods, pp. 105-141. John Wiley & Sons, New York (1994)
170. Wilson J., Rosenberg D.: Rapid prototyping for user interface design. In: Helander, M. (ed.) Handbook of Human-Computer Interaction, pp. 859-875. Elsevier, Amsterdam (1988)
171. Winter, S., Wagner, S., Deissenboeck, F.: A comprehensive model of usability. In: Engineering Interactive Systems Conference. LNCS, vol. 4940, pp. 106-122. Springer, Berlin (2008)
172. Witten I.A., Frank E.: Data Mining: Practical Machine Learning Tools and Techniques. 2nd edn. Morgan Kaufmann, Amsterdam (2005)
173. Yang, D., Huo, H.: Assessment on the adaptivity of adaptive systems. In: International Conference on Management of e-Commerce and e-Government, pp. 437-440. IEEE, Nanchang, China (2008)
174. M. Yudelson O. Medvedeva R. Crowley 2008 A multifactor approach to student model evaluation User Model. User-Adap. Inter. 18 4 349 382
175. Zancanaro, M., Kuflik, T., Boger, Z., Goren-Bar, D., Goldwasser, D.: Analyzing museum visitors' behavior patterns. In: 11th International Conference on User Modeling, Corfu, Greece. LNCS, vol. 4511, pp. 238-246. Springer, Berlin (2007)
176. Zaslow, J.: If TiVo thinks you are gay, here's how to set it straight. Wall Street J. A 1 (Nov 26, 2002)
177. Zhang, T., Rau, P., Salvendy, G.: Developing instrument for handset usability evaluation: A survey study. In: 12th International Conference on Human-Computer Interaction, Beijing, China. LNCS, vol. 4550, pp. 662-671. Springer, Berlin (2007)
178. Ziegler J., Bullinger J.H.: Formal models and techniques in Human-Computer interaction. In: Shackel, B., Richardson, S.J. (eds) Human Factors for Informatics Usability, pp. 183-206. Cambridge University Press, Cambridge (1991)
179. Ziegler, C.-N., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: 14 th International World Wide Web Conference, pp. 22-32. ACM, Chiba, Japan (2005)
180. A. Zimmermann A. Lorenz 2008 LISTEN: A user-adaptive audio-augmented museum guide User Model. User-Adap. Inter. 18 5 389 416
181. A. Zimmermann M. Specht A. Lorenz 2005 Personalization and context management User Model. User-Adap. Inter. 15 3-4 275 302