Future directions in evaluation research: People, organizational, and social issues

Methods of Information in Medicine

Volumn 43, Issue 3, 2004, Pages 215-231

  Kaplan, Bonnie ^a,c   Shaw, N T ^b  

^a YALE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b CHILDREN S AND WOMEN S HEALTH CENTRE OF BRITISH COLUMBIA   (Canada)

^c Kaplan Associates   (United States)

Author keywords

Attitudes towards computers; Barriers; Ethical issues; Evaluation; Implementation; People, organizational, social, issues; Qualitative methods; Sociotechnical; Technology assessment, medical informatics, telemedicine, organizational culture

Indexed keywords

BIOMEDICAL TECHNOLOGY ASSESSMENT; CULTURAL FACTOR; ETHICS; EVALUATION; HEALTH CARE; HUMAN COMPUTER INTERACTION; MEDICAL INFORMATICS; METHODOLOGY; ORGANIZATION; PRIORITY JOURNAL; QUALITATIVE ANALYSIS; RESEARCH; REVIEW; SOCIAL ASPECT; TELECOMMUNICATION; THEORY;

EID: 3142708627     PISSN: 00261270     EISSN: None     Source Type: Journal    
DOI: 10.1055/s-0038-1633862     Document Type: Review

References (303)

1. Paré G, Sicotte C. Information technology sophistication in health care: An instrument validation study among Canadian hospitals. Int J Med Inform 2001; 63 (3): 205-23.
2. Kling R, Rosenbaum H, Hert C. Social informatics in information science: An introduction. J Am Soc Inform Sci 1998; 49 (12): 1047-52.
3. Berg M. Implementing information systems in health care organizations: Myths and challenges. Int J Med Inform 2001; 64 (2-3): 143-56.
4. Anderson JG, Jay SJ (eds). Use and impact of computers in clinical medicine. New York: Springer; 1987.
5. Anderson JG, Aydin CE, Jay SJ (eds). Evaluating health care information systems: Methods and applications. Thousand Oaks, CA: Sage Publications; 1994.
6. van Gennip EMS, Talmon JL (eds). Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995.
7. Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer; 1997.
8. Rigby M. Evaluation: 16 powerful reasons why not to do it - and 6 over-riding imperatives. In: Patel V, Rogers R, Haux R (eds). Medinfo 2001. Amsterdam: IOS Press; 2001. pp. 1198-201.
9. Lorenzi NM, Riley RT. Organizational aspects of health informatics: Managing technological change. New York: Springer; 1995.
10. Lorenzi N, Riley RT. Managing change: An overview. JAMIA 2000; 7 (2): 116-24.
11. Ash J. Managing change: Analysis of a hypothetical case. JAMIA 2000; 7 (2): 125-34.
12. Kaplan B. Organizational evaluation of medical information systems. In: Friedman CP, Wyatt JC (eds). Evaluation methods in medical informatics. New York: Springer; 1997. pp. 255-80.
13. Kaplan B. Addressing organizational issues into the evaluation of medical systems. JAMIA 1997; 4 (2): 94-101.
14. Kuhn KA, Guise DA. From hospital information systems to health information systems - problems, challenges, perspectives. In: Haux R, Kulikowski C (eds). Yearbook of Medical Informatics. Stuttgart: Schattauer; 2001. pp. 63-76.
15. DoH, NHSE, IMG. Project review: Objective evaluation. Guidance for NHS managers on evaluating information systems projects; 1996.
16. Ellis NT, Aarts J, Kaplan B, Leonard K. Alternatives to the controlled trial: New paradigms for evaluation. Proc AMIA Symp (CD ROM) 2000.
17. Goorman E, Berg M. Modelling nursing activities: Electronic patient records and their discontents. Nurs Inq 2000; 200 (7): 3-9.
18. Greatbatch D, Murphy E, Dingwall R. Evaluating medical information systems: Ethnomethodological and interactionist approaches. Health Serv Manag Res 2001; 14 (3): 181-91.
19. Kaplan B, Aarts J, Hebert M, Klecun-Dabrowska E, Lewis D, Vimlarlund V. New approaches to evaluation: Alternatives to the randomized controlled trial - qualitative approaches to design and evaluation: Theory and practice. In: Patel V, Rogers R, Haux R (eds). Medinfo 2001. Amsterdam: IOS Press; 2001. p. 1531.
20. Kaplan B, Anderson JG, Jaffe C, Leonard K, Zitner D. New approaches to evaluation: Alternatives to the randomized controlled trial - qualitative models for evaluation. In: Patel V, Rogers R, Haux R (eds). Medinfo 2001. Amsterdam: IOS Press; 2001. p. 1456.
21. Kaplan B, Atkinson C, Ellis E, Jones M, Klecun-Dabrowska E, Teasdale S. Evaluation in the UK's National Health Service. In: Patel V, Rogers R, Haux R (eds). Medinfo 2001. Amsterdam: IOS Press; 2001. p. 1529.
22. Kushniruk A. Evaluation in the design of health information systems: Application of approaches emerging from usability engineering. Comput Biol Med 2002; 32 (3): 141-49.
23. Protti D. A proposal to use a balanced scorecard to evaluate information for health: An information strategy for the modern NHS (1998-2005). Comput Biol Med 2002; 32 (3): 221-36.
24. Fischer PJ, Stratman WC, Lundsgaarde HP. User reactions to PROMIS: Issues related to acceptability of medical innovations. In: O'Neill JT (ed). Proc Symp Comput Appl Med Care. Silver Spring: IEEE Computer Society; 1980. pp. 1722-30. Reprinted in: Anderson JG, Jay SJ (eds). Use and impact of computers in clinical medicine. New York: Springer; 1987. pp. 284-301.
25. Fischer PJ, Stratman WC, Lundsgaarde HP. User reactions to PROMIS: Issues related to acceptability of medical innovations. In: O'Neill JT (ed). Proc Symp Comput Appl Med Care. Silver Spring: IEEE Computer Society; 1980. pp. 1722-30. Reprinted in: Anderson JG, Jay SJ (eds). Use and impact of computers in clinical medicine. New York: Springer; 1987. pp. 284-301.
26. Lundsgaarde HP, Fischer PJ, Steele DJ. Human problems in computerized medicine. Lawrence, KS: The University of Kansas; 1981.
27. Dlugacz YD, Siegel C, Fischer S. Receptivity towards uses of a new computer application in medicine. In: Blum BL (ed.). Proc Symp Comput Appl Med Care. Silver Spring: IEEE Computer Society; 1980. pp. 384-91.
28. Siegel C, Alexander MJ, Dlugacz YD, Fischer S. Evaluation of a computerized drug review system: Impact, attitudes and interactions. Comput Biomed Res 1984; 17: 419-35.
29. Anderson JG, Jay SL. Computers and clinical judgement: The role of physician networks. Soc Sci Med 1985; 20: 967-79.
30. Anderson JG, Jay SL, Schwerr HM, Anderson MM, Kassing D. Physician communication networks and the adoption and utilization of computer applications in medicine. In: Anderson JG, Jay SJ (eds). Use and impact of computers in clinical medicine. New York: Springer; 1987. pp. 185-99.
31. Barrett JP, Barnum RA, Gordon BB, Pesut RN. Evaluation of a medical information system in a general community hospital. Report no. HSM 110-73-331, PB 233 784 (NTIS PB 248 340). Battelle Columbus Laboratories; 1975.
32. Kaplan BM. Computers in medicine, 1950-1980: The relationship between history and policy [Ph.D.]: University of Chicago; 1983.
33. Collen MF. A history of medical informatics in the United States: 1950-1990. Bethesda, MD: American Medical Informatics Association; 1995.
34. Farlee C, Goldstein B. Hospital organization and computer technology. New Brunswick, NJ: Health Care Systems Research; 1972.
35. Brown B, Harbot B, Kaplan B, Maxwell J. Guidelines for managing the implementation of automated medical systems. In: Heffeman HG (ed.). Proc Symp Comput Applic Med Care. Silver Spring: IEEE Computer Society Press; 1981. pp. 791-96.
36. Brown B, Harbot B, Kaplan B. Management issues in automating medical systems. J Clin Engr 1983; 8 (1): 23-30.
37. Kaplan B. Barriers to medical computing history, diagnosis and therapy for the medical computing 'lag'. In: Ackerman MJ (ed.). Proc Symp Comput Appl Med Care. Silver Spring: IEEE Computer Society Press; 1985. pp. 400-4.
38. Aydin CE. Occupational adaption to computerized medical information systems. J Health Soc Behav 1989; 30: 163-79.
39. Aydin CR, Rice RE. Bringing social worlds together: Computers as catalysts for new interactions in health care organizations. J Health Soc Behav 1992; 33: 168-85.
40. Aydin C. Computerized order entry in a large medical center: Evaluating interactions between departments. In: Anderson JG, Aydin CE, Jay SJ (eds). Evaluating health care information systems: Approaches and applications. Thousand Oaks, Cal: Sage; 1994. pp. 260-75.
41. Sigurdardottir H, Skov D, Bartholdy C, Wann-Larsen S. Evaluating the usefulness of monitoring change readiness in organisations which plan on implementing health informatics systems. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; 2001, September 6-7; Department of Health Policy and Management, Erasmus University, Rotterdam, The Netherlands; 2001. p. 52.
42. Hanmer JC. Diffusion of medical technologies: Comparison with ADP systems in the medical environment. In: O'Neill JT (ed). Proc Symp Comput Applic Med Care. Silver Spring: IEEE Computer Society Press; 1980. pp. 1731-36.
43. Kaplan B. Evaluating informatics applications - some alternative approaches: Theory, social interactionism, and call for methodological pluralism. Int J Med Inform 2001; 64: 39-56.
44. Kaplan B, Duchon D. Combining qualitative and quantitative approaches in information systems research: A case study. Manag Inform Sys Q 1988; 12 (4):571-86.
45. Kaplan B. A model comprehensive evaluation plan for complex information systems: Clinical imaging systems as an example. In: Brown A, Remenyi D (eds). Proc Second European Conference on Information Technology Investment Evaluation. Henley on Thames, Birmingham, England: Operational Research Society; 1995. pp. 14-181.
46. Kaplan B, Morelli R, Goethe J. Preliminary findings from an evaluation of the acceptability of an expert system in psychiatry. Extended Proc AMIA Symp (CD-ROM) 1997.
47. Degoulet Fieschi M. Critical dimensions in medical informatics. Int J Med Inform 1997; 44: 21-6.
48. Sicotte C, Denis JL, Lehoux P. The computer based patient record: A strategic issue in process innovation. J Med Systs 1998; 22 (6): 431-43.
49. Safran C, Jones PC, Rind D, Booker B, Cytryn KN, Patel VL. Electronic communication and collaboration in a health care practice. Artif Intell Med 1998; 12 (2): 137-51.
50. Ash JS, Lyman J, Carpenter J, Fournier L. A diffusion of innovations model of physician order entry. Proc AMIA Symp 2001: 22-6.
51. Mitev N, Kerkham S. Organization and implementation issues of patient data management systems in an intensive care unit. J End User Comput 2001; 13 (3): 20-9.
52. Wetter T. Lessons learned from bringing knowledge-based decision support into routine use. Artif Intell Med 2002; 24 (3): 195-203.
53. Lehoux P, Sicotte C, Denis JL, Berg M, Lacroix A. The theory of use behind telemedicine: How compatible with physicians' clinical routines? Soc Sci Med 2002; 54 (6): 889-904.
54. Hannan TJ, Rotich JK, Odero WW, Menya D, Esamai F, Einterz RM et al. The Mosoriot medical record system; design and initial implementation of an outpatient electronic record system in rural Kenya. Int J Med Inform 2000; 60 (1): 21-8.
55. Wilson M, Howcroft D. The role of gender in user resistance and information systems failure. In: Baskerville R, Stage J, DeGross JI (eds). Organizational and social perspectives on information technology. Boston, Dordrecht, London: Kluwer Academic Publisher; 2000. pp. 453-71.
56. Novek J. IT, gender and professional practice: Or, why an automated drug distribution system was sent back to the manufacturer. Sci Technol Hum Val 2002; 27 (3): 379-403.
57. Kaplan B. Development and acceptance of medical information systems: An historical overview. J Health Hum Resour Admin 1988; 11 (1): 9-29.
58. Massaro TA. Introducing physician order entry at a major academic medical center. I: Impact on organizational culture and behavior. Acad Med 1993; 68 (1): 20-5.
59. Patel VL, Allen VG, Arocha JF, Shortliffe EH. Representing clinical guidelines in GLIF: Individual and collaborative expertise. JAMIA 1998; 5 (5): 467-83.
60. Harrop VM. Virtual healthcare delivery: Defined, modeled and predictive barriers to implementation identified. Proc AMIA Symp 2001: 244-8.
61. May CR, Gask L, Atkinson T, Ellis NT, Mair F, Esmail A. Resisting and promoting new technologies in clinical practice: The case of 'telepsychiatry'. Soc Sci Med 2001; 52 (12): 1889-901.
62. Kaplan B. The computer as Rorschach: Implications for management and user acceptance. In: Dayhoff RE (ed.). Proc Symp Comput Appl Med Care. Silver Spring: IEEE Computer Society Press; 1983. pp. 664-67.
63. van't Reit A, Berg M, Hiddema F, Kees S. Meeting patients' needs with patient information systems: Potential benefits from qualitative research methods. Int J Med Inform 2001; 64: 1-14.
64. Clarke K, O'Moore R, Smeets R, Talmon J, Brender J, McNair P et al. Health systems evaluation of telemedicine: A staged approach. Telemed J 1996; 2 (4): 303-12.
65. Aarts J, Peel V, Wright G. Organizational issues in health informatics: A model approach. Int J Med Inf 1998; 52 (1-3): 235-42.
66. Folz-Murphy N, Partin M, Williams L, Harris CM, Lauer MS. Physician use of an ambulatory medical record system: Matching form and function. Proc AMIA Symp 1998: 260-64.
67. Nordyke RA, Kulikowski CA. An informatics based chronic disease practice - case study of a 35-year computer based longitudinal record. JAMIA 1998; 5 (1): 88-103.
68. Dixon DR. The behavioral side of information technology, Int J Med Inform 1999; 56 (1-3): 117-23.
69. Manning B, Gadd CS. Introducing handheld computing into a residency program: Preliminary results from qualitative and quantitative inquiry. Proc. AMIA Symp 2001: 428-32.
70. Southon G. IT, change and evaluation: An overview of the role of evaluation in health services. Int J Med Inform 1999; 56 (1-3): 125-33.
71. Kaplan B. The influence of medical values and practices on medical computer applications. In: Anderson JG, Jay SJ (eds.). Use and impact of computers in clinical medicine. New York: Springer; 1987. pp. 39-50.
72. Heathfield HA, Wyatt J. Philosophies for the design and development of clinical decision-support systems. Methods Inf Med 1993; 32 (1): 1-8.
73. Kaplan B. The computer prescription: Medical computing, public policy and views of history. Sci Technol Hum Val 1995; 20 (1): 5-38.
74. Forsythe DE. New bottles, old wine: Hidden cultural assumptions in a computerized explanation system for migraine sufferers. Med Anthropol Q 1996; 10 (4): 551-74.
75. Friedman CP. Information technology leadership in academic medical centers: A tale of four cultures. Acad Med 1999; 74 (7): 795-99.
76. Ash JS, Gorman PN, Lavelle M, Lyman J. Multiple perspectives on physician order entry. Proc AMIA Symp 2000: 27-31.
77. Whitley EA, Pouloudi A. Studying the translations of NHSnet. J End User Comput 2001; 13 (3): 31-40.
78. Hannan TJ, Tierney WM, Rotich JK, Odero WW, Smith F, Mamlin JJ et al. The Mosoriot medical record system (MMRS) phase I to phase II implementation: An outpatient computer-based medical record in rural Kenya. In: Patel V, Rogers R, Haux R, editors. Medinfo 2001. Amsterdam: IOS Press; 2001. pp. 619-22.
79. Tierney WM, Rotich JK, Smith FE, Bii J, Einterz RM, Hannan TJ. Crossing the 'digital divide': Implementing an electronic medical record system in a rural Kenyan health center to support clinical care and research. Proc AMIA Symp 2002: 792-95.
80. Brender J, McNair P. User requirements in a system development & evaluation context. Stud Health Technol Inform 2000; 77: 203-7.
81. Bygholm A. End-user support: A necessary issue in the implementation and use of EPR systems. In: Patel V, Rogers R, Haux R (eds). Medinfo 2001. Amsterdam: IOS Press; 2001. pp. 604-08.
82. Brennan PF, Strombom I. Improving health care by understanding patient preferences: The role of computer technology. JAMIA 1998; 5 (3): 257-62.
83. Graves WI, Nyce JM. Normative models and situated practice in medicine. Inform Decis Technol 1992; 18: 143-49.
84. Nyce JM, Timpka T. Work, knowledge and argument in specialist consultations: Incorporaring tacit knowledge into system design and development. Med and Biol Eng and Comput 1993; 31 (1): HTA16-HTA19.
85. Timpka I, Rauch E, Nyce JM. Towards productive knowledge-based systems in clinical organizations: A methods perspective. Artif Intell Med 1994; 6 (6): 501-19.
86. Novek J. Hospital pharmacy automation: Collective mobility or collective control? Soc Sci Med 2000; 51 (4): 491-503.
87. Covvey HD. When is a system only a system? Healthcare Inform. Manag. Comm. Canada 2001; 15 (1): 46-7.
88. Kaplan B, Brennan PF, Dowling AF, Friedman CP, Peel V. Towards an informatics research agenda: Key people and organizational issues. JAMIA 2001; 8 (3): 234-41.
89. Leys M. Health care policy: Qualitative evidence and health technology assessment. Health Policy 2003; 65 (3): 217-26.
90. Lehoux P, Blume S. Technology assessment and the sociopolitics of health technologies. J Health Politics, Policy, and Law 2000; 25 (6): 1083-120.
91. May C, Mort M, Williams T, Mair F, Gask L. Health technology assessment in its local contexts: Studies of telehealthcare. Soc Sci Med 2003; 57 (4): 697-710.
92. Farbey B, Land F, Targett D. The moving stair-case - problems of appraisal and evaluation in a turbulent environment. Inform. Tech. People 1999; 12 (3): 238-52.
93. Burkle T, Ammenwerth E, Prokosch HU, Dudeck J. Evaluation of clinical information systems. What can be evaluated and what cannot? J Eval Clin Pract 2001; 7 (4): 373-85.
94. Land F. Evaluation in a socio-technical context. In: Orlikowski WJ, Walsham G, Jones MR, DeGross JI (eds.). Information technology and changes in organizational work. London: Chapman & Hall; 1996. pp. 115-26.
95. Friedman CF, Haug P. Report on conference track 5: Evaluation metrics and outcome. Int J Med Inform 2003; 69 (2-3): 307-9.
96. Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of health information systems - problems and challenges. Int J Med Inform 2003; 71 (2-3): 125-35.
97. van Gennip EMS, Talmon JL. Introduction. In: van Gennip EMS, Talmon JL (eds). Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995. pp. 1-5.
98. Fallman H. A computer professor questions computerization: The projects will become institutions which no-one can evaluate. Lakartidningen 1997; 94 (4): 204-6.
99. Green CJ, Moehr JR. Performance evaluation frameworks for vertically integrated health care systems: Shifting paradigms in Canada. Proc AMIA Symp 2000: 315-19.
100. Aarts J, Peel V. Using a descriptive model of change when implementing large scale clinical information systems to identify priorities for further research. Int J Med Inform 1999; 56 (1-3): 43-50.
101. Chismar WG, Wiley-Paton S. Test of the technology acceptance model for the Internet in pediatrics. Proc AMIA Symp 2002: 155-59.
102. Gagnon M-P, Godin G, Gagné C, Fortin J-P, Lamothe L, Reinharz D et al. An adaptation of the theory of interpersonal behaviour to the study of telemedicine adoption by physicians. Int J Med Inform 2003; 71 (2-3): 103-15.
103. Grant A, Plante I, Leblanc F. The TEAM methodology for the evaluation of information systems in biomedicine. Comput Biol Med 2002; 32 (3): 195-207.
104. Kaplan B. Social interactionist framework for information systems studies: The 4C's. In: Larson TJ, Levine L, DeGross JI (eds). IFIP WG8.2 & WG8.6 Joint Working Conference on Information Systems: Current issues and future changes: International Federation for Information Processing; 1998. pp. 327-39. Available at http://www.bi.no/dep2/infomgt/wg82-86/proceedings/, accessed 4 Feb 2003.
105. Dwyer SJ, Templeton AW, Martin NL, Lee KR, Levine E, Batnitzky S et al. The cost of managing digital diagnostic images. Radiology 1982; 144 (2): 313-18.
106. Crowe BL. Overview of some methodological problems in assessment of PACS. Int J Biomed Comput 1992; 30 (3-4): 181-86.
107. Kazanjian A, Green CJ. Beyond effectiveness: The evaluation of information systems using a comprehensive health technology assessment framwork. Comput Biol Med 2002; 32 (3): 165-77.
108. Lauer TW, Joshi K, Browdy T. Use of the equity implementation model to review clinical systems implementation efforts: A case report. JAMIA 2000; 7 (1): 91-102.
109. Kokol P. A new microcomputer software system evaluation paradigm: The medical perspective. J Med Syst 1991; 15 (4): 269-75.
110. Kokol P. A tool for software and hardware evaluation. J Med Syst 1996; 20 (3): 167-72.
111. Shaw NT. 'CHEATS': A generic information communication technology (ICT) evaluation framework. Comput Biol Med 2002; 32 (3): 209-20.
112. Kaplan R, Norton D. The balanced score card - measures that drive performance. Harv Bus Rev 1992; 70 (1): 71-9.
113. Gordon D, Chapman R, Kunov H, Dolan A, Carter M. Hospital management decision support: A balanced scorecard approach. In: Cesnik B, McCray AT, Scherrer J-R (eds). Medinfo '98. Amsterdam: IOS Press; 1998. p. Pt 1: 453-56.
114. Niss KU. The use of the balanced scorecard (BSC) in the model for investment and evaluation of medical information systems. Stud Health Technol Inform 1999; 68: 110-4.
115. Heathfield H, Hudson P, Kay S, Roberts R, Williams, J. Issues in the multidisciplinary assessment of healthcare information systems. Inform Tech People 1999; 12 (3): 253-75.
116. Boloix G, Robillard P. A software system evaluation framework. Commun ACM 1995; 28 (12): 17-26.
117. Cornford T, Doukidis GI, Forster D. Experience with a structure, process and outcome framework for evaluating an information system. Omega, Int J Manag Sci 1994; 255 5): 491-504.
118. Donabedian A.The quality of medical care: Methods for assessing and monitoring the quality of care for research and for quality assessment programs. Science 1978; 200: 856-64.
119. Donabedian A. Explorations in quality assessment and monitoring; Vol I: The definition of quality and approaches to its assessment. Ann Arbor: Health Adminisitration Press; 1980.
120. Hebert M. Telehealth success: Evaluation framework development. In: Patel V, Rogers R, Haux R (eds). Medinfo 2001. Amsterdam: IOS Press; 2001. pp. 1145-49.
121. DeLone W, McLean E. Information systems success: The quest for the dependent variable. Inform Syst Res 1992; 3 (1): 60-95.
122. Turunen P. A model for evaluation of health care information systems. In Remenyi D, Brown A (eds). Procs 8th European Conference on Information Technology Evaluation September 17-18, 2001; Oriel College, Oxford, UK. pp. 183-87.
123. Turunen P, Salmela H. The internal stake holders' attitudes toward the evaluation methods for medical information systems. In: Pantzar E, Savolainen R, Tynjälä P (eds). In search for a human-centred information society. Tampere: Tampere University Press; 2001. pp. 253-65.
124. Fineberg H, Bauan R, Sosman M. Computerized cranial tomography - effect on diagnostic and therapeutic plans. JAMIA 1997; 3: 244-7.
125. van der Meijden MJ, Tange HH, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: A literature review. JAMIA 2003; 10 (3): 235-43.
126. Krobock J. A taxonomy: Hospital information systems evaluation methodologies. J Med Systs 1984; 8 (5): 419-29.
127. Grémy F, Degoulet P. Assessment of health information technology: Which questions for which systems? Proposal for a taxonomy. Med Inform (Lond) 1993; 18 (3): 185-93.
128. Anderson JG, Aydin CE. Overview: Theoretical perspectives and methodologies for the evaluation of health care information systems. In: Anderson JG, Aydin CE, Jay SJ (eds). Evaluating health care information systems: Approaches and applications. Thousand Oaks, Cal: Sage; 1994. pp. 5-29.
129. Grant A, Buteau M, Richards Y, Delisle E, Laplante P, Niyonsenga T et al. Informatics methodologies for evaluation research in the practice setting. Methods Inf Med 1998; 37 (2): 178-81.
130. Talmon JL. Workshop WG15: Technology assessment and quality improvement. In: Patel V, Rogers R, Haux R (eds). Medinfo 2001, Amsterdam: IOS Press; 2001. p. 1540.
131. Jones MR. An interpretive method for the formative evaluation of an electronic patient record system. In: Remenyi D, Brown A (eds). Procs of the 8th European conference on IT evaluation 17-18 September 2001; Oriel College, Oxford, UK. pp. 349-56.
132. May CR, Mort M, Williams T, Mair F, Shaw NT. Understanding the evaluation of telemedicine: The play of the social and the technical, and the shifting sands of reliable knowledge. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; September 6-7, 2001; Department of Health Policy and Management, Erasmus University, Rotterdam, The Netherlands; 2001. p. 43.
133. MacFarlane A, Harrison R, Wallace P. The benefits of a qualitative approach to telemedicine research. J Telemed Telecare 2002; 8 Suppl 2: 56-7.
134. May C, Williams T, Mair F, MM., M, Shaw N, Gask L. Factors influencing the evaluation of telehealth interventions: Preliminary results from a qualitative study of evaluation projects in the UK. J Telemed Telecare 2002; 8 Suppl 2: 65-7.
135. Williams T, May C, Mair F, MortM, Shaw NT, Gask L. Normative models of health technology assessment and the social production of evidence about telehealth care. Health Policy 2003; 64 (1): 39-54.
136. Fulop N, Allen P, Clarke A, Black N. From health technology assessment to research on the organisation and delivery of health services: Addressing the balance. Health Policy 2003; 63 (2): 155-65.
137. Turunen P, Talmon J. Stakeholder groups in the evaluation of medical information systems. In: Brown A, Remenyi D (eds). Procs 7th European Conference on the Evaluation of Information Technology. September 28-29, 2000; Dublin, Ireland. pp. 329-34.
138. Chiasson MW, Davidson E. Pushing the contextual envelope: Developing and diffusing IS theory for health information systems research. Inform Organization 2004; in press.
139. IT in Health Care: Sociotechnical Approaches. International Conference Proc. September 6-7, 2001; Department of Health Policy and Management, Erasmus University, Rotterdam, The Netherlands.
140. Protti DJ, Haskell AR. Managing information in hospitals: 60% social, 40% technical. In: Proc IMIA Working Conference on Trends in Hospital Information Systems 1992.
141. Lorenzi NM, Riley RT, Blyth AJC, Southon G, Dixon BJ. Antecedents of the people and organizational aspects of medical informatics: Review of the literature. JAMIA 1997; 4 (2):79-93.
142. Lundsgaarde H. Evaluating medical expert systems. SocSci Med 1987; 24 (10):805-19.
143. Bosk C, Frader J. The impact of place of decision making on medical decisions. In: O'Neill JT (ed). Proc Symp Comput Applic Med Care. Silver Spring: IEEE Computer Society Press; 1980. pp. 1326-29.
144. Barley SR. Technology as an occasion for structuring: Evidence from observations of CT scanners and the social order of radiology departments. Admin Sci Q 1986; 31: 78-108.
145. Barley SR. The social construction of a machine: Ritual, superstition, magical thinking and other pragmatic responses to running a CT scanner. In: Lock M, Gordon D (eds). Biomedicine examined. Dordrecht: Kluwer Academic Publishers; 1988. pp. 497-539.
146. Barley SR. On technology, time, and social order: Technically induced change in the temporal organization of radiological work. In: Dubinskas FA (ed). Making time: Ethnographies of high-technology organizations. Philadelphia: Temple University Press; 1988. pp. 123-69.
147. Barley SR.The alignment of technology and structure through roles and networks. Admin Sci Q 1990; 35 (1): 61-103.
148. Barney S, Nyce JM, Ackerman M, Graves WX III, King J, Koh E, et al. Visualization in the neurosciences: Addressing problems in research, teaching and clinical practice. In: First Conference on Visualization in Biomedical Computing. Los Alamitos, Calif: IEEE Computer Society Press; 1990. pp. 322-27.
149. Nyce JM, Graves WI. The construction of knowledge in neurology: Implications for hypermedia system development. Artif Intell Med 1990; 29 (2): 315-22.
150. Forsythe DE, Buchanan BG. Broadening our approach to evaluating medical information systems. In: Proc Symp Comput Appl Med Care; 1991. pp. 8-12.
151. Saetnan AR. Rigid politics and technological flexibility; the anatomy of a failed hospital innovation. Sci Technol Hum Val 1991; 16 (4):419-47.
152. Yoxen E. Seeing with sound: A study of the development of medical images. In: Bijker WE, Hughes T. Pinch T (eds.). The social construction of technological systems. Cambridge, Mass: MIT Press; 1987. pp. 281-306.
153. Forsythe DE. Using ethnography in the design of an explanation system. Expert Syst Applic 1995; 8 (4):403-17.
154. Lau F, Hayward R. Building a virtual network in a community health research training program. JAMIA 2000; 7 (4): 361-77.
155. Lee RG, Garvin T. Moving from information transfer to information exchange in health and health care. Soc Sci Med 2003; 56: 449-64.
156. Weaver RR. Evaluating the problem knowledge coupler: A case study. In: Anderson JG, Aydin CE, Jay SJ (eds). Evaluating health care information systems: Approaches and applications. Thousand Oaks, CA: Sage; 1994. pp. 203-25.
157. Ash J. Organizational factors that influence information technology diffusion in academic health sciences centers. JAMIA 1997; 4 (2): 102-11.
158. Schubart JR, Einbinder JS. Evaluation of a data warehouse in an academic health sciences center. Int J Med Inform 2000; 60: 319-33.
159. Kaplan B. The medical computing 'lag': Perceptions of barriers to the application of computers to medicine. Int J Technol Assess Health Care 1987; 3 (1): 123-36.
160. Anderson JG, Aydin CE, Kaplan B. An analytical framework for measuring the effectiveness/impacts of computer-based patient record systems. In: Nunamaker JF, Sprague RH (eds). IV: Information Systems - collaboration systems and technology, organizational systems and technology. Los Alamitos, Cal: IEEE Computer Society Press; 1995. pp. 767-76.
161. Anderson JG. Evaluation in health informatics: Social network analysis. Comput Biol Med 2002; 32 (3): 179-93.
162. May C, Ellis NT. When protocols fail: technical evaluation, biomedical knowledge and the social production of 'facts' about a telemedicine clinic. Soc Sci Med 2001; 53 (8): 989-1002.
163. Sicotte C, Dennis JL, Lehoux P, Champagne F. The computer-based patient record: Challenges towards timeless and spaceless medical practice. J Med Systs 1998; 22 (4): 237-56.
164. Kristensen M. Assessing a prerequisite for a constructivist approach - change readiness in organisations implementing health informatics systems. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; 2001 6-7 September 2001; Department of Health Policy and Management, Erasmus University, Rotterdam, The Netherlands; 2001. p. 38.
165. Aarts J. On articulation and localization - some sociotechnical issues in design, implementation and evaluation of knowledge based systems. In: Quaglini S, Barahona P, Andreassen S (eds). Artificial Intelligence in Medicine: Proc. of the 8th Conference on Artificial Intelligence in Medicine in Europe AIME 2001. Berlin: Springer; 2001. pp. 16-19.
166. Berg M, Langenberg C, Berg I, Kwakkemaat J. Considerations for sociotechnical design: Experiences with an electronic patient record in a clinical context. Int J Med Inform 1998; 52 (1-3): 243-51.
167. Berg M, Goorman E. The contextual nature of medical information. Int J Med Inform 1999; 56 (1-3): 51-60.
168. Berg M. Patient care information systems and health care work: A sociotechnical approach. Int J Med Inform 1999; 55 (2): 87-101.
169. Reddy M, Pratt W, Dourish P, Shabot MM. Sociotechnical requirements analysis for clinical systems. Methods Inf Med 2003; 42 (4): 437-44.
170. Stoop AP, Berg M. Integrating quantitative and qualitative methods in patient care information system evaluation: Guidance for the organizational decision maker. Methods Inf Med 2003; 42 (4): 458-62.
171. Bygholm A. Activity theory as a framework for conducting end-use support. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; September 6-7, 2001; Department of Health Policy and Management Erasmus University, Rotterdam, The Netherlands; 2001. p. 21.
172. Grémy F, Fessler JM, Bonnin M. Information systems evaluation and subjectivity. Int J Med Inform 1999; 56 (1-3): 13-23.
173. Davidson EJ. Analyzing genre of organizational communication in clinical information systems. Inform Tech People 2000; 33 (3): 196-209.
174. Klecun-Dabrowska E, Cornford T. Evaluating telehealth: The search for an ethical perspective. Camb Q Healthc Ethics 2001; 10 (2): 161-69.
175. Zerubavel E. Patterns of time in hospital life. Chicago and London: University of Chicago Press; 1979.
176. Ash J, Berg M. Report of conference track 4: Socio-technical issues of HIS. Int J Med Inform 2003; 69 (2-3): 305-6.
177. Mathiassen L. Collaborative research practice. In: Orlikowski WJ, Walsham G, Jones MR, DeGross JI (eds). Information technology and changes in organizational work. London: Chapman & Hall; 1996. pp. 127-48.
178. Sawyer S, Rosenbaum H. Social informatics in the information sciences: Current activities and emerging directions. Inform Sci 2002; 3 (2).
179. Reddy M, Pratt W, Dourish P, Shabot MM. Asking questions: Information needs in a surgical intensive care unit. Proc AMIA Symp 2002: 647-51.
180. Methods Inf Med 2003; 42 (4).
181. Kaplan B, Lau F, Aarts J, Forsythe DE. Information systems qualitative research in health care. In: Lee AS, Liebenau J, DeGross JI (eds). Information systems and qualitative research. London: Chapman & Hall; 1997. pp. 569-70.
182. Aarts J, Goorman E, Heathfield H, Kaplan B. Successful development, implementation and evaluation of information systems: Does healthcare serve as a model for networked organizations? In: Baskerville R, Stage J, DeGross JI, editors. Organizational and social perspective on information technology. Boston, Dordrecht, London: Kluwer Academic Publishers; 2000. pp. 517-20.
183. Kaplan B, Kvasny L, Sawyer S, Trauth EM. New words and old books: Challenging conventional discourses about domain and theory in information systems research. In: Myers MD, Whitley EA, Wynn E, De Gross JI (eds). Global and organizational discourse about information technology. London: Kluwer; 2002. pp. 539-45.
184. Bloomfield B, Coombs R, Cooper D, Rea D. Machines and manoeuvres: Responsibility accounting and the construction of hospital information systems. Accounting Manag Inform Technol 1992; 2: 197-219.
185. Bloomfield BP, McLean C. Madness and organization: Informed management and empowerment. In: Orlikowski WJ, Walsham G, Jones MR, DeGross JI (eds). Information technology and changes in organizational work. London: Chapman & Hall; 1996. pp. 371-93.
186. Inform Tech People 1999; 12 (3).
187. Silva L, Backhouse J. Becoming part of the furniture: The institutionalization of infromation systems. In: Lee AS, Leibenau J, DeGross. JI (eds). Information systems and qualitative research. London: Chapman & Hall; 1997. pp. 389-414.
188. J End User Comput 2001; 13 (3 and 4).
189. Walsham G. IT and changing professional identity: Micro-studies and macro-theory. J Am Soc Inform Sci 1998; 49: 1081-89.
190. Avgerou C. Information systems and global diversity. Oxford: Oxford University Press; 2002.
191. Hersh WR, Patterson PK, Kraemer DF. Telehealth: The need for evaluation redux. JAMA 2002; 9 (1): 89-91.
192. Nykanen P, Chowdhury S, Wigertz O. Evaluation of decision support systems in medicine. Comput Methods Programs Biomed. 1991; 34: 229-38.
193. Allen D. Telemental health services today. Telemed Today 1994; 2 (2): 1-24.
194. Burghgraeve P, De Maeseneer J. Improved methods for assessing information technology in primary health care and an example from telemedicine. J Telemed Telecare 1995; 1: 157-64.
195. Bashur R. On the definition and evaluation of telemedicine. Telemed J 1995; 1 (1): 19-30.
196. Filiberti D, Wallace J, Koteeswaran R, Neft D. A telemedicine transaction model. Telemed J 1995; 1 (3): 237-47.
197. Grigsby J. You got an attitude problem or what? Telemed Today 1995: 32-4.
198. Grigsby J, Schlenker R, Kaehny M, Shaughnessy P. Analytic framework for evaluation of telemedicine. Telemed J 1995; 1 (1): 31-9.
199. Grigsby J. Sentenced to life without parole doing outcomes research. Telemed Today 1996: 40-1.
200. DeChant H, Tohme W, Mun S, Hayes W. Health systems evaluation of telemedicine: A staged approach. Telemed J 1996; 2 (4): 303-12.
201. McIntosh E, Cairns J. A framework for the economic evaluation of telemedicine. J Telemed Telecare 1997; 3: 132-9.
202. Yellowlees P. Practical evaluation of telemedicine systems in the real world. J. Telemed. Telecare 1998; 4 (S1): 56-7.
203. Perednia DM. Telemedicine system evaluation, transaction models and multicentered research. J Am Health Inform Manag Assn 1996; 67 (1): 60-3.
204. Rigby M. Health informatics as a tool to improve quality in non-acute care - new opportunities and a matching need for a new evaluation paradigm. Int J Med Inform 1999; 56 (1-3): 141-50.
205. Tohme WG, Hayes WS, Dai H, Komo D. Evaluation of a telemedicine platform for three medical applications. Comp Assisted Rad 1996: 505-11.
206. Wyatt JC. Commentary: Telemedicine trials - clinical pull or technology push? BMJ 1996; 313: 380-1.
207. May C, Gask L, Ellis NT, Atkinson TA, Mair F, Smith C et al. Telepsychiatry evaluation in the North West of England: Preliminary results. Telemed Telecare 2000; 6 (S1): 20-2.
208. Qavi T, Corley L, Kay S. Nursing staff requirements for telemedicine in the neonatal intensive care unit. J End User Comput 2001; 13 (3): 5-13.
209. Reiser SJ. Medicine and the reign of technology. Cambridge: Cambridge University Press; 1978.
210. Kaplan B, Brennan PF. Consumer informatics supporting patients as co-producers of quality. JAMIA 2001; 8 (4): 309-16.
211. Mort M, May CR, Williams T. Remote doctors and absent patients: Acting at a distance in telemedicine? Sci Technol Hum Val 2003; 28 (2): 274-95.
212. Turner J. Telemedicine: Generating the virtual office visit. In: Eder LB (ed). Managing healthcare information systems with web-enabled technologies. Hershey and London: Idea Group Publishing; 2000. pp. 59-68.
213. Balas EA, Jaffrey F, Kuperman GJ, Boren SA, Brown GD, Pinciroli F et al. Electronic communication with patients. Evaluation of distance medicine technology. JAMA 1997; 278 (2): 152-9.
214. Finkelstein J, Cabrera MR, Hripcsak G. Internet-based home asthma telemonitoring: Can patients handle the technology? Chest 2000; 117 (1): 148-55.
215. Sandelowski M. Visible humans, vanishing bodies, and virtual nursing: Complications of life, presence, place, and identity. Advances in Nursing Science 2002; 24 (3): 58-70.
216. Allaert FA, Weinberg D, Dusserre P, Yvon PJ, Dusserre L, Cotran P. Evaluation of a telepathology system between Boston (USA) and Dijon (France): Glass slides versus telediagnostic tv-monitor. Proc Annu Symp Comput Appl Med Care 1995: 596-600.
217. Allaert FA, Weinberg D, Dusserre P, Yvon PJ, Dusserre L, Retaillau B et al. Evaluation of an international telepathology system between Boston (USA) and Dijon: Glass slides versus telediagnostic television monitor. J Telemed Telecare 1996; 2 Suppl 1: 27-30.
218. Travers DA, Downs SM. Comparing the user acceptance of a computer system in two pediatric offices: A qualitative study. Proc AMIA Symp 2000: 853-7.
219. Kaplan B. Objectification and negotiation in interrupting clinical images: Implications for computer-based patient records. Artif Intell Med 1995; 7: 439-54.
220. Kaplan B, Lundsgaarde HP. Toward an evaluation of a clinical imaging system: Identifying benefits Methods Inf Med 1996; 35: 221-9.
221. Billinghurst M, Hirokazu K. Collaborative augmented reality. Commun ACM 2002; 45 (7): 64-70.
222. Moehr JR. Evaluation: Salvation or nemesis of medical informatics? Comput Biol Med 2002; 32 (3): 113-25.
223. May CR, Mort M, Mair F, EllisNT, Gask L. Evaluation of new technologies in health care systems: What's the context? Health Inform J 2000; 6 (2): 67-70.
224. Harrison R, MacFarlane A, Wallace P. Implementation of telemedicine: The problem of evaluation. J Telemed Telecare 2002; 8 Suppl 2: 39-40.
225. Pinch TJ, Bijker WE. The social construction of facts and artifacts: Or how the sociology of science and the sociology of technology might benefit each other. In: Bijker W, Hughes T, Pinch T (eds). The social construction of technological systems. Cambridge, Mass: MIT Press; 1987. pp. 17-50.
226. Nøhr C. The evaluation of expert diagnostic systems - how to assess outcomes and quality parameters? Artif Intell Med 1994; 6 (2): 23-35.
227. Grémy F, Bonnin M. Evaluation of automatic health information systems. In: van Gennip EMS, Talmon JL, editors. Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995. pp. 9-20.
228. Jørgensen T. Measuring effects. In: van Gennip EMS, Talmon JL (eds). Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995. pp. 99-109.
229. Miller RA. Medical diagnostic decision support systems - past, present and future - a threaded bibliography and brief commentary. JAMIA 1994; 1: 8-27.
230. Heathfield HA, Buchan IE. Current evaluations of information technology in health care are often inadequate. BMJ 1996; 313 (7063): 1008.
231. Gosling AS, Westbrook JI, Cojera EW. Variation in the use of online clinical evidence: A qualitative analysis. Int J Med Inform 2003 (1): 1-16.
232. Friedman RB, Gustafson DH. Computers in clinical medicine, a critical review. Comput Biomed Res 1977; 10 (3): 199-204.
233. Bankowitz RA, Lave JR, McNeil MA. A method for assessing the impact of a computer-based decision support system on health care outcomes. Methods Inf Med 1992; 31 (1): 3-10.
234. Goldberg HS, Morales A, Gottlieb L, Meador L, Safran C. Reinventing patient-centered computing for the twenty-first century. In: Patel V, Rogers R, Haux R (eds.). Medinfo 2001. Amsterdam: IOS Press; 2001. pp. 1455-58.
235. Kreps GL. Evaluating new health information technologies: Expanding the frontiers of health care delivery and health promotion. Stud Health Technol Inform 2002; 80: 205-12.
236. Pugh GE, Tan JK. Computerized databases for emergency care: What impact on patient care? Methods Inf Med 1994; 33 (5): 507-13.
237. Banta H. Embracing or rejecting innovations: Clinical diffusion of health care technology. In: Anderson JG, Jay SJ (eds). Use and impact of computers in clinical care. New York: Springer; 1987. pp. 132-60.
238. Miller RA. Why the standard view is standard: People, not machines, understand patients' problems. J Med Philos 1990; 15 (6): 581-91.
239. Harris KD, Donaldson JF, Campbell JD. Introducing computer-based telemedicine in three rural Missouri counties. J End User Comput 2001; 13 (4): 26.
240. Penrod LE, Gadd CS. Attitudes of academic-based and community-based physicians regarding EMR use during outpatient encounters. Proc AMIA Symp 2001: 529-32.
241. Ash JS, Gorman PN, Hersh WR. Physician order entry in U.S. Hospitals. Proc AMIA Symp 1998: 235-9.
242. Ash JS, Gorman PN, Hersh WR, Poulson SP. Perceptions of house officers who use physician order entry. Proc AMIA Symp 1999: 471-5.
243. Ash J, Gorman P, Lavelle M, Lyman J, Fournier L. Investigating physician order entry in the field: Lessons learned in a multi-centered study. In: Patel V, Rogers R, Haux R (eds). Medinfo 2001. Amsterdam: IOS Press; 2001. pp. 1107-11.
244. Ash JS, Gorman PN, Lavelle M, Stavri PZ, Lyman J, Fournier L et al. Perceptions of physician order entry: Results of a cross-site qualitative study. Methods Inf Med 2003; 42 (4): 313-23.
245. Ash JS, Gorman PN, Lavelle M, Payne TH, Massaro TA, Frantz GL et al. A cross-site qualitative study of physician order entry. JAMIA 2003; 10 (2): 188-200.
246. Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a sucessful CPOE implementation. JAMIA 2003; 10 (3): 229-34.
247. Coombs CR, Doherty NF, Loan-Clark J. The importance of user ownership and positive user attitudes in the successful adoption of community information systems. J End User Comput 2001; 13 (4): 5.
248. Lau F, Doze S, Vincent D, Wilson D, Noseworthy T, Hayward R. Patterns of improvisation for evidence-based practice in clinical settings. Inform Tech People 1999; 12 (3): 287-303.
249. Isaacs S. The power distance between users of information technology and experts and satisfaction with the information system. In: Patel V, Rogers R, Haux R (eds). Medinfo 2001. Amsterdam: IOS Press; 2001. pp. 1155-7.
250. Talmon J, Enning J, Castãneda G, Eurlings F, Hoyer D, Nykänen P et al. The VATAM guidelines. Int J Med Inform 1999; 56 (1-3): 107-15.
251. Suchman LA. Plans and situated actions: The problem of human-machine interaction. Cambridge: Cambridge University Press; 1987.
252. Suchman L. Representations of work. Commun ACM 1995; 38 (9): 33-4.
253. Butler KA, Esposito C, Hebron R. Connecting the design of software to the design of work. Commun ACM 1999; 42 (1): 38-46.
254. Fafchamps D, Young CY, Tang PC. Modelling work practices: Input to the design of a physician's workstation. In: Clayton PD (ed.). Proc Symp Comput Appl Med Care. New York: McGraw Hill; 1991. pp. 788-92.
255. Holtzblatt K, Bryer HR. Apprenticing with the customer. Commun ACM 1995; 28 (1): 45-52.
256. Artif Intell Med 1995; 7.
257. Greenbaum J, Kyng M. Design at work: Cooperative design of computer systems. Hillsdale, NJ: Lawrence Erlbaum Associates; 1991.
258. Winograd T, Flores F. Understanding computers and cognition: A new foundation for design. Norwood, NJ: Ablex; 1986.
259. Timpka T, Hedblom P, Holmgren J. Action design: Using an object oriented environment for group process development of medical software. In: Timmers T (ed.). Software engineering in medical informatics. Amsterdam: Elsevier; 1991, pp. 151-65.
260. Sjöberg C, Timpka T. Participatory design of information systems in health care. J Am Health Inform Manag Assn 1998; 5 (2): 177-83.
261. Musen MA. Architectures for architects. Methods Inf Med 1993; 32: 12-3.
262. Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: Cognitive approaches to evaluation of information systems and user interfaces. Proc AMIA Symp 1997: 218-22.
263. Kushniruk AW, Patel VL. Cognitive evaluation of decision making processes and assessment of information technology in medicine. Int J Med Inform 1998; 51 (2-3): 83-90.
264. Patel V, Kaufman D. Medical informatics and the science of cognition. JAMIA 1998; 5 (6): 493-502.
265. Beuscart-Zéphir MC, Anceaux AF, Crinquette C, Renarda JM. Integrating users' activity modeling in the design and assessment of hospital electronic patient records. Int J Med Inform 2001; 64 (2-3): 157-71.
266. Kaplan B, Drickamer M, Marattoli RA. Deriving design recommendations through discount usability engineering: Ethnographic observation and thinking-aloud protocol in usability testing for computer-based teaching cases. Proc AMIA Symp 2003: 346-50.
267. Gardner RM, Lundsgaarde HP. Evaluation of user acceptance of a clinical expert system. JAMIA 1994; 1 (6): 428-38.
268. Saranummi N. Supporting system development with technology assessment. In: van Gennip EMS, Talmon JL (eds). Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995. pp. 35-44.
269. Faber M. Design and introduction of an electronic patient record: How to involve users? Methods Inf Med 2003; 42 (4): 371-5.
270. Smith ME. Success rates for different types of organizational change. Performance Improvement 2002; 41 (1): 26-33.
271. Lau F, Hebert M. Experiences from health informatics information system implementation projects reported in Canada between 1991 and 1997. J End User Comput 2001; 13 (4): 17.
272. Dowling AF. Do hospital staff interfere with computer system implementation? Health Care Manag Rev 1980; 5: 23-32.
273. Amatayakul M. The state of the computer-based patient record. JAHIMA 1998; 69: 34.
274. Dambro MR, Weiss BD, McClure C, Vuturo AF. An unsuccessful experience with computerized medical records in an academic medical centre. J Med Ed 1988; 63: 617-23.
275. Flowers S. Software failure, management failure: Amazing stories and cautionary tales. Wiley; 1986.
276. Chessare JB, Torok KE. Implementation of COSTAR in an academic group practice of general pediatrics. MD Comput 1993; 10 (1): 23-27.
277. Tonnesen AS, LeMaistre A, Tucker D. Electronic medical record implementation barriers encountered during implementation. Proc AMIA Symp 1999: 625-66.
278. Jeffcott M. Technology alone will never work: Understanding how organizational issues contribute to user neglect and information systems failure in healthcare. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; September 6-7,2001; Department of Health Policy and Management, Erasmus University, Rotterdam, The Netherlands; 2001. p. 35.
279. Zuiderent T Changing care and theory: A theoretical joyride as the safest journey. In: IT in Health Care: Sociotechnical Approaches. International Conference Proc.; September 6-7,2001; Department of Health Policy and Management, Erasmus University, Rotterdam,The Netherlands; 2001. p. 62.
280. Southon F, Sauer C, Dampney C. Information technology in complex health services: Organizational impediments to successful technology transfer and diffusion. JAMIA 1997; 4: 112-24.
281. Balka E. Getting the big picture: The macro-politics of information system development (and failure) in a Canadian hospital. Methods Inf Med 2003; 42 (4): 324-30.
282. Benyon-Davies P. The London ambulance service's computerised dispatch system: A case study in information system failure: University of Glamorgan Pontyprid; 1993.
283. Benyon-Davies P. Information systems 'failure': The case of the London ambulance service's computer aided despatch project. Eur J Inform Syst 1995; 4: 171-84.
284. Friedman CP, Wyatt JC. Publication bias in medical informatics. JAMIA 2001; 8 (2): 189-91.
285. Keen P. Information systems and organizational change. Commun ACM 1981; 24 (1): 24-33.
286. Kling R, Iacono S. The mobilization of support for computerization: The role of computerization movements. Soc Problems 1988; 35 (3): 226-43.
287. Stead WW. Matching the level of evaluation to a project's stage of development. JAMIA 1996; 3 (1): 92-94.
288. Forsythe DE. The construction of work in artificial intelligence. In: Hess DJ (eds). Studying those who study us: An anthropologist in the world of artificial intelligence. Stanford. Calif.: Stanford University Press; 2001. pp. 16-34.
289. Kaplan B. Evaluating informatics applications - clinical decision support systems literature review. Int J Med Inform 2001; 64 (1): 15-37.
290. Markus M, Power, politics, and MIS implementation. Commun ACM 1983; 26: 430-44.
291. Barsade SG, Brief AP, Spartaro S. The affective revolution of organizational behavior: The emergence of a paradigm. In: Greenberg J (ed). OB: The state of the science. Second ed. Hillsdale, NJ: L. Erlbaum Associates; 2002.
292. Forsythe DE, Buchanan B, Osheroff J, Miller R. Expanding the concept of medical information: An obsevational study of physicians' information needs. Comput Biomed Res 1992; 25: 181-200.
293. Kaplan B, Farzanfar R, Friedman RH. Personal relationships with an intelligent interactive telephone health behavior advisor system: A multimethod study using surveys and ethnographic interviews. Int J Med Inform 2003; 71 (1): 33-41.
294. Tierney WM, Overhage JM, McDonald CJ. A plea for controlled trials in medical informatics. JAMIA 1994; 1 (4): 353-5.
295. Wyatt JC, Wyatt SM. When and how to evaluate health information systems? Int J Med Inform 2003; 69 (2-3): 251-9.
296. Mair F, Whitten P. Systematic review of studies of patient satisfaction with telemedicine. BMJ 2000; 320 (7248): 1517-20.
297. Banta D. The development of health technology assessment. Health Policy 2003; 63 (2): 121-32.
298. van der Loo RP. Overview of published assessment and evaluation studies. In: van Gennip EMS, Talmon JL (eds.). Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995. pp. 261-82.
299. Heathfield HA, Peel V, Hudson P, Kay S, Mackay L, Marley T et al. Evaluating large scale health information systems: From practice towards theory. Proc AMIA Symp 1997: 116-20.
300. Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: Barriers and challenges. BMJ 1998; 316 (7149): 1959-61.
301. Paré G. Implementing clinical information systems: A multiple-case study within a US hospital. Health Ser Manag Res 2002; 15 (2): 71-92.
302. Kaplan B, Maxwell JA. Qualitative research methods for evaluating computer information systems. In: Anderson JG, Aydin CE, Jay SJ (eds). Evaluating health care information systems; approaches and applications. Thousand Oaks, CA: Sage; 1994. pp. 45-68.
303. Sawyer S, Eschenfelder KR. Social informatics: Perspectives, examples and trends. In: Cronon B (eds). Annual review of information technology. Medford, NJ; 2002. pp. 427-65.