Should essays and other open-ended-Type questions retain a place in written summative assessment in clinical medicine?

BMC Medical Education

Volumn 14, Issue 1, 2014, Pages

  Hift, Richard J ^a  

^a UNIVERSITY OF KWAZULU NATAL   (South Africa)

Author keywords

Assessment; Conceptual change; Essay; Mental models; MEQ; Multiple choice

Indexed keywords

BIOLOGY; CLINICAL MEDICINE; CLINICAL PRACTICE; CONSTRUCT VALIDITY; COST EFFECTIVENESS ANALYSIS; EXERCISE; HUMAN; HUMAN EXPERIMENT; LEARNING THEORY; MATHEMATICS; MULTIPLE CHOICE TEST; PROBLEM SOLVING; RELIABILITY; THEORETICAL MODEL; CLINICAL COMPETENCE; COMPARATIVE STUDY; EDUCATION; EVIDENCE BASED MEDICINE; FEMALE; INTERNAL MEDICINE; MALE; MEDICAL EDUCATION; PROCEDURES; QUESTIONNAIRE; REPRODUCIBILITY; WRITING;

CLINICAL COMPETENCE; EDUCATION, MEDICAL, UNDERGRADUATE; EDUCATIONAL MEASUREMENT; EVIDENCE-BASED MEDICINE; FEMALE; HUMANS; INTERNAL MEDICINE; MALE; REPRODUCIBILITY OF RESULTS; SURVEYS AND QUESTIONNAIRES; WRITING;

EID: 84964313380     PISSN: None     EISSN: 14726920     Source Type: Journal    
DOI: 10.1186/s12909-014-0249-2     Document Type: Review

References (120)

1. Siemens G: Connectivism: Learning as Network-Creation. [http://www. elearnspace.org/Articles/networks.htm]
2. Siemens G: Connectivism: A learning theory for the digital age. Int J Instr Technol Distance Learn 2005, 2:3-10.
3. Perkins DN, Salomon G: Learning transfer. In International Encyclopaedia of adult education and training. 2nd edition. Edited by Tuijnman AC. Tarrytown, NY: Pergamon Press; 1996:422-427.
4. Haskell EH: Transfer of learning: Cognition, Instruction, and Reasoning. New York: Academic Press; 2001.
5. Spelke E: Initial Knowledge: Six Suggestions. In Cognition on cognition. Edited by Mehler J, Franck S. Cambridge, MA US: The MIT Press; 1995:433-447.
6. Barnett SM, Ceci SJ: When and where do we apply what we learn-A taxonomy for far transfer. Psychol Bull 2002, 128:612-637.
7. Brown AL: Analogical Learning and Transfer: What Develops-In Similarity and Analogical Reasoning. Edited by Vosniadou S, Ortony A. New York: Cambridge University Press; 1989:369-412.
8. Gick ML, Holyoak KJ: Schema Induction and Analogical Transfer. New York, NY US: Psychology Press; 2004.
9. Bloom BS: The Cognitive Domain. In Taxonomy of Educational Objectives, Handbook I. New York: David McKay Co Inc; 1956.
10. Anderson LW, Krathwohl DR, Airasian PW, Cruikshank KA, Mayer RE, Pintrich PR, Raths J, Wittrock MC: A Taxonomy for Learning, Teaching, and Assessing: A revision of Bloom's Taxonomy of Educational Objectives. New York: Longman; 2001.
11. Anderson LW, Sosniak LA: Bloom's Taxonomy: A Forty-year Retrospective. In Ninety-Third yearbook of the National Society for the Study of Education: Part II. Edited by Anderson LW, Sosniak LA. Chicago IL: University of Chicago Press; 1994.
12. Conklin J: A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Educ Horiz 2005, 83:154-159.
13. Haladyna TM, Downing SM: A taxonomy of multiple-choice item-writing rules. Appl Meas Educ 1989, 2:37-51.
14. Haladyna TM: Developing and Validating Multiple-choice Test Items. Mahwah NJ: L. Erlbaum Associates; 1999.
15. Miller GE: The assessment of clinical skills/competence/performance. Acad Med 1990, 65:S63-S67.
16. Dreyfus HL, Dreyfus SE, Athanasiou T: Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer. New York: Free Press; 1986.
17. Norcini JJ, Swanson DB, Grosso LJ, Webster GD: Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence. Med Educ 1985, 19:238-247.
18. Taconnat L, Froger C, Sacher M, Isingrini M: Generation and associative encoding in young and old adults: The effect of the strength of association between cues and targets on a cued recall task. Exp Psychol 2008, 55:23-30.
19. Baddeley AD, Eysenck MW, Anderson M: Memory. New York: Psychology Press; 2010.
20. Karpicke J, Grimaldi P: Retrieval-based learning: A perspective for enhancing meaningful learning. Educ Psychol Rev 2012, 24:401-418.
21. Rohrer D, Pashler H: Recent research on human learning challenges conventional instructional strategies. Educ Res 2010, 39:406-412.
22. Smith MA, Roediger HL III, Karpicke JD: Covert retrieval practice benefits retention as much as overt retrieval practice. J Exp Psychol Learn Mem Cogn 2013, 39:1712-1725.
23. McDermott KB, Agarwal PK, D-Antonio L, Roediger HL III, McDaniel MA: Both multiple-choice and short-Answer quizzes enhance later exam performance in middle and high school classes. J Exp Psychol Appl 2014, 20:3-21.
24. Cutting MF, Saks NS: Twelve tips for utilizing principles of learning to support medical education. Med Teach 2012, 34:20-24.
25. Schuwirth LWT, Van der Vleuten CPM: General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach 2011, 33:783-797.
26. Van der Vleuten CP, Schuwirth LW: Assessing professional competence: from methods to programmes. Med Educ 2005, 39:309-317.
27. Schuwirth L, Colliver J, Gruppen L, Kreiter C, Mennin S, Onishi H, Pangaro L, Ringsted C, Swanson D, Van der Vleuten C, Wagner-Menghin M: Research in assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011, 33:224-233.
28. Schuwirth LWT, Van der Vleuten CPM: Programmatic assessment and Kane's validity perspective. Med Educ 2012, 46:38-48.
29. Case SM, Swanson DB: Constructing Written Test Questions for the Basic and Clinical Sciences. 3rd edition. Philadelphia: National Board of Medical Examiners; 2002.
30. Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, Galbraith R, Hays R, Kent A, Perrott V, Roberts T: Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011, 33:206-214.
31. Shepard LA: The role of assessment in a learning culture. Educ Res 2000, 29:4-14.
32. Coburn CA, Yerkes RM: A study of the behavior of the crow corvus americanus Aud. By the multiple choice method. J Anim Behav 1915, 5:75-114.
33. Yerkes RM, Coburn CA: A study of the behavior of the pig Sus Scrofa by the multiple choice method. J Anim Behav 1915, 5:185-225.
34. Brown W, Whittell F: Yerkes' multiple choice method with human adults. J Comp Psychol 1923, 3:305-318.
35. Yerkes RM: A New method of studying the ideational behavior of mentally defective and deranged as compared with normal individuals. J Comp Psychol 1921, 1:369-394.
36. Davidson C: Davidson CN: Now You See It: How the Brain Science of Attention Will Transform the Way We Live, Work, and Learn. New York: Viking Press; 2011:113-122.
37. Frederiksen JR, Collins A: A Systems Approach to Educational Testing. Technical Report No. 2. New York: Center for Technology in Education; 1990.
38. Guthrie JT: Testing higher level skills. J Read 1984, 28:188-190.
39. Nickerson RS: New directions in educational assessment. Educ Res 1989, 18:3-7.
40. Stratford P, Pierce-Fenn H: Modified essay question. Phys Ther 1985, 65:1075-1079.
41. Wass V, Van der Vleuten C, Shatzer J, Jones R: Assessment of clinical competence. Lancet 2001, 357:945.
42. Rotfield H: Are we teachers or job trainers-Acad Mark Sci Q 1998, 2:2.
43. Crocker L, Algina J: Introduction to Classical & Modern Test Theory. Fort Worth, TX: Holt, Rinehart and Winston, Inc.; 1986.
44. Angoff W: Test reliability and effective test length. Psychometrika 1953, 18:1-14.
45. Palmer EJ, Devitt PG: Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions-Research paper. BMC Med Educ 2007, 7:49-49.
46. Feletti GI, Smith EK: Modified essay questions: Are they worth the effort-Med Educ 1986, 20:126-132.
47. Palmer EJ, Duggan P, Devitt PG, Russell R: The modified essay question: its exit from the exit examination-Med Teach 2010, 32:e300-e307.
48. Schuwirth LW, Van der Vleuten CPM: Different written assessment methods: what can be said about their strengths and weaknesses-Med Educ 2004, 38:974-979.
49. Lukhele R, Thissen D, Wainer H: On the relative value of multiple-choice, constructed response, and examinee-selected items on two achievement tests. J Educ Meas 1994, 31:234-250.
50. Wainer H, Thissen D: Combining multiple-choice and constructed-response test scores: Toward a Marxist theory of test construction. Appl Meas Educ 1993, 6:103-118.
51. Facione PA: The California Critical Thinking Skills Test-College Level. Technical Report #1. Experimental Validation and Content Validity. Millbrae CA: California Academic Press; 1990.
52. Facione PA, Facione NC, Blohm SW, Giancarlo CAF: The California Critical Thinking Skills Test [Revised]. In Millbrae CA: California Academic Press; 2007.
53. Rodriguez MC: Construct equivalence of multiple-choice and constructed-response items: A random effects synthesis of correlations. J Educ Meas 2003, 40:163-184.
54. Falk B, Ancess J, Darling-Hammond L: Authentic Assessment in Action: Studies of Schools and Students at Work. United States of America: Teachers College Press; 1995.
55. Rethans JJ, Norcini JJ, Baron-Maldonado M, Blackmore D, Jolly BC, LaDuca T, Lew S, Page GG, Southgate LH: The relationship between competence and performance: implications for assessing practice performance. Med Educ 2002, 36:901-909.
56. Wilkinson TJ, Frampton CM: Comprehensive undergraduate medical assessments improve prediction of clinical performance. Med Educ 2004, 38:1111-1116.
57. Baker EL: Standards for Educational and Psychological Testing. Sage Publications, Inc; 2012.
58. Eignor DR: The Standards for Educational and Psychological Testing. In APA Handbook of Testing and Assessment in Psychology, Vol 1: Test Theory and Testing and Assessment in Industrial and Organizational Psychology. Edited by Geisinger KF, Bracken BA, Carlson JF, Hansen J-IC, Kuncel NR, Reise SP, Rodriguez MC. Washington, DC, US: American Psychological Association; 2013:245-250.
59. Eignor DR: Standards for the development and use of tests: The Standards for Educational and Psychological Testing. Eur J Psychol Assess 2001, 17:157-163.
60. Downing SM: Validity: on the meaningful interpretation of assessment data. Med Educ 2003, 37:830.
61. Messick S: The interplay of evidence and consequences in the validation of performance assessments. Educ Res 1994, 23:13-23.
62. Kuechler WL, Simkin MG: Why is performance on multiple-choice tests and constructed-response tests Not more closely related-Theory and an empirical test. Decis Sci J Innov Educ 2010, 8:55-73.
63. Norman GR, Smith EK, Powles AC, Rooney PJ: Factors underlying performance on written tests of knowledge. Med Educ 1987, 21:297-304.
64. Bacon DR: Assessing learning outcomes: A comparison of multiple-choice and short-Answer questions in a marketing context. J Mark Educ 2003, 25:31-36.
65. Kastner M, Stangla B: Multiple choice and constructed response tests: Do test format and scoring matter-Procedia-Social and Behav Sci 2011, 12:263-273.
66. Nichols P, Sugrue B: The lack of fidelity between cognitively complex constructs and conventional test development practice. Educ Measurement: Issues Pract 1999, 18:18-29.
67. Bennett RE, Rock DA, Wang M: Equivalence of free-response and multiple-choice items. J Educ Meas 1991, 28:77-92.
68. Bridgeman B, Rock DA: Relationships among multiple-choice and open-ended analytical questions. J Educ Meas 1993, 30:313-329.
69. Thissen D, Wainer H: Are tests comprising both multiple-choice and free-response items necessarily less unidimensional. J Educ Meas 1994, 31:113.
70. Lissitz RW, Xiaodong H, Slater SC: The contribution of constructed response items to large scale assessment: measuring and understanding their impact. J Appl Testing Technol 2012, 13:1-52.
71. Traub RE, Fisher CW: On the equivalence of constructed-response and multiple-choice tests. Appl Psychol Meas 1977, 1:355-369.
72. Martinez ME: Cognition and the question of test item format. Educ Psychol 1999, 34:207-218.
73. Hee-Sun L, Liu OL, Linn MC: Validating measurement of knowledge integration in science using multiple-choice and explanation items. Appl Meas Educ 2011, 24:115-136.
74. Wilson M, Wang W-C: Complex composites: Issues that arise in combining different modes of assessment. Appl Psychol Meas 1995, 19:51-71.
75. Ercikan K, Schwartz RD, Julian MW, Burket GR, Weber MM, Link V: Calibration and scoring of tests with multiple-choice and constructedresponse item types. J Educ Meas 1998, 35:137-154.
76. Epstein ML, Lazarus AD, Calvano TB, Matthews KA, Hendel RA, Epstein BB, Brosvic GM: Immediate feedback assessment technique promotes learning and corrects inaccurate first responses. Psychological Record 2002, 52:187-201.
77. Schuwirth LWT, Van der Vleuten CPM: Programmatic assessment: From assessment of learning to assessment for learning. Med Teach 2011, 33:478-485.
78. Bridgeman B, Morgan R: Success in college for students with discrepancies between performance on multiple-choice and essay tests. J Educ Psychol 1996, 88:333-340.
79. Bleske-Rechek A, Zeug N, Webb RM: Discrepant performance on multiple-choice and short answer assessments and the relation of performance to general scholastic aptitude. Assessment Eval Higher Educ 2007, 32:89-105.
80. Hakstian AR: The Effects of Type of Examination Anticipated on Test Preparation and Performance. J Educ Res 1971, 64:319.
81. Scouller K: The influence of assessment method on Students' learning approaches: multiple choice question examination versus assignment essay. High Educ 1998, 35:453-472.
82. Thomas PR, Bain JD: Contextual dependence of learning approaches: The effects of assessments. Human Learning: J Pract Res Appl 1984, 3:227-240.
83. Watkins D: Factors influencing the study methods of Australian tertiary students. High Educ 1982, 11:369-380.
84. Minbashian A, Huon GF, Bird KD: Approaches to studying and academic performance in short-essay exams. High Educ 2004, 47:161-176.
85. Yonker JE: The relationship of deep and surface study approaches on factual and applied test-bank multiple-choice question performance. Assess Eval Higher Educ 2011, 36:673-686.
86. Joughin G: The hidden curriculum revisited: A critical review of research into the influence of summative assessment on learning. Assess Eval Higher Educ 2010, 35:335-345.
87. Scouller KM, Prosser M: Students' experiences in studying for multiple choice question examinations. Stud High Educ 1994, 19:267.
88. Hadwin AF, Winne PH, Stockley DB, Nesbit JC, Woszczyna C: Context moderates students' self-reports about how they study. J Educ Psychol 2001, 93:477-487.
89. Birenbaum M: Assessment and instruction preferences and their relationship with test anxiety and learning strategies. High Educ 2007, 53:749-768.
90. Birenbaum M: Assessment preferences and their relationship to learning strategies and orientations. High Educ 1997, 33:71-84.
91. Smith SN, Miller RJ: Learning approaches: examination type, discipline of study, and gender. Educ Psychol 2005, 25:43-53.
92. Rabinowitz HK, Hojat M: A comparison of the modified essay question and multiple choice question formats: Their relationship to clinical performance. Fam Med 1989, 21:364-367.
93. Paterson DG: Do new and old type examinations measure different mental functions-School Soc 1926, 24:246-248.
94. Schuwirth LW, Verheggen MM, Van der Vleuten CPM, Boshuizen HP, Dinant GJ: Do short cases elicit different thinking processes than factual knowledge questions do-Med Educ 2001, 35:348-356.
95. Tanner DE: Multiple-choice items: Pariah, panacea or neither of the above-Am Second Educ 2003, 31:27.
96. Cilliers FJ, Schuwirth LW, van der Vleuten CP: Modelling the pre-Assessment learning effects of assessment: evidence in the validity chain. Med Educ 2012, 46:1087-1098.
97. Jonassen DH, Strobel J: Modeling for Meaningful Learning. In Engaged Learning with Emerging Technologies. Edited by Hung D. Amsterdam: Springer; 2006:1-27.
98. Derry SJ: Cognitive schema theory in the constructivist debate. Educ Psychol 1996, 31:163-174.
99. Kim MK: Theoretically grounded guidelines for assessing learning progress: cognitive changes in Ill-structured complex problem-solving contexts. Educ Technol Res Dev 2012, 60:601-622.
100. Mayer RE: Models for Understanding. Rev Educ Res 1989, 59:43-64.
101. Jonassen D, Strobel J, Gottdenker J: Model building for conceptual change. Interact Learn Environ 2005, 13:15-37.
102. Jonassen DH: Tools for representing problems and the knowledge required to solve them. Edited by Tergan S-O, Keller T. Berlin, Heidelberg: Springer; 2005:82-94.
103. Bogard T, Liu M, Chiang Y-H: Thresholds of knowledge development in complex problem solving: A multiple-case study of advanced Learners' cognitive processes. Educ Technol Res Dev 2013, 61:465-503.
104. Van Gog T, Ericsson KA, Rikers RMJP: Instructional design for advanced learners: establishing connections between the theoretical frameworks of cognitive load and deliberate practice. Educ Technol Res Dev 2005, 53:73-81.
105. Schmidt HG, Norman GR, Boshuizen HP: A cognitive perspective on medical expertise: Theory and implication. Acad Med 1990, 65:611-621.
106. Schmidt HG, Rikers RMJP: How expertise develops in medicine: knowledge encapsulation and illness script formation. Med Educ 2007, 41:1133-1139.
107. Norman G, Young M, Brooks L: Non-Analytical models of clinical reasoning: The role of experience. Med Educ 2007, 41:1140-1145.
108. Ericsson KA, Prietula MJ, Cokely ET: The Making of an Expert. Harv Bus Rev 2007, 85:114-121.
109. Hoffman RR: How Can Expertise be Defined-Implications of Research From Cognitive Psychology. In Exploring Expertise. Edited by Williams R, Faulkner W, Fleck J. Edinburgh: University of Edinburgh Press; 1996:81-100.
110. Norman GR: Problem-solving skills, solving problems and problem-based learning. Med Educ 1988, 22:279-286.
111. Ifenthaler D, Seel NM: Model-based reasoning. Comput Educ 2013, 64:131-142.
112. Jonassen D: Using cognitive tools to represent problems. J Res Technol Educ 2003, 35:362-381.
113. Mayer RE, Wittrock MC: Problem-Solving Transfer. In Handbook of Educational Psychology. Edited by Berliner DC, Calfee RC. New York, NY: Macmillan Library Reference USA; 1996:47-62.
114. Zhang J, Norman DA: Representations in distributed cognitive tasks. Cogn Sci 1994, 18:87-122.
115. Simon HA: Information-Processing Theory of Human Problem Solving. In Handbook of Learning & Cognitive Processes: V Human Information. Edited by Estes WK. Oxford England: Lawrence Erlbaum; 1978:271-295.
116. Jensen JL, Woodard SM, Kummer TA, McDaniel MA: Teaching to the test-or testing to teach: exams requiring higher order thinking skills encourage greater conceptual understanding. Educ Psychol Rev 2014, 26:307-329.
117. Cohen-Schotanus J, Van der Vleuten CPM: A standard setting method with the best performing students as point of reference: practical and affordable. Med Teach 2010, 32:154-160.
118. Desjardins I, Touchie C, Pugh D, Wood TJ, Humphrey-Murto S: The impact of cueing on written examinations of clinical decision making: A case study. Med Educ 2014, 48:255-261.
119. Pretz JE, Naples AJ, Sternberg RJ: Recognizing, Defining, and Representing Problems. In The Psychology of Problem Solving. Edited by Davidson JE, Sternberg RJ. New York, NY US: Cambridge University Press; 2003:3-30.
120. Schuwirth LWT, Schuwirth LWT, Van der Vleuten CPM: ABC of learning and teaching in medicine: written assessment. BMJ: British Med J (International Edition) 2003, 326:643-645.