Estimating the minimum number of judges required for test-centred standard setting on written assessments. Do discussion and iteration have an influence?

Advances in Health Sciences Education

Volumn 13, Issue 1, 2008, Pages 11-24

  Fowell, S L ^a   Fewtrell, R ^a   McLaughlin, P J ^a  

^a UNIVERSITY OF LIVERPOOL   (United Kingdom)

Author keywords

Angoff method; Assessment; Ebel method; Extended matching questions; Short answer questions; Standard setting

Indexed keywords

ARTICLE; CURRICULUM; EDUCATION; HUMAN; MEDICAL EDUCATION; METHODOLOGY; REPRODUCIBILITY; STANDARD; UNITED KINGDOM;

CURRICULUM; EDUCATION, MEDICAL; EDUCATIONAL MEASUREMENT; ENGLAND; HUMANS; REPRODUCIBILITY OF RESULTS;

EID: 38349154210     PISSN: 13824996     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10459-006-9027-1     Document Type: Article

References (26)

1. Angoff W.H. (1971). Scales, norms, and equivalent scores. In: Thorndike R.L. (Eds.), Educational Measurement. American Council on Education, Washington D.C., pp. 508-600
2. Berk R.A. (1986). A consumer's guide to setting performance standards on criterion-referenced tests Review of Education Research 56:137-172
3. Berk R.A. (1996). Standard setting: The next generation: Where few psychometricians have gone before Applied Measurement in Education 9:215-235
4. Brandon P.R. (2004). Conclusions about frequently studied modified Angoff standard-setting topics Applied Measurement in Education 17:59-88
5. Brennan R.L. (2001). Generalizability Theory. Springer-Verlag, New York
6. Brennan R.L., Gao X., Colton D. (1995). Generalizability analyses of work keys listening and writing tests Educational and Psychological Measurement 55:157-176
7. Brennan R.L., Lockwood R.E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory Applied Psychological Measurement 4:219-240
8. Case, S.M. & Swanson, D.B. (1996). Constructing Written Test Questions for the Basic and Clinical Sciences: National Board of Medical Examiners
9. Cizek G.J. (1996). Standard setting guidelines Educational Measurement: Issues and Practice 15:13-21
10. Cusimano M.D. (1996). Standard setting in medical education Academic Medicine 71:S112-S120
11. Downing S.M., Lieska N.G., Raible M.D. (2003). Establishing passing standards for classroom achievement tests in medical education: A comparative study of four methods Academic Medicine 78:S85-S87
12. Ebel R.L. (1979). Determination of the passing score. In: Ebel R.L. (Eds.), Essentials of Educational Measurement (3rd ed.). Prentice-Hall, Englewood Cliffs, New Jersey, pp 3337-3342
13. Hambleton R.K., Powell S. (1983). A framework for viewing the process of standard setting Evaluation and the Health Professions 6:3-24
14. Hurtz G.M., Hertz N.R. (1999). How many raters should be used for establishing cutoff scores with the Angoff method? A generalizability theory study Educational and Psychological Measurement 59:885-897
15. Kaufman D.M., Mann K.V., Muijtjens A.M.M., van der Vleuten C.P.M. (2000). A comparison of standard-setting procedures for an OSCE in undergraduate medical education Academic Medicine 75:267-271
16. Lowry S. (1993). Assessment of Students British Medical Journal 306:51-54
17. Maurer T.J., Alexander R.A., Callahan C.M., Bailey J.J., Dambrot F.H. (1991). Methodological and psychometric issues in setting cutoff scores using the Angoff method Personnel Psychology 44:235-262
18. Morrison H., McNally H., Wylie C., McFaul P., Thompson W. (1996). The passing score in the objective structured clinical examination Medical Education 30:345-348
19. Muijtjens A.M.M., Hoogenboom R.J.I., Verwinjen G.M., van der Vleuten C.P. (1998). Relative or absolute standards in assessing medical knowledge using progress tests Advances in Health Sciences Education 3:81-87
20. Norcini J.J. (2003). Setting standards on educational tests Medical Education 37:464-469
21. Norcini J.J., Lipner R.S., Langdon L.O., Strecker C.A. (1987). A comparison of three variations on a standard-setting method Journal of Educational Measurement 24:56-64
22. Searle J. (2000). Defining competency - the role of standard setting Medical Education 34:363-366
23. Streiner, D.L. & Norman, G.R. (1995). Reliability (Chapter 8). In Health Measurement Scales: A Practical Guide to their Development and Use, 2nd edn., pp. 104-127. Oxford University Press
24. Verhoeven B.H., van der Steeg A.F., Scherpbier A.J., Muijtjens A.M., Verwijnen G.M., van der Vleuten C.P. (1999). Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judges Medical Education 33:832-837
25. Verhoeven B.H., Verwijnen G.M., Muijtjens A.M.M., Scherpbier A.J., van der Vleuten C.P.M. (2002). Panel expertise for an Angoff standard setting procedure in progress testing: Item writers compared to recently graduated students Medical Education 36:860-867
26. Wilkinson T.J., Newble D.I., Frampton C.M. (2001). Standard setting in an objective structured clinical examination: Use of global ratings of borderline performance to determine the passing score Medical Education 35:1043-1049