Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: Meta-regression of six UK longitudinal studies

BMC Medicine

Volumn 11, Issue 1, 2013, Pages

  McManus, IC ^a,b   Dewberry, Chris ^c   Nicholson, Sandra ^d   Dowell, Jonathan S ^e   Woolf, Katherine ^a   Potts, Henry W W ^a  

^a ROYAL FREE AND UNIVERSITY COLLEGE MEDICAL SCHOOL   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY OF LONDON   (United Kingdom)

^d Department of General Practice   (United Kingdom)

^e UNIVERSITY OF DUNDEE   (United Kingdom)

Author keywords

Aptitude tests; Criterion related construct validity; Educational attainment; Grade inflation; Markov Chain Monte Carlo algorithm; Medical student selection; Postgraduate performance; Range restriction; Right censorship; Undergraduate performance

Indexed keywords

ACADEMIC ACHIEVEMENT; APTITUDE TEST; ARTICLE; COHORT ANALYSIS; CONSTRUCT VALIDITY; CORRELATION ANALYSIS; EDUCATIONAL STATUS; HUMAN; INTELLIGENCE; LONGITUDINAL STUDY; MEASUREMENT ERROR; MEDICAL EDUCATION; MEDICAL SCHOOL; MEDICAL STUDENT; PHYSICIAN; POSTGRADUATE EDUCATION; POSTGRADUATE STUDENT; PREDICTIVE VALIDITY;

APTITUDE TESTS; EDUCATIONAL MEASUREMENT; GREAT BRITAIN; HUMANS; LONGITUDINAL STUDIES; MARKOV CHAINS; MODELS, STATISTICAL; MONTE CARLO METHOD; REPRODUCIBILITY OF RESULTS; SCHOOL ADMISSION CRITERIA; STUDENTS, MEDICAL;

EID: 84887663049     PISSN: None     EISSN: 17417015     Source Type: Journal    
DOI: 10.1186/1741-7015-11-243     Document Type: Article

References (68)

1. McManus IC, Powis DA, Wakeford R, Ferguson E, James D, Richards P. Intellectual aptitude tests and A levels for selecting UK school leaver entrants for medical school. BMJ 2005, 331:555-559. 10.1136/bmj.331.7516.555, 1200591, 16150766.
2. McManus IC, Ferguson E, Wakeford R, Powis D, James D. Predictive validity of the BioMedical Admissions Test (BMAT): an evaluation and case study. Med Teach 2011, 33:53-57. 10.3109/0142159X.2010.525267, 21182383.
3. McManus IC, Smithers E, Partridge P, Keeling A, Fleming PR. A levels and intelligence as predictors of medical careers in UK doctors: 20 year prospective study. BMJ 2003, 327:139-142. 10.1136/bmj.327.7407.139, 165701, 12869457.
4. McManus IC, Woolf K, Dacre J, Paice E, Dewberry C. The Academic Backbone: Longitudinal Continuities in Educational Achievement from Secondary School and Medical School to MRCP(UK) and the Specialist Register in UK Medical Students and Doctors. BMC Med 2013, 11:242.
5. Ghiselli EE, Campbell JP, Zedeck S. Measurement Theory for the Behavioral Sciences 1981, San Francisco: W H Freeman.
6. Streiner DL, Norman GR. Health Measurement Scales: a Practical Guide to their Development and Use 2008, Oxford, UK: Oxford University Press, 4.
7. The Concept of Validity: Revisions, New Directions, and Applications 2009, Charlotte, NC: Information Age Publishing, Lissitz RW.
8. Hunter JE, Schmidt FL, Le H. Implications of direct and indirect range restriction for meta-analysis methods and findings. J Appl Psychol 2006, 91:594-612.
9. Bagg DG. A-levels and university performance. Nature 1970, 225:1105-1108. 10.1038/2251105a0, 16056951.
10. General Medical Council Conference on Methods of Examination and Assessment, February 28, 1973 1973, London: General Medical Council, General Medical Council.
11. Burt C. Validating tests for personnel selection. BJP 1943, 34:1-19.
12. Pearson K. Mathematical contributions to the theory of evolution. XI: On the influence of natural selection on the variability and correlation of organs. Phil Trans R Soc Lond A 1903, 200:1-66.
13. Lawley D. A note on Karl Pearson's selection formulae. Proc R Soc Edinburgh: Sec A 1944, 62:28-30.
14. Rydberg S. Bias in Prediction: on Correction Methods 1963, Stockholm: Almqvist & Wiksell.
15. Lord FM, Novick MR. Statistical theories of Mental Test Scores 1968, Reading, MA, USA: Addison-Wesley.
16. Schmidt FL, Ones DS, Hunter JE. Personnel-selection. Annu Rev Psychol 1992, 43:627-670.
17. Callahan CA, Hojat M, Veloksi J, Erdmann JB, Gonnella JS. The predictive validity of three versions of the MCAT in relation to performance in medical school, residency, and licensing examinations: a longitudinal study of 36 classes of Jefferson Medical College. Acad Med 2010, 85:980-987. 10.1097/ACM.0b013e3181cece3d, 20068426.
18. Admissions to Higher Education Steering Group Fair Admissions to Higher Education: Recommendations for Good Practice 2004, Nottingham, UK: Department for Education and Skills Publications, [http://www.admissions-review.org.uk], Admissions to Higher Education Steering Group.
19. Heim AW. AH5 Group Test of High-Grade Intelligence 1968, Windsor: NFER-Nelson.
20. UKCAT UKCAT: 2006 Annual Report 2008, Nottingham: United Kingdom Clinical Aptitude Test, available at [http://www.ukcat.ac.uk/App_Media/uploads/pdf/Annual%20Report%202006.pdf], UKCAT.
21. McManus IC, Dewberry C, Nicholson S, Dowell J. The UKCAT-12 Study: Educational Attainment, Aptitude Test Performance, Demographic and Socio-Economic Contextual Factors as Predictors of First Year Outcome in a Collaborative Study of Twelve UK Medical Schools. BMC Med 2013, 11:244.
22. Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted?. Stat Med 2002, 21:1559-1573. 10.1002/sim.1187, 12111920.
23. McManus IC, Richards P. An audit of admission to medical school: 1 - Acceptances and rejects. Br Med J (Clin Res Ed) 1984, 289:1201-1204.
24. McManus IC, Richards P, Maitlis SL. Prospective study of the disadvantage of people from ethnic minority groups applying to medical schools in the United Kingdom. BMJ 1989, 298:723-726. 10.1136/bmj.298.6675.723, 1835995, 2496822.
25. McManus IC, Richards P, Winder BC, Sproston KA, Styles V. Medical school applicants from ethnic minorities: identifying if and when they are disadvantaged. BMJ 1995, 310:496-500. 10.1136/bmj.310.6978.496, 2548873, 7888888.
26. Woolf K, McManus IC, Potts HW, Dacre JE. The mediators of minority ethnic underperformance in final medical school examinations. Br J Educ Psychol 2013, 83:135-159. 10.1111/j.2044-8279.2011.02060.x, 23369179.
27. McManus IC, Dewberry C, Nicholson S, Dowell J. The UKCAT-12 Study: Technical Report 2012, Nottingham: UKCAT Consortium, available at [http://www.ukcat.ac.uk/App_Media/uploads/pdf/UKCAT-TechnicalReport-March2012-WithBackgroundAndSummary-Sep2013v2.pdf].
28. McManus IC, Elder AT, De Champlain A, Dacre JE, Mollon J, Chis L. Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations. BMC Med 2008, 6:5. 10.1186/1741-7015-6-5, 2265293, 18275598.
29. McManus IC, Ludka K. Resitting a high-stakes postgraduate medical examination on multiple occasions: nonlinear multilevel modelling of performance in the MRCP(UK) examinations. BMC Med 2012, 10:60. 10.1186/1741-7015-10-60, 3394208, 22697599.
30. McManus IC, Richards P, Winder BC, Sproston KA. Clinical experience, performance in final examinations, and learning style in medical students: prospective study. BMJ 1998, 316:345-350. 10.1136/bmj.316.7128.345, 2665541, 9487168.
31. McManus IC, Richards P, Winder BC. Intercalated degrees, learning styles, and career preferences: prospective longitudinal study of UK medical students. BMJ 1999, 319:542-546. 10.1136/bmj.319.7209.542, 28204, 10463892.
32. McManus IC, Winder BC, Paice E. How consultants, hospitals, trusts and deaneries affect pre-registration house officer posts: a multilevel model. Med Educ 2002, 36:35-44. 10.1046/j.1365-2923.2002.01123.x, 11849522.
33. Paice E, Rutter H, Wetherell M, Winder BC, McManus IC. Stressful incidents, stress and coping strategies in the pre-registration house officer year. Med Educ 2002, 36:56-65. 10.1046/j.1365-2923.2002.01101.x, 11849525.
34. McManus IC, Keeling A, Paice E. Stress, burnout and doctors' attitudes to work are determined by personality and learning style: a twelve year longitudinal study of UK medical graduates. BMC Med 2004, 2:29. 10.1186/1741-7015-2-29, 516448, 15317650.
35. McManus IC, Jonvik H, Richards P, Paice E. Vocation and avocation: leisure activities correlate with professional engagement, but not burnout, in a cross-sectional study of UK doctors. BMC Med 2011, 9:100. 10.1186/1741-7015-9-100, 3196901, 21878123.
36. McManus IC, Richards P, Winder BC, Sproston KA. Final examination performance of students from ethnic minorities. Med Educ 1996, 30:195-200. 10.1111/j.1365-2923.1996.tb00742.x, 8949553.
37. McManus IC, Richards P. An audit of admission to medical school: 2 - Shortlisting and interviews. Br Med J (Clin Res Ed) 1984, 289:1288-1290.
38. McManus IC, Richards P. An audit of admission to medical school: 3 - Applicants' perceptions and proposals for change. Br Med J (Clin Res Ed) 1984, 289:1365-1367.
39. McManus IC, Richards P. Prospective survey of performance of medical students during preclinical years. Br Med J (Clin Res Ed) 1986, 293:124-127.
40. Haario H, Laine M, Mira A, Saksman E. DRAM: Efficient adaptive MCMC. Statistics and Computing 2006, 16:339-354.
41. Field AP, Gillett R. How to do a meta-analysis. Brit J Math Stat Psychol 2010, 63:665-694.
42. Bramley T, Dhawan V. Estimates of reliability of qualifications 2011, Coventry: Office of Qualifications and Examinations Regulation, [http://www.ofqual.gov.uk/files/reliability/11-03-16-Estimates-of-Reliability-of-qualifications.pdf].
43. UKCAT UKCAT: 2007 Annual Report 2008, Nottingham: United Kingdom Clinical Aptitude Test, available at [http://www.ukcat.ac.uk/App_Media/uploads/pdf/Annual%20Report%202007.pdf], UKCAT.
44. UKCAT UKCAT 2008 Annual Report 2008, Nottingham, UK: UKCAT, available at http://www.ukcat.ac.uk/App_Media/uploads/pdf/Annual%20Report%202008.pdf, UKCAT.
45. Bacon DR, Bean B. GPA in research studies: an invaluable but neglected opportunity. J Marketing Educ 2006, 28:35-42.
46. McManus IC, Mooney-Somers J, Dacre JE, Vale JA. Reliability of the MRCP(UK) Part I Examination, 1984-2001. Med Educ 2003, 37:609-611. 10.1046/j.1365-2923.2003.01568.x, 12834418.
47. Tighe J, McManus IC, Dewhurst NG, Chis L, Mucklow J. The standard error of measurement is a more appropriate measure of quality in postgraduate medical assessments than is reliability: An analysis of MRCP(UK) written examinations. BMC Med Educ 2010, 10:40. 10.1186/1472-6920-10-40, 2893515, 20525220.
48. McManus IC, Thompson M, Mollon J. Assessment of examiner leniency and stringency ('hawk-dove effect') in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling. BMC Med Educ 2006, 6:4. 10.1186/1472-6920-6-4, 1361794, 16409640.
49. Yates J, James D. Predicting the "strugglers": a case-control study of students at Nottingham University Medical School. BMJ 2006, 332:1009-1013. 10.1136/bmj.38730.678310.63, 1450046, 16543299.
50. Yates J, James D. Risk factors at medical school for subsequent professional misconduct: multicentre retrospective case-control study. BMJ 2010, 340:c2040. 10.1136/bmj.c2040, 3191727, 20423965.
51. Yates J. Development of a 'toolkit' to identify medical students at risk of failure to thrive on the course: an exploratory retrospective case study. BMC Med Educ 2011, 11:95. 10.1186/1472-6920-11-95, 3229499, 22098629.
52. Hunter JE, Schmidt FL. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings 2004, Thousand Oaks, CA: Sage, 2.
53. Rubin DB. A new perspective on meta-analysis. The Future of Meta-Analysis 1990, 155-166. New York: Russell Sage, Wachter KW, Straf ML.
54. McManus IC, Woolf K, Dacre J. The educational background and qualifications of UK medical students from ethnic minorities. BMC Med Educ 2008, 8:21. 10.1186/1472-6920-8-21, 2359745, 18416818.
55. Coe R, Searle J, Barmby P, Jones K, Higgins S. Relative difficulty of examinations in different subjects 2008, Durham, UK: CEMCENTRE, http://www.cem.org/attachments/SCORE2008report.pdf.
56. Kirkup C, Wheater R, Morrison J, Durbin B, Pomati M. Use of an aptitude test in university entrance: A validity Study. Final report 2010, National Foundation for Educational Research: Slough, UK.
57. Garlick PB, Brown G. Widening participation in medicine. BMJ 2008, 336:1111-1113. 10.1136/bmj.39508.606157.BE, 2386657, 18483054.
58. Brown G, Garlick P. Changing geographies of access to medical education in London. Health Place 2007, 13:520-531. 10.1016/j.healthplace.2006.07.001, 16962817.
59. Brown G. The Place of Aspirations: Thoughts from a Research Project. [http://placeofaspirations.wordpress.com/2011/07/08/attaining-privilege-the-geography-of-admissions-to-elite-universities/].
60. Ip H, McManus IC. Increasing diversity among clinicians. BMJ 2008, 336:1082-1083. 10.1136/bmj.39569.609641.80, 2386618, 18483017.
61. Garlick P. Can Students with BCC Rather than AAA at A Level Succeed at Medical School? The Evidence from King's College London 2012, London: King's College London, [http://www.slideserve.com/omer/can-students-with-bcc-rather-than-aaa-at-a-level-succeed-at-medical-school-the-evidence-from-king-s-college-london].
62. Bekhradnia B, Thompson J. Who does Best at University? 2002, London: Higher Education Funding Council England, [http://webarchive.nationalarchives.gov.uk/20081202000732/http://hefce.ac.uk/Learning/whodoes/].
63. Mahesan N, Crichton S, Sewell H, Howell S. The effect of an intercalated BSc on subsequent academic performance. BMC Med Educ 2011, 11:76. 10.1186/1472-6920-11-76, 3200165, 21967682.
64. McManus IC, Woolf K, Dacre JE. Even one star at A level could be "too little, too late" for medical student selection. BMC Med Educ 2008, 8:16. 10.1186/1472-6920-8-16, 2335100, 18394196.
65. Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance medical board licensing examinations: a meta-analysis of the published research. Acad Med 2007, 82:100-106. 10.1097/01.ACM.0000249878.25186.b7, 17198300.
66. Choppin BHL, Orr L, Fara P, Kurie SDM, Fogelman KR, James G. After A-Level? A Study of the Transition from School to Higher Education 1972, Slough: NFER Publishing Company Ltd.
67. Choppin B, Orr L. Aptitude Testing at Eighteen-Plus 1976, Windsor: NFER Publishing Company Ltd.
68. Kirkup C, Schagen I, Wheater R, Morrison J, Whetton C. Use of an Aptitude Test in University Entrance - A Validity Study: Relationships between SAT Scores, Attainment Measures and Background Variables 2007, London: Department for Education and Skills, Research Report RR846.