Simpson's paradox: How performance measurement can fail even with perfect risk adjustment

BMJ Quality and Safety

Volumn 23, Issue 9, 2014, Pages 701-705

  Marang van De Mheen, Perla J ^a   Shojania, Kaveh G ^b  

^a LEIDEN UNIVERSITY MEDICAL CENTER   (Netherlands)

^b UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

CRITICALLY ILL PATIENT; EDITORIAL; HEALTH CARE SYSTEM; HIGH RISK PATIENT; HUMAN; LOW RISK PATIENT; MORTALITY; PERFORMANCE MEASUREMENT SYSTEM; RISK ASSESSMENT; SCORING SYSTEM; SIMPSON PARADOX; STANDARDIZED MORTALITY RATIO; STATISTICAL PARAMETERS; DIAGNOSIS RELATED GROUP; HOSPITAL MORTALITY; PEDIATRIC INTENSIVE CARE UNIT; STATISTICS AND NUMERICAL DATA;

DIAGNOSIS-RELATED GROUPS; HOSPITAL MORTALITY; HUMANS; INTENSIVE CARE UNITS, PEDIATRIC;

EID: 84906672606     PISSN: 20445415     EISSN: None     Source Type: Journal    
DOI: 10.1136/bmjqs-2014-003358     Document Type: Editorial

References (29)

1. Iezzoni LI. 100 apples divided by 15 red herrings: a cautionary tale from the mid-19th century on comparing hospital mortality rates. Ann Intern Med 1996;124:1079-85.
2. Lee DS, Austin PC, Rouleau JL, et al. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. JAMA 2003;290:2581-7. (Pubitemid 37443288)
3. Tu JV, Sykora K, Naylor CD. Assessing the outcomes of coronary artery bypass graft surgery: how many risk factors are enough? Steering Committee of the Cardiac Care Network of Ontario. J Am Coll Cardiol 1997;30:1317-23. (Pubitemid 27501452)
4. Beck DH, Smith GB, Pappachan JV, et al. External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study. Intensive Care Med 2003;29:249-56. (Pubitemid 36384415)
5. Cook DA. Methods to assess performance of models estimating risk of death in intensive care patients: a review. Anaesth Intensive Care 2006;34:164-75.
6. Castella X, Artigas A, Bion J, et al. A comparison of severity of illness scoring systems for intensive care unit patients: results of a multicenter, multinational study. The European/North American Severity Study Group. Crit Care Med 1995;23:1327-35.
7. Maggard-Gibbons M. The use of report cards and outcome measurements to improve the safety of surgical care: the American College of Surgeons National Surgical Quality Improvement Program. BMJ Qual Saf 2014;23:589-99.
8. Snijders HS, Henneman D, van Leersum NL, et al. Anastomotic leakage as an outcome measure for quality of colorectal cancer surgery. BMJ Qual Saf 2013;22:759-67.
9. Simpson EH. The interpretation of interaction in contingency tables. J R Stat Soc B Methodol 1951;13:238-41.
10. Yule GU. On some points relating to the vital statistics of occupational mortality. J R Statist Soc 1934;97:1-72.
11. Ross K. A Mathematician at the Ballpark: odds and Probabilities for Baseball Fans. Pi Press, 2004.
12. Manktelow BN, Evans TA, Draper ES. Differences in case-mix can influence the comparison of standardised mortality ratios even with optimal risk adjustment: an analysis of data from paediatric intensive care. BMJ Qual Saf 2014;23:782-8.
13. Pavlides MG, Perlman MD. How likely is Simpson's paradox? Am Stat 2009;63:226-33.
14. Chan WK, Redelmeier DA. Simpson's paradox and the association between vitamin D deficiency and increased heart disease. Am J Cardiol 2012;110:143-4.
15. Bottle A, Jarman B, Aylin P. Strengths and weaknesses of hospital standardised mortality ratios. BMJ 2011;342:c7116.
16. Jarman B, Gault S, Alves B, et al. Explaining differences in English hospital death rates using routinely collected data. BMJ 1999;318:1515-20. (Pubitemid 29255794)
17. Jarman B, Pieter D, van der Veen AA, et al. The hospital standardised mortality ratio: a powerful tool for Dutch hospitals to assess their quality of care? Qual Saf Health Care 2010;19:9-13.
18. Wen E, Sandoval C, Zelmer J, et al. Understanding and using the hospital standardized mortality ratio in Canada: challenges and opportunities. Healthc Pap 2008;8:26-36; discussion 69-75.
19. Lilford R, Pronovost P. Using hospital mortality rates to judge hospital performance: a bad idea that just won't go away. BMJ 2010;340:c2016.
20. Mohammed MA, Deeks JJ, Girling A, et al. Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals. BMJ 2009;338:b780.
21. Shojania KG, Forster AJ. Hospital mortality: when failure is not a good measure of success. CMAJ 2008;179:153-7. (Pubitemid 352020122)
22. Howell J, Yonan N, Dunn PM, et al. Performance league tables. League tables are unreasonably simple. BMJ 2002;324:542.
23. Julious SA, Nicholl J, George S. Why do we continue to use standardized mortality ratios for small area comparisons? J Public Health Med 2001;23:40-6. (Pubitemid 32246300)
24. Shahian DM, Normand SL. Comparison of "risk-adjusted" hospital outcomes. Circulation 2008;117:1955-63.
25. Pouw ME, Peelen LM, Lingsma HF, et al. Hospital standardized mortality ratio: consequences of adjusting hospital mortality with indirect standardization. PLoS ONE 2013;8:e59160.
26. Brinkman S, Abu-Hanna A, van der Veen A, et al. A comparison of the performance of a model based on administrative data and a model based on clinical data: effect of severity of illness on standardized mortality ratios of intensive care units. Crit Care Med 2012;40:373-8.
27. Fine MJ, Auble TE, Yealy DM, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med 1997;336:243-50. (Pubitemid 27044625)
28. Rao JN. Hospital league tables. Analysis is flawed. BMJ 2001;322:992-3. (Pubitemid 32374698)
29. Shahian DM, Wolf RE, Iezzoni LI, et al. Variability in the measurement of hospital-wide mortality rates. N Engl J Med 2010;363:2530-9.