What is the probability of detecting poorly performing hospitals using funnel plots?

BMJ Quality and Safety

Volumn 22, Issue 10, 2013, Pages 870-876

  Seaton, Sarah E ^a   Barker, Lisa ^b   Lingsma, Hester F ^c   Steyerberg, Ewout W ^c   Manktelow, Bradley N ^a  

^a UNIVERSITY OF LEICESTER   (United Kingdom)

^b UNIVERSITY HOSPITALS OF LEICESTER NHS TRUST   (United Kingdom)

^c ERASMUS MEDICAL CENTER   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; HEALTH CARE PERSONNEL; HEALTH CARE QUALITY; HOSPITAL; HUMAN; MANAGER; MEDICAL PRACTICE; PERFORMANCE; PROBABILITY; STANDARDIZED MORTALITY RATIO;

HEALTHCARE QUALITY IMPROVEMENT; MORTALITY (STANDARDIZED MORTALITY RATIOS); PERFORMANCE MEASURES; STATISTICAL PROCESS CONTROL;

ENGLAND; HOSPITAL MORTALITY; HOSPITALS, PUBLIC; HUMANS; PROBABILITY; QUALITY ASSURANCE, HEALTH CARE; QUALITY INDICATORS, HEALTH CARE;

EID: 84884641849     PISSN: 20445415     EISSN: None     Source Type: Journal    
DOI: 10.1136/bmjqs-2012-001689     Document Type: Article

References (44)

1. The Mid Staffordshire NHS Foundation Trust Public Inquiry-Chaired by Robert Francis QC. 2013. http://www.midstaffspublicinquiry.com/report
2. Dyer C. Suspension of paediatric heart surgery in Leeds provokes controversy. BMJ 2013;346:f2098.
3. Lister S. Hundreds die at 'Third World' hospital: poor care took fatal toll over three years. The Times (2nd edn). 18 March 2009; News:3.
4. Lyall S. English hospital report cites 'appalling' suffering. The New York Times (Final edn). 7 February 2013;10.
5. Donnelly L, Hall M. Twenty children may have died needlessly before heart unit shut. The Daily Telegraph (2nd edn). 30 March 2013;News:4.
6. Suspension of children's heart unit 'premature'. The Daily Telegraph (2nd edn). 2 April 2013;News:2.
7. National Clinical Audit Advisory Group. Detection and management of outliers. London: Department of Health, 2011.
8. National Joint Registry. National Joint Registry adopted statistical methodolgy for potential outlier identification. London: NJR, 2008.
9. PICANet. Proposed PICANet policy on PICUs lying outside the control limits of the mortality ratio funnel plots. Leeds: PICANet, 2005. http://www.picanet.org.uk.
10. Jarman B, Gault S, Alves B, et al. Explaining differences in English hospital death rates using routinely collected data. BMJ 1999;318:1515-20.
11. Spiegelhalter D. Funnel plots for comparing institutional performance. Stat Med 2005;24:1185-202.
12. Seaton S, Manktelow B. The probability of being identified as an outlier with commonly used funnel plot control limits for the standardised mortality ratio. BMC Med Res Methodol 2012;12:98.
13. Jones H, Ohlssen D, Spiegelhalter D. Use of the false discovery rate when comparing multiple health care providers. J Clin Epidemiol 2008;61:232-40.
14. Field D, Manktelow B, Boyle E, et al. The neonatal survey report 2010. Leicester: TIMMS, 2010. http://www.le.ac.uk/timms.
15. New York State Department of Health. Percutaneous coronary interventions (PCI) in New York State 2008-2010. New York: New York State Department of Health, 2012.
16. Sparrow JM, Taylor H, Qureshi K, et al. The cataract national data set electronic multi-centre audit of 55 567 operations: case-mix adjusted surgeon's outcomes for posterior capsule rupture. Eye 2011;25:1010-15.
17. Blomqvist P, Andersson R, Granath F, et al. Mortality after appendectomy in Sweden, 1987-1996. Ann Surg 2001;233:455-60.
18. Robb E, Jarman B, Suntharalingam G, et al. Using care bundles to reduce in-hospital mortality: quantitative survey. BMJ 2010;340:c1234.
19. NHS. Summary Hospital-level Mortality Indicator (SHMI). http://www.ic.nhs.uk/services/SHMI (accessed 9 5 2013).
20. Black N. Assessing the quality of hospitals: hospital standardised mortality ratios should be abandoned. BMJ 2010;340:933-4.
21. Penfold R, Dean S, FlemonsW, et al. Do hospital standardized mortality ratios measure patient safety? HSMRs in the Winnipeg Regional Health Authority. Healthc Pap 2008;8:8-24.
22. Scott I, Brand C, Phelps G, et al. Using hospital standardised mortality ratios to assess quality of care-proceed with extreme caution. Med J Aust 2011;194:645-8.
23. van Gestel Y, Lemmens V, Lingsma H, et al. The hospital standardized mortality ratio fallacy: a narrative review. Med Care 2012;50:662-7.
24. Pronovost PJ, Lilford R. A road map for improving the performance of performance measures. Health Aff 2011;30:569-73.
25. Aziz O, Fink D, Hobbs L, et al. Hospital mortality under surgical care. Ann R Coll Surg Engl 2011;93:193-200.
26. Shojania K, Forster A. Hospital mortality: when failure is not a good measure of success. Can Med Assoc J 2008;179:153-7.
27. Corrao G, Corazza G, Bagnardi V, et al. Mortality in patients with coeliac disease and their relatives: a cohort study. Lancet 2001;358:356-61.
28. Jarman B, Bottle A, Aylin P, et al. Dr Foster's case notes: monitoring changes in hospital standardised mortality ratios. BMJ 2005;330:329.
29. Jarman B, Pieter D, van der Veen A, et al. The hospital standardised mortality ratio: a powerful tool for Dutch hospitals to assess their quality of care? BMJ Qual Saf 2010;19:9-13.
30. Canadian Institute for Health Information. HSMR: a new approach for measuring hospital mortality trends in Canada. Ottawa: CIHI, 2007.
31. Dr Foster Intelligence. Inside your hospital: Dr Foster hospital guide. London: Dr Foster Intelligence, 2011.
32. Barnett AG, van der Pols JC, Dobson AJ. Regression to the mean: what it is and how to deal with it. Int J Epidemiol 2005;34:215-20.
33. van Dishoeck A, Lingsma H, Mackenbach J, et al. Random variation and rankability of hospitals using outcome indicators. BMJ Qual Saf 2011;20:869-74.
34. Tekkis P, McCulloch P, Steger A, et al. Mortality control charts for comparing performance of surgical units: validation study using hospital mortality data. BMJ 2003;326:786.
35. Mayer E, Bottle A, Rao C, et al. Funnel plots and their emerging application in surgery. Ann Surg 2009;249:376-83.
36. van Dishoeck A, Looman C, van der Wilden-van Lier E, et al. Displaying random variation in comparing hospital performance. BMJ Qual Saf 2011;20:651-7.
37. Neuberger J, Cromwell D, Hutchings A, et al. Funnel plots for comparing provider performance based on patient-reported outcome measures. BMJ Qual Saf 2011;20:1020-6.
38. Henderson G, Mead G, van Dijke M, et al. Use of statistical process control charts in stroke medicine to determine if clinical evidence and changes in service delivery were associated with improvements in the quality of care. BMJ Qual Saf 2008;17:301-6.
39. Kirkham J, Bouamra O. The use of statistical process control for monitoring institutional performance in trauma care. J Trauma 2008;65:1494-501.
40. Mohammed A, Deeks J. In the context of performance monitoring, the caterpillar plot should be mothballed in favor of the funnel plot. Ann Thorac Surg 2008;86:348-55.
41. Association of Public Health Observatories. Statistical process control methods in public health intelligence. York: APHO, 2009.
42. Freiman JA, Chalmers TC, Smith H Jr, et al. The importance of Beta, the type II error and sample size in the design and interpretation of the randomized control trial-Survey of 71 negative trials. N Engl J Med 1978;299:690-4.
43. Nolte E. International benchmarking of healthcare quality: a review of the literature. Santa Monica, CA: RAND Corporation, 2010.
44. Dimick J, Welch H, Birkmeyer J. Surgical mortality as an indicator of hospital quality: the problem with small sample size. JAMA 2004;292:847-51.