Evaluation of occupational health interventions using a randomized controlled trial: Challenges and alternative research designs

Scandinavian Journal of Work, Environment and Health

Volumn 41, Issue 5, 2015, Pages 491-503

  Schelvis, Roosmarijn M C ^a,b   Oude Hengel, Karen M ^a,b   Burdorf, Alex ^c   Blatter, Birgitte M ^a,b   Strijk, Jorien E ^a   van der Beek, Allard J ^b  

^a TNO   (Netherlands)

^b VU UNIVERSITY MEDICAL CENTER   (Netherlands)

^c ERASMUS MEDICAL CENTER   (Netherlands)

Author keywords

Difference in difference; Effectiveness; Experimental study design; Instrumental variable; Interrupted time series; Multiple baseline design; Observational study design; Propensity score; RCT; Regression discontinuity; Stepped wedge design

Indexed keywords

DESIGN METHOD; EXPERIMENTAL STUDY; FEASIBILITY STUDY; HEALTH RISK; OBSERVATIONAL METHOD; OCCUPATIONAL EXPOSURE; REGRESSION ANALYSIS; RESEARCH METHOD; TIME SERIES; WORKPLACE;

ABSENTEEISM; ARTICLE; CONFOUNDING VARIABLE; DIFFERENCES IN DIFFERENCES METHOD; EXPERIMENTAL STUDY; HUMAN; INSTRUMENTAL VARIABLE ANALYSIS; INTERNAL VALIDITY; INTERRUPTED TIME SERIES DESIGN; INTERVENTION STUDY; METHODOLOGY; MULTIPLE BASELINE DESIGN; OBSERVATIONAL STUDY; OCCUPATIONAL HEALTH; PRIORITY JOURNAL; PROGRAM EVALUATION; PROPENSITY SCORE; RANDOMIZED CONTROLLED TRIAL (TOPIC); REGRESSION DISCONTINUITY; STRESS MANAGEMENT; STUDY DESIGN; EPIDEMIOLOGY; PROCEDURES; RANDOMIZATION;

ACARICIDE;

ACARICIDES; HUMANS; INTERRUPTED TIME SERIES ANALYSIS; OBSERVATIONAL STUDIES AS TOPIC; OCCUPATIONAL HEALTH; RANDOM ALLOCATION; RANDOMIZED CONTROLLED TRIALS AS TOPIC; RESEARCH DESIGN;

EID: 84940826556     PISSN: 03553140     EISSN: 1795990X     Source Type: Journal    
DOI: 10.5271/sjweh.3505     Document Type: Article

References (79)

1. Crofton JW, Clegg JW, Mitchison DA, Hurford JV, Douglas Smith BJ, Snell WE, et al. STREPTOMYCIN treatment of pulmonary tuberculosis. A Medical Research Council Investigation. Brit Med J. 1948;2(4582):769–82. http://dx.doi.org/10.1136/bmj.2.4582.769
2. Martimo KP, Shiri R, Miranda H, Ketola R, Varonen H, Viikari-Juntura E. Effectiveness of an ergonomic intervention on the productivity of workers with upper-extremity disorders -A randomized controlled trial. Scand J Work Environ Health. 2010;36(1):25–33. http://dx.doi.org/10.5271/sjweh.2880
3. Wolff N. Using randomized controlled trials to evaluate socially complex services: problems, challenges and recommendations. J Ment Health Policy Econ. 2000;3(2):97–109. http://dx.doi.org/10.1002/1099-176X(200006)3:2<97::AIDMHP77> 3.0.CO;2-S.
4. Blackwood B. Methodological issues in evaluating complex healthcare interventions. J Adv Nurs. 2006;54(5):612–22. http://dx.doi.org/10.1111/j.1365-2648.2006.03869.x.
5. Hawe P, Shiell A, Riley T. Important considerations for standardizing complex interventions. J Adv Nurs. 2008;62(2):267. http://dx.doi.org/10.1111/j.1365-2648.2008.04686.x.
6. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med. 2004;27(1):3–12. http://dx.doi.org/10.1207/s15324796abm2701_2
7. Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, et al. Designing and evaluating complex interventions to improve health care. Brit Med J. 2007;334(7591):455–9. http://dx.doi.org/10.1136/bmj.39108.379965.BE.
8. Shepperd S, Lewin S, Straus S, Clarke M, Eccles MP, Fitzpatrick R, et al. Can we systematically review studies that evaluate complex interventions? PLoS Med. 2009;6(8):e1000086. http://dx.doi.org/10.1371/journal.pmed.1000086
9. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: The new Medical Research Council guidance. Int J Nurs Stud. 2013;50(5):587–92. http://dx.doi.org/10.1016/j.ijnurstu.2012.09.010
10. Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93(8):1261–7. http://dx.doi.org/10.2105/AJPH.93.8.1261
11. Ioannidis JPA, Haidich A, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA-J Am Med Assoc. 2001;286(7):821–30. http://dx.doi.org/10.1001/jama.286.7.821
12. Campbell MK, Piaggio G, Elbourne DR, Altman DG. Consort 2010 statement: Extension to cluster randomised trials. BMJ. 2012;345:e5661. http://dx.doi.org/10.1136/bmj.e5661
13. Puffer S, Torgerson DJ, Watson J. Cluster randomized controlled trials. J Eval Clin Pract. 2005;11(5):479–83. http://dx.doi.org/10.1111/j.1365-2753.2005.00568.x.
14. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al. Improving the reporting of pragmatic trials: An extension of the CONSORT statement. Brit Med J. 2008;337(7680):1223–6. http://dx.doi.org/10.1136/bmj.a2390
15. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. New Engl J Med. 2000;342(25):1887–92. http://dx.doi.org/10.1056/NEJM200006223422507
16. Last JM, Spasoff RA, Harris SS, editors. A Dictionary of Epidemiology. 4th ed. New York: Oxford University Press; 2000
17. Oude Hengel KM, Blatter BM, Joling CI, van der Beek AJ, Bongers PM. Effectiveness of an intervention at construction worksites on work engagement, social support, physical workload, and need for recovery: results from a cluster randomized controlled trial. BMC Public Health. 2012;12:1008. http://dx.doi.org/10.1186/1471-2458-12-1008
18. Kristensen TS. Intervention studies in occupational epidemiology. Occup Environ Med. 2005;62(3):205–10. http://dx.doi.org/10.1136/oem.2004.016097
19. Nielsen K, Taris TW, Cox T. The future of organizational interventions: Addressing the challenges of today’s organizations. Work Stress. 2010;24(3):219–33. http://dx.doi.org/10.1080/02678373.2010.519176
20. Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: A new approach to evaluating complex public health interventions. Soc Sci Med. 2012;75(12):2299–306
21. Griffiths A. Organizational interventions. Facing the limits of the natural science paradigm. Scand J Work Environ Health. 1999;25(6):589–96. http://dx.doi.org/10.5271/sjweh.485
22. West SG, Duan N, Pequegnat W, Gaist P, Des Jarlais DC, Holtgrave D, et al. Alternatives to the randomized controlled trial. Am J Public Health. 2008;98(8):1359–66. http://dx.doi.org/10.2105/AJPH.2007.124446
23. Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007;28(2):182–91. http://dx.doi.org/10.1016/j.cct.2006.05.007
24. Brown CA, Lilford RJ. The stepped wedge trial design: A systematic review. BMC Med Res Methodol. 2006;6:54. http://dx.doi.org/10.1186/1471-2288-6-54
25. Mdege ND, Man M-, Taylor CA, Torgerson DJ. Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. J Clin Epidemiol. 2011;64(9):936–48. http://dx.doi.org/10.1016/j.jclinepi.2010.12.003
26. Sanson-Fisher RW, D’Este CA, Carey ML, Noble N, Paul CL. Evaluation of systems-oriented public health interventions: Alternative research designs. Annu Rev Publ Health. 2014;35:9–27. http://dx.doi.org/10.1146/annurevpublhealth-032013-182445
27. Zwijsen SA, Gerritsen DL, Eefsting JA, Smalbrugge M, Hertogh CM, Pot AM. Coming to grips with challenging behaviour: a cluster randomised controlled trial on the effects of a new care programme for challenging behaviour on burnout, job satisfaction and job demands of care staff on dementia special care units. Int J Nurs Stud. 2015;52(1):68–74. http://dx.doi.org/10.1016/j.ijnurstu.2014.10.003
28. Kotz D, Spigt M, Arts ICW, Crutzen R, Viechtbauer W. Researchers should convince policy makers to perform a classic cluster randomized controlled trial instead of a stepped wedge design when an intervention is rolled out. J Clin Epidemiol. 2012;65(12):1255–6. http://dx.doi.org/10.1016/j.jclinepi.2012.06.016
29. Kotz D, Spigt M, Arts ICW, Crutzen R, Viechtbauer W. Use of the stepped wedge design cannot be recommended: A critical appraisal and comparison with the classic cluster randomized controlled trial design. J Clin Epidemiol. 2012;65(12):1249-52. http://dx.doi.org/10.1016/j.jclinepi.2012.06.004
30. D’Agostino RB. Propensity score methods for bias reduction in the comparison of a treatment to a nonrandomized control group. Stat Med. 1998;17(19):2265-81. http://dx.doi.org/10.1002/(SICI)1097-0258(19981015)17:19<2265::AID-SIM918>3.0.CO;2-B.
31. Shadish WR, Luellen JK, Clark MH. Propensity Scores and Quasi-Experiments: A Testimony to the Practical Side of Lee Sechrest. In: Bootzin RR, McKnight PE, editors. Strengthening research methodology: Psychological measurement and evaluation. Washington, DC, US: American Psychological Association; 2006. p. 143–57. http://dx.doi.org/10.1037/11384-008
32. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41–55. http://dx.doi.org/10.1093/biomet/70.1.41
33. Luellen JK, Shadish WR, Clark MH. Propensity scores: an introduction and experimental test. Evaluation Rev. 2005;29(6):530–58. http://dx.doi.org/10.1177/0193841X05275596
34. Austin PC. An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies. Multivar Behav Res. 2011;46(3):399–424. http://dx.doi.org/10.1080/00273171.2011.568786
35. Cousens S, Hargreaves J, Bonell C, Armstrong B, Thomas J, Kirkwood BR, et al. Alternatives to randomisation in the evaluation of public-health interventions: statistical analysis and causal inference. J Epidemiol Commun H. 2011;65(7):576-81. http://dx.doi.org/10.1136/jech.2008.082610
36. Schafer JL, Kang J. Average causal effects from nonrandomized studies: a practical guide and simulated example. Psychol Methods. 2008;13(4):279–313. http://dx.doi.org/10.1037/a0014268
37. Saltychev M, Laimi K, Oksanen T, Pentti J, Kivimaki M, Vahtera J. Does perceived work ability improve after a multidisciplinary preventive program in a population with no severe medical problems? The Finnish Public Sector Study. Scand J Work Environ Health. 2013;39(1):57–65. http://dx.doi.org/10.5271/sjweh.3298
38. Saltychev M, Laimi K, El-Metwally A, Oksanen T, Pentti J, Virtanen M, et al. Effectiveness of multidisciplinary primary prevention in decreasing the risk of work disability in a lowrisk population. Scand J Work Environ Health. 2012;38(1):27-37. http://dx.doi.org/10.5271/sjweh.3169
39. Austin PC. Double propensity-score adjustment: A solution to design bias or bias due to incomplete matching. Stat Methods Med Res. 2014;1–22. http://dx.doi.org/10.1177/0962280214543508
40. Steiner PM, Cook TD, Shadish WR. On the Importance of Reliable Covariate Measurement in Selection Bias Adjustments Using Propensity Scores. J Educ Behav Stat. 2011;36(2):213-36. http://dx.doi.org/10.3102/1076998610375835
41. Steiner PM, Cook TD, Shadish WR, Clark MH. The importance of covariate selection in controlling for selection bias in observational studies. Psychol Methods. 2010;15(3):250–67. http://dx.doi.org/10.1037/a0018719
42. Newhouse JP, McClellan M. Econometrics in outcomes research: the use of instrumental variables. Annu Rev Publ Health. 1998;19:17–34. http://dx.doi.org/10.1146/annurev.publhealth.19.1.17
43. Bennett DA. An introduction to instrumental variables analysis: part 1. Neuroepidemiology. 2010;35(3):237–40. http://dx.doi.org/10.1159/000319455
44. Rassen JA, Brookhart MA, Glynn RJ, Mittleman MA, Schneeweiss S. Instrumental variables I: instrumental variables exploit natural variation in nonexperimental data to estimate causal relationships. J Clin Epidemiol. 2009;62(12):1226–32. http://dx.doi.org/10.1016/j.jclinepi.2008.12.005
45. Elovainio M, Heponiemi T, Kuusio H, Jokela M, Aalto A, Pekkarinen L, et al. Job demands and job strain as risk factors for employee wellbeing in elderly care: an instrumentalvariables analysis. Eur J Public Health. 2015;25(1):103–8. http://dx.doi.org/10.1093/eurpub/cku115
46. Behncke S. Does retirement trigger ill health? Health Econ. 2012;21(3):282–300. http://dx.doi.org/10.1002/hec.1712
47. Martens EP, Pestman WR, de Boer A, Belitser SV, Klungel OH. Instrumental variables: application and limitations. Epidemiology. 2006;17(3):260–7. http://dx.doi.org/10.1097/01.ede.0000215160.88317.cb.
48. Moeyaert M, Ugille M, Ferron JM, Beretvas SN, Van den Noortgate W. Modeling external events in the three-level analysis of multiple-baseline across-participants designs: a simulation study. Behav Res Methods. 2013;45(2):547–59. http://dx.doi.org/10.3758/s13428-012-0274-1
49. Hawkins NG, Sanson-Fisher RW, Shakeshaft A, D’Este C, Green LW. The multiple baseline design for evaluating population-based research. Am J Prev Med. 2007;33(2):162–8. http://dx.doi.org/10.1016/j.amepre.2007.03.020
50. Rhoda DA, Murray DM, Andridge RR, Pennell ML, Hade EM. Studies with staggered starts: multiple baseline designs and group-randomized trials. Am J Public Health. 2011;101(11):2164–9. http://dx.doi.org/10.2105/AJPH.2011.300264
51. Camden M, Ludwig T. Absenteeism in Health Care: Using Interlocking Behavioral Contingency Feedback to Increase Attendance With Certified Nursing Assistants. Journal of Organizational Behavior Management. 2013;33(3):165–84. http://dx.doi.org/10.1080/01608061.2013.814521
52. Cuvo AJ. Multiple-baseline design in instructional research: pitfalls of measurement and procedural advantages. Am J Ment Def. 1979;84(3):219–28
53. Biglan A, Ary D, Wagenaar AC. The value of interrupted time-series experiments for community intervention research. Prevention Science. 2000;1(1):31–49. http://dx.doi.org/10.1023/A:1010024016308
54. Fretheim A, Soumerai SB, Zhang F, Oxman AD, Ross-Degnan D. Interrupted time-series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result. J Clin Epidemiol. 2013;66(8):883–7. http://dx.doi.org/10.1016/j.jclinepi.2013.03.016
55. Zhang F, Wagner AK, Soumerai SB, Ross-Degnan D. Methods for estimating confidence intervals in interrupted time series analyses of health interventions. J Clin Epidemiol. 2009;62(2):143–8. http://dx.doi.org/10.1016/j.jclinepi.2008.08.007
56. Farina E, Bena A, Pasqualini O, Costa G. Are regulations effective in reducing construction injuries? An analysis of the Italian context. Occup Environ Med. 2013;70(9):611–6. http://dx.doi.org/10.1136/oemed-2012-101087
57. Ramsay CR, Matowe L, Grilli R, Grimshaw JM, Thomas RE. Interrupted time series designs in health technology assessment: Lessons from two systematic reviews of behavior change strategies. Int J Technol Assess. 2003;19(4):613–23. http://dx.doi.org/10.1017/S0266462303000576
58. Matowe LK, Leister CA, Crivera C, Korth-Bradley JM. Interrupted time series analysis in clinical research. Ann Pharmacother. 2003;37(7-8):1110–6. http://dx.doi.org/10.1345/aph.1A109
59. Angrist J, Krueger A. Emperical strategies in Labor Economics. In: Ashenfelter O, Card D, editors. Handbook in Labor Economics. Elsevier; 2000. p. 1277-366
60. Blundell R, Costa Dias M. Alternative Approaches to Evaluation in Empirical Microeconomics. J Hum Resour. 2009;44:565–640. http://dx.doi.org/10.1353/jhr.2009.0009
61. Blundell R, MaCurdy T. Labor Supply. In: Ashenfelter O, Card D, editors. Handbook of Labor Economics. Elsevier; 2000. p. 1559–695
62. Wickizer TM, Franklin G, Fulton-Kehoe D, Gluck J, Mootz R, Smith-Weller T, et al. Improving quality, preventing disability and reducing costs in workers’ compensation healthcare: a population-based intervention study. Med Care. 2011;49(12):1105–11. http://dx.doi.org/10.1097/MLR.0b013e31823670e3
63. Bertrand B, Duflo E, Mullainathan S. How much should we trust difference-in-difference estimates? The Quarterly Journal of Economics. 2004;119(1):249–75. http://dx.doi.org/10.1162/003355304772839588
64. O’Keeffe AG, Geneletti S, Baio G, Sharples LD, Nazareth I, Petersen I. Regression discontinuity designs: an approach to the evaluation of treatment efficacy in primary care using observational data. Brit Med J. 2014;349:g5293. http://dx.doi.org/10.1136/bmj.g5293
65. Moscoe E, Bor J, Bärnighausen T. Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: A review of current and best practice. J Clin Epidemiol. 2015;68(2):122–33. http://dx.doi.org/10.1016/j.jclinepi.2014.06.021
66. Bor J, Moscoe E, Mutevedzi P, Newell ML, Barnighausen T. Regression discontinuity designs in epidemiology: causal inference without randomized trials. Epidemiology. 2014; 25(5): 729–37. http://dx.doi.org/10.1097/EDE.0000000000000138
67. Caliendo M, Tatsiramos K, Uhlendorff A. Benefit duration, unemployment duration and job match quality: a regressiondiscontinuity approach. J Appl Econom. 2013;28:604–27. http://dx.doi.org/10.1002/jae.2293
68. Pennell ML, Hade EM, Murray DM, Rhoda DA. Cutoff designs for community-based intervention studies. Stat Med. 2011;30(15):1865–82. http://dx.doi.org/10.1002/sim.4237
69. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7. http://dx.doi.org/10.2105/AJPH.89.9.1322
70. Nielsen K, Randall R. Opening the black box: Presenting a model for evaluating organizational-level interventions. European Journal of Work and Organizational Psychology. 2013;22(5):601–17. http://dx.doi.org/10.1080/1359432X.2012.690556
71. Groves P, Kayyali B, Knott D, Van Kuiken S. The ‘big data’ revolution in healthcare: Accelarting value and innovation. Center for US Health System Reform Business Technology Office: McKinsey & Company; 2013
72. Mooney SJ, Westreich DJ, El-Sayed AM. Commentary: epidemiology in the era of big data. Epidemiology. 2015; 26(3): 390–4. http://dx.doi.org/10.1097/EDE.0000000000000274
73. Meijster T, Tielemans E, Heederik D. Effect of an intervention aimed at reducing the risk of allergic respiratory disease in bakers: Change in flour dust and fungal alpha-amylase levels. Occup Environ Med. 2009;66(8):543–9. http://dx.doi.org/10.1136/oem.2008.042564
74. Smedley J, Harris EC, Cox V, Ntani G, Coggon D. Evaluation of a case management service to reduce sickness absence. Occup Med. 2013;63(2):89–95. http://dx.doi.org/10.1093/occmed/kqs223
75. Kobayashi Y, Kaneyoshi A, Yokota A, Kawakami N. Effects of a worker participatory program for improving work environments on job stressors and mental health among workers: a controlled trial. J Occup Health. 2008;50(6):455-70. http://dx.doi.org/10.1539/joh.L7166
76. Frykman M, Hasson H, Muntlin Athlin A, Von Thiele Schwarz U. Functions of behavior change interventions when implementing multi-professional teamwork at an emergency department: A comparative case study. BMC Health Services Research. 2014;14(1). http://dx.doi.org/10.1186/1472-6963-14-218
77. Steenhuis I, van Assema P, Reubsaet A, Kok G. Process evaluation of two environmental nutrition programmes and an educational nutrition programme conducted at supermarkets and worksite cafeterias in the Netherlands. J Hum Nutr Diet. 2004;17(2):107–15. http://dx.doi.org/10.1111/j.1365-277X.2004.00507.x.
78. Oude Hengel KM, Blatter BM, Van Der Molen HF, Joling CI, Proper KI, Bongers PM, et al. Meeting the challenges of implementing an intervention to promote work ability and health-related quality of life at construction worksites: A process evaluation. J Occup Environ Med. 2011;53(12):1483-91. http://dx.doi.org/10.1097/JOM.0b013e3182398e03
79. van Deurssen EH, Pronk A, Meijster T, Tielemans E, Heederik D, Oude Hengel KM. Process evaluation of an intervention program to reduce occupational quartz exposure among Dutch construction workers. J Occup Environ Med. 2015;57(4):428-35. http://dx.doi.org/10.1097/JOM.0000000000000382.