Salvaging psychotherapy research: A manifesto

Journal of Evidence-Based Psychotherapies

Volumn 14, Issue 2, 2014, Pages 105-124

  Coyne, James C ^a,b   Kok, Robin N ^c,d  

^a UNIVERSITY MEDICAL CENTER GRONINGEN   (Netherlands)

^b RUTGERS UNIVERSITY   (United States)

^c UNIVERSITY OF AMSTERDAM   (Netherlands)

^d AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

Confirmatory bias; Conflict of interest; CONSORT; Investigator allegiance; Meta analysis; Reporting standards

Indexed keywords

EID: 84932080776     PISSN: 23600853     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (89)

1. Arain, M., Campbell, M. J., Cooper, C. L., & Lancaster, G. A. (2010). What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology, 10, 67. doi:10.1186/1471-2288-10-67
2. Bach, P., & Hayes, S. C. (2002). The use of acceptance and commitment therapy to prevent the rehospitalization of psychotic patients: a randomized controlled trial. Journal of Consulting and Clinical Psychology, 70(5), 1129-39. doi:10.1037//0022-006X.70.5.1129
3. Becker, B. J. (2005). Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. (H. R. Rothstein, A. J. Sutton, & M. Borenstein, Eds.) (pp. 111-125). Wiley.
4. Booth, A., Clarke, M., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2011). An international registry of systematic-review protocols. Lancet, 377(9760), 108-9. doi:10.1016/S0140-6736(10)60903-8
5. Chan, A., Tetzlaff, J. M., Gøtzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., … Moher, D. (2013). SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ (Clinical Research Ed.), 346, e7586. doi:10.1136/bmj.e7586
6. Chan, A., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA : The Journal of the American Medical Association, 291(20), 2457-65. doi:10.1001/jama.291.20.2457
7. Coyne, J. C. (2013a). No Title. Retrieved from https://jcoynester.wordpress.com/tag/bill-silverman-prize/
8. Coyne, J. C. (2013b). Shhh! Keeping quiet about the sad state of couples interventions for cancer patients research. Retrieved September 15, 2014, from http://blogs.plos.org/mindthebrain/2013/04/09/shhh-keeping-quiet-about-the-sad-state-of-couples-interventions-for-cancer-patients-research/
9. Coyne, J. C., Bhar, S. S., Pignotti, M., Tovote, K. A., & Beck, A. T. (2011). Missed Opportunity to Rectify or Withdraw a Flawed Metaanalysis of Longer-Term Psychodynamic Psychotherapy. Psychotherapy and Psychosomatics, 80(1), 53-54. doi:10.1159/000317537
10. Coyne, J. C., Cook, J. M., Palmer, S. C., & Rusiewicz, A. (2004). Clarification of clinical trial standards. Monitor on Psychology, 35(11), 4-8.
11. Coyne, J. C., & Kwakkenbos, L. (2013). Triple P-Positive Parenting programs : the folly of basing social policy on underpowered flawed studies. BMC Medicine, 11(1), 11. doi:10.1186/1741-7015-11-11
12. Coyne, J. C., Thombs, B. D., & Hagedoorn, M. (2010). Ain't necessarily so: review and critique of recent meta-analyses of behavioral medicine interventions in health psychology. Health Psychology : Official Journal of the Division of Health Psychology, American Psychological Association, 29(2), 107-16. doi:10.1037/a0017633
13. Coyne, J. C., & van Linschoten, H. S. (2014). Cognitive therapy for patients with schizophrenia. Lancet, 384(9941), 400. doi:10.1016/S0140-6736(14)61273-3
14. Cristea, I. A., Kafesciogly, N., & Coyne, J. C. (2013). The worrisome state of research concerning couples interventions for cancer patients: A systematic review. Psychology & Health, 28 Suppl 1, 2-335. doi:10.1080/08870446.2013.810851
15. Cristea, I. A., Kok, R. N., & Cuijpers, P. (in press). The efficacy of cognitive bias modification interventions in anxiety and depression: a meta-analysis. British Journal of Psychiatry. doi:10.1192/bjp.bp.114.146761
16. D'Agostino, R. B., Massaro, J. M., & Sullivan, L. M. (2003). Non-inferiority trials: design concepts and issues-the encounters of academic consultants in statistics. Statistics in Medicine, 22(2), 169-86. doi:10.1002/sim.1425
17. Dimidjian, S., Hollon, S. D., Dobson, K. S., Schmaling, K. B., Kohlenberg, R. J., Addis, M. E., … Jacobson, N. S. (2006). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of Consulting and Clinical Psychology, 74(4), 658-70. doi:10.1037/0022-006X.74.4.658
18. Dragioti, E., Dimoliatis, I., & Evangelou, E. (2014). EPA-0834-Documentation of therapeutic investigator allegiance in meta-analyses of randomised controlled trials. European Psychiatry, 29, 1. doi:10.1016/S0924-9338(14)78172-6
19. Duval, S., & Tweedie, R. (2000). Trim and Fill: A Simple Funnel-Plot-Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis. Biometrics, 56(2), 455-463. doi:10.1111/j.0006-341X.2000.00455.x
20. Dwan, K., Gamble, C., Williamson, P. R., & Kirkham, J. J. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias-an updated review. PloS One, 8(7), e66844. doi:10.1371/journal.pone.0066844
21. Ebrahim, S., Sohani, Z. N., Montoya, L., Agarwal, A., Thorlund, K., Mills, E. J., & Ioannidis, J. P. (2014). Reanalyses of Randomized Clinical Trial Data. JAMA, 312(10), 1024. doi:10.1001/jama.2014.9646
22. Egger, M., & Smith, G. D. (1998). Meta-analysis bias in location and selection of studies. British Medical Journal, 316(7124), 61-66. doi:10.1136/bmj.316.7124.61
23. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629-634. doi:10.1136/bmj.315.7109.629
24. Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33(5), 517-517. doi:10.1037/0003-066X.33.5.517.a
25. Flint, J., Cuijpers, P., Horder, J., Koole, S. L., & Munafò, M. R. (2014). Is there an excess of significant findings in published studies of psychotherapy for depression? Psychological Medicine, 1-8. doi:10.1017/S0033291714001421
26. Furukawa, T. A., Noma, H., Caldwell, D. M., Honyashiki, M., Shinohara, K., Imai, H., … Churchill, R. (2014). Waiting list may be a nocebo condition in psychotherapy trials: a contribution from network meta-analysis. Acta Psychiatrica Scandinavica, 1-12. doi:10.1111/acps.12275
27. Goldacre, B. (2012). Bad Pharma. London: HarperCollins.
28. Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006). A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Statistics in Medicine, 25(20), 3443-57. doi:10.1002/sim.2380
29. Hart, S. L., Hoyt, M. A., Diefenbach, M., Anderson, D. R., Kilbourn, K. M., Craft, L. L., … Stanton, A. L. (2012). Meta-analysis of efficacy of interventions for elevated depressive symptoms in adults diagnosed with cancer. Journal of the National Cancer Institute, 104(13), 990-1004. doi:10.1093/jnci/djs256
30. Hayes, S. C., Luoma, J. B., Bond, F. W., Masuda, A., & Lillis, J. (2006). Acceptance and commitment therapy: model, processes and outcomes. Behaviour Research and Therapy, 44(1), 1-25. doi:10.1016/j.brat.2005.06.006
31. Heene, M. (2010, October 12). A brief history of the Fail Safe Number in Applied Research. Retrieved from http://arxiv.org/abs/1010.2326
32. Hesser, H., Weise, C., Rief, W., & Andersson, G. (2011). The effect of waiting: A meta-analysis of wait-list control groups in trials for tinnitus distress. Journal of Psychosomatic Research, 70(4), 378-84. doi:10.1016/j.jpsychores.2010.12.006
33. Hopewell, S., Clarke, M., Moher, D., Wager, E., Middleton, P., Altman, D. G., & Schulz, K. F. (2008). CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration. PLoS Medicine, 5(1), e20. doi:10.1371/journal.pmed.0050020
34. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124
35. Ioannidis, J. P. (2008). Why most discovered true associations are inflated. Epidemiology (Cambridge, Mass.), 19(5), 640-8. doi:10.1097/EDE.0b013e31818131e7
36. Ioannidis, J. P. (2012). Scientific inbreeding and same-team replication: type D personality as an example. Journal of Psychosomatic Research, 73(6), 408-10. doi:10.1016/j.jpsychores.2012.09.014
37. Ioannidis, J. P. (2013). Clarifications on the application and interpretation of the test for excess significance and its extensions. Journal of Mathematical Psychology, 57(5), 184-187. doi:10.1016/j.jmp.2013.03.002
38. Ioannidis, J. P., Patsopoulos, N. A., & Evangelou, E. (2007). Uncertainty in heterogeneity estimates in meta-analyses. BMJ (Clinical Research Ed.), 335(7626), 914-6. doi:10.1136/bmj.39343.408449.80
39. Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials (London, England), 4(3), 245-53. doi:10.1177/1740774507079441
40. Jensen, A. M., & Ramasamy, A. (2009). Treating Spider Phobia Using Neuro Emotional TechniqueTM: Findings from a Pilot Study. The Journal of Alternative and …, 15(12), 1363-74. doi:10.1089/acm.2008.0595
41. Johnson, S. M., Hunsley, J., Greenberg, L., & Schindler, D. (2006). Emotionally Focused Couples Therapy: Status and Challenges. Clinical Psychology: Science and Practice, 6(1), 67-79. doi:10.1093/clipsy.6.1.67
42. Khurana, G., Henderson, S., Walter, G., & Martin, A. (2012). Conflict of interest and disclosure policies in psychiatry and medicine: a comparative study of peer-reviewed journals. Academic Psychiatry : The Journal of the American Association of Directors of Psychiatric Residency Training and the Association for Academic Psychiatry, 36(1), 17-22. doi:10.1176/appi.ap.09120248
43. Klingberg, S., Wittorf, A., Meisner, C., Wölwer, W., Wiedemann, G., Herrlich, J., … Buchkremer, G. (2010). Cognitive behavioural therapy versus supportive therapy for persistent positive symptoms in psychotic disorders: the POSITIVE Study, a multicenter, prospective, single-blind, randomised controlled clinical trial. Trials, 11, 123. doi:10.1186/1745-6215-11-123
44. Kok, B. E., Coffey, K. A., Cohn, M. A., Catalino, L. I., Vacharkulksemsuk, T., Algoe, S. B., … Fredrickson, B. L. (2013). How positive emotions build physical health: perceived positive social connections account for the upward spiral between positive emotions and vagal tone. Psychological Science, 24(7), 1123-32. doi:10.1177/0956797612470827
45. Koole, S. L., & Lakens, D. (2012). Rewarding Replications: A Sure and Simple Way to Improve Psychological Science. Perspectives on Psychological Science, 7(6), 608-614. doi:10.1177/1745691612462586
46. Kühberger, A., Fritz, A., & Scherndl, T. (2014). Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size. PloS One, 9(9), e105825. doi:10.1371/journal.pone.0105825
47. Leichsenring, F., & Rabung, S. (2011). Double standards in psychotherapy research. Psychotherapy and Psychosomatics, 80(1), 48-51; author reply 53-4. doi:10.1159/000315365
48. Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626-9. doi:10.1016/j.jpsychires.2010.10.008
49. Levin, M., & Hayes, S. C. (2009). Is Acceptance and Commitment Therapy Superior to Established Treatment Comparisons? Psychotherapy and Psychosomatics, 78(6), 380-381. doi:10.1159/000235978
50. Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., … Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Medicine, 6(7), e1000100. doi:10.1371/journal.pmed.1000100
51. Luborsky, L., Diguer, L., Seligman, D. A., Rosenthal, R., Krause, E. D., Johnson, S., … Schweizer, E. (2006). The Researcher's Own Therapy Allegiances: A "Wild Card" in Comparisons of Treatment Efficacy. Clinical Psychology: Science and Practice, 6(1), 95-106. doi:10.1093/clipsy.6.1.95
52. Malički, M., & Marušić, A. (2014). Is there a solution to publication bias? Researchers call for changes in dissemination of clinical research results. Journal of Clinical Epidemiology, 67(10), 1103-10. doi:10.1016/j.jclinepi.2014.06.002
53. Mentorcoach, L. (2014). Love 2.0: the master class. Retrieved from http://www.mentorcoach.com/love2/. (Archived by WebCite® at http://www.webcitation.org/6Sbs8qQRb)
54. Milette, K., Roseman, M., & Thombs, B. D. (2011). Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: A systematic review. Journal of Psychosomatic Research, 70(3), 205-17. doi:10.1016/j.jpsychores.2010.09.015
55. Mohr, D. C., Spring, B., Freedland, K. E., Beckner, V., Arean, P., Hollon, S. D., … Kaplan, R. (2009). The selection and design of control conditions for randomized controlled trials of psychological interventions. Psychotherapy and Psychosomatics, 78(5), 275-84. doi:10.1159/000228248
56. Morrison, A. P., Turkington, D., Pyle, M., Spencer, H., Brabban, A., Dunn, G., … Hutton, P. (2014). Cognitive therapy for people with schizophrenia spectrum disorders not taking antipsychotic drugs: a single-blind randomised controlled trial. Lancet, 383(9926), 1395-403. doi:10.1016/S0140-6736(13)62246-1
57. Munder, T., Brütsch, O., Leonhart, R., Gerger, H., & Barth, J. (2013). Researcher allegiance in psychotherapy outcome research: an overview of reviews. Clinical Psychology Review, 33(4), 501-11. doi:10.1016/j.cpr.2013.02.002
58. Niemeyer, H., Musch, J., & Pietrowsky, R. (2013). Publication bias in meta-analyses of the efficacy of psychotherapeutic interventions for depression. Journal of Consulting and Clinical Psychology, 81(1), 58-74. doi:10.1037/a0031152
59. Nowak, C., & Heinrichs, N. (2008). A comprehensive meta-analysis of Triple P-Positive Parenting Program using hierarchical linear modeling: effectiveness and moderating variables. Clinical Child and Family Psychology Review, 11(3), 114-44. doi:10.1007/s10567-008-0033-0
60. Nutt, D., Allgulander, C., Lecrubier, Y., Peters, T., & Wittchen, U. (2008). Establishing non-inferiority in treatment trials in psychiatry: guidelines from an Expert Consensus Meeting. Journal of Psychopharmacology (Oxford, England), 22(4), 409-16. doi:10.1177/0269881108091068
61. Onishi, A., & Furukawa, T. A. (2014). Publication bias is underreported in systematic reviews published in high-impact-factor journals: metaepidemiologic study. Journal of Clinical Epidemiology. doi:10.1016/j.jclinepi.2014.07.002
62. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8(2), 157-159.
63. Öst, L.-G. (2014). The efficacy of Acceptance and Commitment Therapy: An updated systematic review and meta-analysis. Behaviour Research and Therapy, (2014). doi:10.1016/j.brat.2014.07.018
64. Pandis, N., Shamseer, L., Kokich, V. G., Fleming, P. S., & Moher, D. (2014). Active implementation strategy of CONSORT adherence by a dental specialty journal improved randomized clinical trial reporting. Journal of Clinical Epidemiology, 67(9), 1044-8. doi:10.1016/j.jclinepi.2014.04.001
65. Peat, G., Riley, R. D., Croft, P., Morley, K. I., Kyzas, P. A., Moons, K. G. M., … Hemingway, H. (2014). Improving the transparency of prognosis research: the role of reporting, data sharing, registration, and protocols. PLoS Medicine, 11(7), e1001671. doi:10.1371/journal.pmed.1001671
66. Perlis, R. H., Perlis, C. S., Wu, Y., Hwang, C., Joseph, M., & Nierenberg, A. A. (2005). Industry sponsorship and financial conflict of interest in the reporting of clinical trials in psychiatry. The American Journal of Psychiatry, 162(10), 1957-60. doi:10.1176/appi.ajp.162.10.1957
67. Piaggio, G., Elbourne, D. R., Altman, D. G., Pocock, S. J., & Evans, S. J. W. (2006). Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA, 295(10), 1152. doi:10.1001/jama.295.10.1152
68. Pinto, R. Z., Elkins, M. R., Moseley, A. M., Sherrington, C., Herbert, R. D., Maher, C. G., … Ferreira, M. L. (2013). Many randomized trials of physical therapy interventions are not adequately registered: a survey of 200 published trials. Physical Therapy, 93(3), 299-309. doi:10.2522/ptj.20120206
69. Plint, A. C., Moher, D., Morrison, A., Schulz, K., Altman, D. G., Hill, C., & Gaboury, I. (2006). Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. MEDICAL JOURNAL OF AUSTRALIA, 185(5), 263-267.
70. Posternak, M. A., & Miller, I. (2001). Untreated short-term course of major depression: a meta-analysis of outcomes from studies using wait-list control groups. Journal of Affective Disorders, 66(2-3), 139-46.
71. Rücker, G., Carpenter, J. R., & Schwarzer, G. (2011). Detecting and adjusting for small-study effects in meta-analysis. Biometrical Journal. Biometrische Zeitschrift, 53(2), 351-68. doi:10.1002/bimj.201000151
72. Sanders, M. R., Kirby, J. N., Tellegen, C. L., & Day, J. J. (2014). The Triple P-Positive Parenting Program: a systematic review and meta-analysis of a multi-level system of parenting support. Clinical Psychology Review, 34(4), 337-57. doi:10.1016/j.cpr.2014.04.003
73. Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Annals of Internal Medicine, 152(11), 726-32. doi:10.7326/0003-4819-152-11-201006010-00232
74. Seppälä, E. M., Nitschke, J. B., Tudorascu, D. L., Hayes, A., Goldstein, M. R., Nguyen, D. T. H., … Davidson, R. J. (2014). Breathing-based meditation decreases posttraumatic stress disorder symptoms in u.s. Military veterans: a randomized controlled longitudinal study. Journal of Traumatic Stress, 27(4), 397-405. doi:10.1002/jts.21936
75. Shimazawa, R., & Ikeda, M. (2014). Conflicts of interest in psychiatry: strategies to cultivate literacy in daily practice. Psychiatry and Clinical Neurosciences, 68(7), 489-97. doi:10.1111/pcn.12201
76. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-66. doi:10.1177/0956797611417632
77. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology. General, 143(2), 534-47. doi:10.1037/a0033242
78. Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752-760. doi:10.1037/0003-066X.32.9.752
79. Smulders, Y. M. (2013). A two-step manuscript submission process can reduce publication bias. Journal of Clinical Epidemiology, 66(9), 946-7. doi:10.1016/j.jclinepi.2013.03.023
80. Staines, G. L., & Cleland, C. M. (2007). Bias in meta-analytic estimates of the absolute efficacy of psychotherapy. Review of General Psychology, 11(4), 329-347. doi:10.1037/1089-2680.11.4.329
81. Sterne, J. A, & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology, 54(10), 1046-55. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11576817
82. Thase, M. E. (2006). What Is the Investigator Allegiance Effect and What Should We Do About It? Clinical Psychology: Science and Practice, 6(1), 113-115. doi:10.1093/clipsy.6.1.113
83. Thompson, S. G., & Sharp, S. J. (1999). Explaining heterogeneity in meta-analysis: a comparison of methods. Statistics in Medicine, 18(20), 2693-708. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/10521860
84. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. The New England Journal of Medicine, 358(3), 252-60. doi:10.1056/NEJMsa065779
85. Vaughan, B., Goldstein, M. H., Alikakos, M., Cohen, L. J., & Serby, M. J. (2014). Frequency of reporting of adverse events in randomized controlled trials of psychotherapy vs. psychopharmacotherapy. Comprehensive Psychiatry, 55(4), 849-55. doi:10.1016/j.comppsych.2014.01.001
86. Viswanathan, M., Carey, T. S., Belinson, S. E., Berliner, E., Chang, S. M., Graham, E., … White, C. M. (2014). A proposed approach may help systematic reviews retain needed expertise while minimizing bias from nonfinancial conflicts of interest. Journal of Clinical Epidemiology. doi:10.1016/j.jclinepi.2014.02.023
87. Wolitzky-Taylor, K. B., Horowitz, J. D., Powers, M. B., & Telch, M. J. (2008). Psychological approaches in the treatment of specific phobias: a meta-analysis. Clinical Psychology Review, 28(6), 1021-37. doi:10.1016/j.cpr.2008.02.007
88. Yavchitz, A., Boutron, I., Bafeta, A., Marroun, I., Charles, P., Mantz, J., & Ravaud, P. (2012). Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Medicine, 9(9), e1001308. doi:10.1371/journal.pmed.1001308
89. Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011). The ClinicalTrials.gov results database--update and key issues. The New England Journal of Medicine, 364(9), 852-60. doi:10.1056/NEJMsa1012065