For Better or Worse: An Individual Patient Data Meta-Analysis of Deterioration Among Participants Receiving Internet-Based Cognitive Behavior Therapy

Journal of Consulting and Clinical Psychology

Volumn 85, Issue 2, 2017, Pages 160-177

  Rozental, Alexander ^a   Magnusson, Kristoffer ^b   Boettcher, Johanna ^c   Andersson, Gerhard ^b,d   Carlbring, Per ^a  

^a STOCKHOLM UNIVERSITY   (Sweden)

^b KAROLINSKA INSTITUTE   (Sweden)

^c FREIE UNIVERSITÄT BERLIN   (Germany)

^d LINKÖPING UNIVERSITY   (Sweden)

Author keywords

deterioration; individual patient data meta analysis; Internet based cognitive behavior therapy; negative effects; predictors

Indexed keywords

COGNITIVE BEHAVIORAL THERAPY; CONFIDENCE INTERVAL; CONTROLLED STUDY; DETERIORATION; HUMAN; INTERNET; META ANALYSIS; ODDS RATIO; PATIENT CODING; PSYCINFO; SCIENTIST; STATISTICAL ANALYSIS; STATISTICAL MODEL; UNIVERSITY;

EID: 85007258142     PISSN: 0022006X     EISSN: 19392117     Source Type: Journal    
DOI: 10.1037/ccp0000158     Document Type: Article

References (124)

1. Almlöv, J., Carlbring, P., Kallqvist, K., Paxling, B., Cuijpers, P., & Andersson, G. (2011). Therapist Effects in Guided Internet-Delivered CBT for Anxiety Disorders. Behavioural and Cognitive Psychotherapy, 39, 311-322. http://dx.doi.org/10.1017/S135246581000069X
2. American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (DSM-IV-TR). Washington, DC: Author.
3. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (DSM-5). Arlington, VA: Author.
4. Andersson, G., Carlbring, P., Holmstrom, A., Sparthan, E., Furmark, T., Nilsson-Ihrfelt, E., Ekselius, L. (2006). Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 74, 677-686. http://dx.doi.org/10.1037/0022-006x.74.4.677
5. Andersson, G., Hesser, H., Veilord, A., Svedling, L., Andersson, F., Sleman, O., Carlbring, P. (2013). Randomised controlled noninferiority trial with 3-year follow-up of internet-delivered versus faceto-face group cognitive behavioural therapy for depression. J Affect Disord, 151, 986-994. http://dx.doi.org/10.1016/j.jad.2013.08.022
6. Andersson, G., Paxling, B., Roch-Norlund, P., Östman, G., Norgren, A., Almlov, J., Silverberg, F. (2012). Internet-based psychodynamic versus cognitive behavioral guided self-help for generalized anxiety disorder: A randomized controlled trial. Psychotherapy amd Psychosomatics, 81, 344-355. http://dx.doi.org/10.1159/000339371
7. Andersson, G., & Titov, N. (2014). Advantages and limitations of Internetbased interventions for common mental disorders. World Psychiatry: Official Journal of the World Psychiatric Association, 13, 4-11. http://dx.doi.org/10.1002/wps.20083
8. Andersson, G., Waara, J., Jonsson, U., Malmaeus, F., Carlbring, P., & Ost, L. G. (2009). Internet-based self-help versus one-session exposure in the treatment of spider phobia: A randomized controlled trial. Cogn Behav Ther, 38, 114-120. http://dx.doi.org/10.1080/16506070902931326
9. Andersson, G., Waara, J., Jonsson, U., Malmaeus, F., Carlbring, P., & Öst, L.-G. (2013). Internet-Based Exposure Treatment Versus One-Session Exposure Treatment of Snake Phobia: A Randomized Controlled Trial. Cognitive Behaviour Therapy, 42, 284-291. http://dx.doi.org/10.1080/16506073.2013.844202
10. Andersson, E., Walen, C., Hallberg, J., Paxling, B., Dahlin, M., Almlov, J., Andersson, G. (2011). A Randomized Controlled Trial of Guided Internet-delivered Cognitive Behavioral Therapy for Erectile Dysfunction. Journal of Sexual Medicine, 8, 2800-2809. http://dx.doi.org/10 .1111/j.1743-6109.2011.02391.x
11. Arnberg, F. K., Linton, S. J., Hultcrantz, M., Heintz, E., & Jonsson, U. (2014). Internet-delivered psychological treatments for mood and anxiety disorders: A systematic review of their efficacy, safety, and costeffectiveness. PLoS ONE, 9, e98118. http://dx.doi.org/10.1371/journal .pone.0098118
12. Arrindell, W. A. (1993). The fear of fear concept: Stability, retest artifact and predictive power. Behaviour Research and Therapy, 31, 139-148.
13. Arrindell, W. A. (2001). Changes in waiting-list patients over time: Data on some commonly-used measures. Beware! Behaviour Research and Therapy, 39, 1227-1247. http://dx.doi.org/10.1016/S0005-7967(00)00104-2
14. Barlow, D. H. (2010). Negative effects from psychological treatments: A perspective. American Psychologist, 65, 13-20. http://dx.doi.org/10 .1037/a0015643
15. Bates, D., Machler, M., Bolker, B. M., & Walker, S. C. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1-48. http://dx.doi.org/10.18637/jss.v067.i01
16. Baucom, D. H., Hahlweg, K., & Kuschel, A. R. (2003). Are waiting-list control groups needed in future marital therapy outcome research Behavior Therapy, 34, 179-188. http://dx.doi.org/10.1016/S0005-7894(03)80012-6
17. Bauer, S., Lambert, M. J., & Nielsen, S. L. (2004). Clinical significance methods: A comparison of statistical techniques. Journal of Personality Assessment, 82, 60-70. http://dx.doi.org/10.1207/s15327752jpa8201-11
18. Beck, A. T., & Steer, R. A. (1996). Beck Depression Inventory manual, Svensk version (Swedish version). Fagernes, Norway: Psykologiförlaget AB.
19. Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561-571. http://dx.doi.org/10.1001/archpsyc.1961.01710120031004
20. Bergin, A. E. (1966). Some implications of psychotherapy research for therapeutic practice. Journal of Abnormal Psychology, 71, 235-246. http://dx.doi.org/10.1037/h0023577
21. Blake, D. D., Weathers, F. W., Nagy, L. M., Kaloupek, D. G., Gusman, F. D., Charney, D. S., & Keane, T. M. (1995). The development of a Clinician-Administered PTSD Scale. Journal of Traumatic Stress, 8, 75-90. http://dx.doi.org/10.1002/jts.2490080106
22. Boettcher, J., Astrom, V., Pahlsson, D., Schenstrom, O., Andersson, G., & Carlbring, P. (2014). Internet-Based Mindfulness Treatment for Anxiety Disorders: A Randomized Controlled Trial. Behavior Therapy, 45, 241-253.
23. Boettcher, J., Hasselrot, J., Sund, E., Andersson, G., & Carlbring, P. (2013). Combining Attention Training with Internet-Based Cognitive-Behavioural Self-Help for Social Anxiety: A Randomised Controlled Trial. Cognitive Behaviour Therapy, 43, 34-48. http://dx.doi.org/10 .1080/16506073.2013.809141
24. Boettcher, J., Rozental, A., Andersson, G., & Carlbring, P. (2014). Side effects in Internet-based interventions for social anxiety disorder. Internet Interventions, 1, 3-11. http://dx.doi.org/10.1016/j.invent.2014.02 .002
25. Boisvert, C. M., & Faust, D. (2002). Iatrogenic symptoms in psychotherapy. A theoretical exploration of the potential impact of labels, language, and belief systems. American Journal of Psychotherapy, 56, 244-259.
26. Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy Research, 25, iii. http://dx.doi .org/10.1080/10503307.2014.942531
27. Bower, P., Kontopantelis, E., Sutton, A., Kendrick, T., Richards, D. A., Gilbody, S., Liu, E. T. (2013). Influence of initial severity of depression on effectiveness of low intensity interventions: Meta-analysis of individual patient data. British Medical Journal (Clinical Research Ed.), 346, f540. http://dx.doi.org/10.1136/bmj.f540
28. Buntrock, C., Ebert, D., Lehr, D., Riper, H., Smit, F., Cuijpers, P., & Berking, M. (2015). Effectiveness of a web-based cognitive behavioural intervention for subthreshold depression: Pragmatic randomised controlled trial. Psychotherapy and Psychosomatics, 84, 348-358. http://dx .doi.org/10.1159/000438673
29. Busner, J., & Targum, S. D. (2007). The clinical global impressions scale: Applying a research tool in clinical practice. Psychiatry, 4, 28-37.
30. Carey, M. P., Spector, I. P., Lantinga, L. J., & Krauss, D. J. (1993). Reliability of the Dyadic Adjustment Scale. Psychological Assessment, 5, 238-240.
31. Carlbring, P., Degerman, N., Jonsson, J., & Andersson, G. (2012). Internet-based treatment of pathological gambling with a three-year follow-up. Cognitive Behaviour Therapy, 41, 321-334. http://dx.doi.org/10.1080/16506073.2012.689323
32. Carlbring, P., Gunnarsdóttir, M., Hedensjö, L., Andersson, G., Ekselius, L., & Furmark, T. (2007). Treatment of social phobia from a distance: A randomized trial of internet delivered cognitive behavior therapy (CBT) and telephone support. British Journal of Psychiatry, 190, 123-128.
33. Carlbring, P., Hägglund, M., Luthström, A., Dahlin, M., Kadowaki, A., Vernmark, K., & Andersson, G. (2013). Internet-based behavioral activation and acceptance-based treatment for depression: A randomized controlled trial. J Affect Disord, 148, 331-337. http://dx.doi.org/10 .1016/j.jad.2012.12.020
34. Carlbring, P., Lindner, P., Martell, C., Hassmen, P., Forsberg, L., Strom, L., & Andersson, G. (2013). The effects on depression of Internetadministered behavioural activation and physical exercise with treatment rationale and relapse prevention: Study protocol for a randomised controlled trial. Trials, 14. http://dx.doi.org/10.1186/1745-6215-14-35
35. Carlbring, P., Maurin, L., Törngren, C., Linna, E., Eriksson, T., Sparthan, E., Andersson, G. (2011). Individually-tailored, Internet-based treatment for anxiety disorders: A randomized controlled trial. Behaviour Research and Therapy, 49, 18-24. http://dx.doi.org/10.1016/j.brat .2010.10.002
36. Carlbring, P., Nilsson-Ihrfelt, E., Waara, J., Kollenstam, C., Buhrman, M., Kaldo, V., Andersson, G. (2005). Treatment of panic disorder: live therapy vs self-help via the Internet. Behaviour Research and Therapy, 43, 1321-1333. http://dx.doi.org/10.1016/j.brat.2004.10.002
37. Castonguay, L. G., Boswell, J. F., Constantino, M. J., Goldfried, M. R., & Hill, C. E. (2010). Training implications of harmful effects of psychological treatments. American Psychologist, 65, 34-49. http://dx.doi.org/10.1037/a0017330
38. Christensen, H., Griffiths, K. M., & Farrer, L. (2009). Adherence in Internet interventions for anxiety and depression: Systematic review. Journal of Medicine Internet Research, 11, e13.
39. Clarke, M. J. (2005). Individual patient data meta-analyses. Best Practice & Research Clinical Obstetrics & Gynaecology, 19, 47-55. http://dx.doi .org/10.1016/j.bpobgyn.2004.10.011
40. Cooper, A. A., & Conklin, L. R. (2015). Dropout from individual psychotherapy for major depression: A meta-analysis of randomized clinical trials. Clinical Psychology Review, 40, 57-65. http://dx.doi.org/10.1016/j.cpr.2015.05.001
41. Cuijpers, P., Donker, T., van Straten, A., Li, J., & Andersson, G. (2010). Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders A systematic review and meta-analysis of comparative outcome studies. Psychological Medicine, 40, 1943-1957. http://dx.doi.org/10.1017/S0033291710000772
42. Dagöö, J., Asplund, R. P., Bsenko, H. A., Hjerling, S., Holmberg, A., Westh, S., Andersson, G. (2014). Cognitive behavior therapy versus interpersonal psychotherapy for social anxiety disorder delivered via smartphone and computer: A randomized controlled trial. Journal of Anxiety Disorders, 28, 410-417. http://dx.doi.org/10.1016/j.janxdis .2014.02.003
43. Dahlin, M., Andersson, G., Magnusson, K., Johansson, T., Sjögren, J., Hakansson, A., Carlbring, P. (2016). Internet-delivered acceptancebased behaviour therapy for generalized anxiety disorder: A randomized controlled trial. Behaviour Research and Therapy, 77, 86-95. http://dx .doi.org/10.1016/j.brat.2015.12.007
44. Davydov, D. M., Stewart, R., Ritchie, K., & Chaudieu, I. (2010). Resilience and mental health. Clinical Psychology Review, 30, 479-495. http://dx.doi.org/10.1016/j.cpr.2010.03.003
45. Debray, T. P., Moons, K. G. M., Abo-Zaid, G. M. A., Koffijberg, H., & Riley, R. D. (2013). Individual participant data meta-analysis for a binary outcome: One-stage or two-stage PLoS ONE, 8.
46. Debray, T. P., Moons, K. G., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H., the GetReal Methods Review Group. (2015). Get real in individual participant data (IPD) meta-analysis: A review of the methodology. Research Synthesis Methods, 6, 293-309. http://dx.doi .org/10.1002/jrsm.1160
47. Dimidjian, S., & Hollon, S. D. (2010). How would we know if psychotherapy were harmful American Psychologist, 65, 21-33. http://dx.doi .org/10.1037/a0017299
48. Edwards, D. W., Yarvis, R. M., Mueller, D. P., Zingale, H. C., & Wagman, W. J. (1978). Test taking and the stability of adjustment scales: Can we assess patient deterioration Evaluation Quarterly, 2, 275-291. http://dx.doi.org/10.1177/0193841X7800200206
49. Evans, C., Margison, F., & Barkham, M. (1998). The contribution of reliable and clinically significant change methods to evidence-based mental health. Evidence-Based Mental Health, 1, 70-72. http://dx.doi .org/10.1136/ebmh.1.3.70
50. Evans, C., Mellor-Clark, J., Margison, F., Barkham, M., Audin, K., Connell, J., & McGrath, G. (2000). CORE: Clinical outcomes in routine evaluation. Journal of Mental Health, 9, 247-255. http://dx.doi.org/10 .1080/713680250
51. Fantino, B., & Moore, N. (2009). The self-reported Montgomery-Asberg Depression rating scale is a useful evaluative tool in major depressive disorder. BMC Psychiatry, 9, 26.
52. First, M. B., Gibbon, M., Spitzer, R. L., & Williams, J. B. (1997). Structured clinical interview for DSM-IV Axis I Disorders (SCID-I). Washington, DC: American Psychiatric Press.
53. Furmark, T., Carlbring, P., Hedman, E., Sonnenstein, A., Clevberger, P., Bohman, B., Andersson, G. (2009). Bibliotherapy and Internet delivered cognitive behaviour therapy with additional guidance for social anxiety disorder: randomised controlled trial. British Journal of Psychiatry, 195, 440-447.
54. Furukawa, T. A., Noma, H., Caldwell, D. M., Honyashiki, M., Shinohara, K., Imai, H., Churchill, R. (2014). Waiting list may be a nocebo condition in psychotherapy trials: A contribution from network metaanalysis. Acta Psychiatrica Scandinavica, 130, 181-192. http://dx.doi .org/10.1111/acps.12275
55. Gerstein, D. R., Volberg, R. A., Toce, M. T., Harwood, H., Johnson, R. A., Buie, T., Sinclair, S. (1999). Gambling impact and behavior study: Report to the National Gambling Impact Study Commission. Chicago, IL: NORC at the University of Chicago.
56. Hadjistavropoulos, H. D., Pugh, N. E., Hesser, H., & Andersson, G. (2016). Predicting response to therapist-assisted internet-delivered cognitive behavior therapy for depression or anxiety within an open dissemination trial. Behavior Therapy, 47, 155-165. http://dx.doi.org/10.1016/j.beth .2015.10.006
57. Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002). The psychotherapy dose-response effect and its implications for treatment delivery services. Clinical Psychology: Science and Practice, 9, 329-343. http://dx.doi.org/10.1093/clipsy.9.3.329
58. Harel, O. (2007). Inferences on missing information under multiple imputation and two-stage multiple imputation. Statistical Methodology, 4, 75-89. http://dx.doi.org/10.1016/j.stamet.2006.03.002
59. Heeren, A., Maurange, P., Rossignol, M., Vanhaelen, M., Peschard, V., Eeckhout, C., & Phillippot, P. (2012). Self-report version of the Liebowitz Social Anxiety Scale: Psychometric properties of the French version. Canadian Journal of Behavioural Science, 44, 99-107.
60. Hesser, H., Weise, C., Rief, W., & Andersson, G. (2011). The effect of waiting: A meta-analysis of waitlist control groups in trials for tinnitus distress. Journal of Psychosomatic Research, 70, 378-384. http://dx.doi .org/10.1016/j.jpsychores.2010.12.006
61. Hofmann, S. G., Asnaani, A., Vonk, I. J. J., Sawyer, A. T., & Fang, A. (2012). The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognitive Therapy and Research, 36, 427-440. http://dx .doi.org/10.1007/s10608-012-9476-1
62. Hsu, L. M. (1995). Regression toward the mean associated with measurement error and the identification of improvement and deterioration in psychotherapy. Journal of Consulting and Clinical Psychology, 63, 141-144. http://dx.doi.org/10.1037/0022-006X.63.1.141
63. -Ivarsson, D., Blom, M., Hesser, H., Carlbring, P., Enderby, P., Nordberg, R., & Andersson, G. (2014). Guided Internet-delivered cognitive behaviour therapy for post-traumatic stress disorder: A randomized controlled trial. Internet Interventions, 1, 33-40. http://dx.doi.org/10.1016/j.invent .2014.03.002
64. Jacobson, N. S., Follette, W. C., & Revenstorf, D. (1984). Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behavior Therapy, 15, 336-352. http://dx.doi.org/10.1016/S0005-7894(84)80002-7
65. Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12-19. http://dx.doi .org/10.1037/0022-006X.59.1.12
66. Jolani, S., Debray, T. P. A., Koffijberg, H., van Buuren, S., & Moons, K. G. M. (2015). Imputation of systematically missing predictors in an individual participant data meta-analysis: A generalized approach using MICE. Statistics in Medicine, 34, 1841-1863. http://dx.doi.org/10.1002/sim.6451
67. Jonsson, U., Alaie, I., Parling, T., & Arnberg, F. K. (2014). Reporting of harms in randomized controlled trials of psychological interventions for mental and behavioral disorders: A review of current practice. Contemporary Clinical Trials, 38, 1-8. http://dx.doi.org/10.1016/j.cct.2014.02 .005
68. Karyotaki, E., Kleiboer, A., Smit, F., Turner, D. T., Pastor, A. M., Andersson, G., Cuijpers, P. (2015). Predictors of treatment dropout in self-guided web-based interventions for depression: An "individual patient data" meta-analysis. Psychological Medicine, 45, 2717-2726. http://dx.doi.org/10.1017/S0033291715000665
69. Klorman, R., Weerts, T. C., Hastings, J. E., Melamed, B. G., & Lang, P. J. (1974). Psychometric properties of some specific-fear questionnaires. Behavior Therapy, 5, 401-409.
70. Kuckertz, J. M., Gildebrant, E., Liliequist, B., Karlstrom, P., Vappling, C., Bodlund, O., Carlbring, P. (2014). Moderation and mediation of the effect of attention training in social anxiety disorder. Behaviour Research and Therapy, 53, 30-40. http://dx.doi.org/10.1016/j.brat.2013.12 .003
71. Lambert, M. (2007). Presidential address: What we have learned from a decade of research aimed at improving psychotherapy outcome in routine care. Psychotherapy Research, 17, 1-14. http://dx.doi.org/10.1080/10503300601032506
72. Lambert, M. J. (2013). Outcome in psychotherapy: The past and important advances. Psychotherapy, 50, 42-51. http://dx.doi.org/10.1037/a0030682
73. Lancee, J., Eisma, M. C., van Straten, A., & Kamphuis, J. H. (2015). Sleep-related safety behaviors and dysfunctional beliefs mediate the efficacy of online CBT for insomnia: A randomized controlled trial. Cognitive Behaviour Therapy, 44, 406-422. http://dx.doi.org/10.1080/16506073.2015.1026386
74. Lee, E. H., Kim, J. H., & Yu, B. H. (2009). Reliability and validity of the self-report version of the Panic Disorder Severity Scale in Korea. Depression and Anxiety, 26, E120-E123.
75. Liebowitz, M. R. (1987). Social phobia. Modern Problems of Pharmacopsychiatry, 22, 141-173. http://dx.doi.org/10.1159/000414022
76. Lilienfeld, S. O. (2007). Psychological treatments that cause harm. Perspectives on Psychological Science, 2, 53-70. http://dx.doi.org/10.1111/j.1745-6916.2007.00029.x
77. Linden, M. (2013). How to define, find and classify side effects in psychotherapy: From unwanted events to adverse treatment reactions. Clinical Psychology & Psychotherapy, 20, 286-296. http://dx.doi.org/10 .1002/cpp.1765
78. Lindner, P., Ivanova, E., Ly, K., Andersson, G., & Carlbring, P. (2013). Guided and unguided CBT for social anxiety disorder and/or panic disorder via the Internet and a smartphone application: study protocol for a randomised controlled trial. Trials, 14, 437. http://dx.doi.org/10.1186/1745-6215-14-437
79. Martinez, R., Whitfield, G., Dafters, R., & Williams, C. (2008). Can people read self-help manuals for depression A challenge for the stepped care model and book prescription schemes. Behavioural and Cognitive Psychotherapy, 36, 89-97.
80. Martinovich, Z., Saunders, S., & Howard, K. (1996). Some comments on "assessing clinical significance." Psychotherapy Research, 6, 124-132. http://dx.doi.org/10.1080/10503309612331331648
81. May, P. R. A. (1971). For better or for worse Psychotherapy and variance change: A critical review of the literature. Journal of Nervous and Mental Disease, 152, 184-192. http://dx.doi.org/10.1097/00005053-197103000-00004
82. McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments. A review of current efforts. American Psychologist, 65, 73-84. http://dx.doi.org/10 .1037/a0018121
83. -Miloff, A., Marklund, A., & Carlbring, P. (2015). The Challenger app for social anxiety disorder: New advances in mobile psychological treatment. Internet Interventions, 2, 382-391. http://dx.doi.org/10.1016/j .invent.2015.08.001
84. Mohr, D. C., Beutler, L. E., Engle, D., Shoham-Salomon, V., Bergan, J., Kaszniak, A. W., & Yost, E. B. (1990). Identification of patients at risk for nonresponse and negative outcome in psychotherapy. Journal of Consulting and Clinical Psychology, 58, 622-628. http://dx.doi.org/10 .1037/0022-006X.58.5.622
85. Muris, P., & Merckelbach, H. (1996). A comparison of two spider fear questionnaires. Journal of Behavior Therapy and Experimental Psychiatry, 27, 241-244.
86. Nordgren, L. B., Hedman, E., Etienne, J., Bodin, J., Kadowaki, A., Eriksson, S., Carlbring, P. (2014). Effectiveness and costeffectiveness of individually tailored Internet-delivered cognitive behavior therapy for anxiety disorders in a primary care population: A randomized controlled trial. Behaviour Research and Therapy, 59, 1-11. http://dx.doi.org/10.1016/j.brat.2014.05.007
87. Oxman, A. D., Clarke, M. J., & Stewart, L. A. (1995). From science to practice. Meta-analyses using individual patient data are needed. Journal of the American Medical Association, 274, 845-846. http://dx.doi.org/10.1001/jama.1995.03530100085040
88. Pallesen, S., Nordhus, I. H., Carlstedt, B., Thayer, J. F., & Johnsen, T. B. (2006). A Norwegian adaption of the Penn State Worry Questionnaire: Factor structure, reliability, validity and norms. Scandinavian Journal of Psychology, 47, 281-291.
89. Posternak, M. A., & Miller, I. (2001). Untreated short-term course of major depression: A meta-analysis of outcomes from studies using waitlist control groups. Journal of Affective Disorders, 66, 139-146. http://dx .doi.org/10.1016/S0165-0327(00)00304-9
90. Powers, E., & Witmer, H. L. (1951). An experiment in the prevention of delinquency: The Cambridge Somerville youth study. New York, NY: Columbia University Press.
91. Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010). Meta-analysis of individual participant data: Rationale, conduct, and reporting. British Medical Journal (Clinical Research Ed.), 340, c221. http://dx.doi.org/10.1136/bmj.c221
92. Rosen, R. C., Riley, A., Wagner, G., Osterloh, I. H., Kirkpatrick, J., & Mishra, A. (1997). The international index of erectile function (IIEF): A multidimensional scale for assessment of erectile dysfunction. Urology, 49, 822-830.
93. Rozental, A., Andersson, G., Boettcher, J., Ebert, D. D., Cuijpers, P., Knaevelsrud, C., Carlbring, P. (2014). Consensus statement on defining and measuring negative effects of Internet interventions. Internet Interventions, 1, 12-19. http://dx.doi.org/10.1016/j.invent.2014.02 .001
94. Rozental, A., Boettcher, J., Andersson, G., Schmidt, B., & Carlbring, P. (2015). Negative effects of internet interventions: A qualitative content analysis of patients' experiences with treatments delivered online. Cognitive Behaviour Therapy, 44, 223-236. http://dx.doi.org/10.1080/16506073.2015.1008033
95. Rozental, A., Kottorp, A., Boettcher, J., Andersson, G., & Carlbring, P. (2016). Negative effects of psychological treatments: An exploratory factor analysis of the negative effects questionnaire for monitoring and reporting adverse and unwanted events. PLoS ONE, 11, e0157503. http://dx.doi.org/10.1371/journal.pone.0157503
96. Saemundsson, B. R., Borsdottir, F., Kristjansdottir, H., Olason, D. B., Smari, J., & Sigurdsson, J. F. (2011). Psychometric properties of the Icelandic version of the Beck Anxiety Inventory in a clinical and a student population. European Journal of Psychological Assessment, 27, 133-141.
97. Scott, K. M., Wells, J. E., Angermeyer, M., Brugha, T. S., Bromet, E., Demyttenaere, K., Kessler, R. C. (2010). Gender and the relationship between marital status and first onset of mood, anxiety and substance use disorders. Psychological Medicine, 40, 1495-1505. http://dx.doi.org/10 .1017/S0033291709991942
98. Sharf, J., Primavera, L. H., & Diener, M. J. (2010). Dropout and therapeutic alliance: A meta-analysis of adult individual psychotherapy. Psychotherapy, 47, 637-645. http://dx.doi.org/10.1037/a0021175
99. Sharpe, J. P., & Gilbert, D. G. (1998). Effects of repeated administration of the Beck Depression Inventory and other measures of negative mood states. Personality and Individual Differences, 24, 457-463. http://dx .doi.org/10.1016/S0191-8869(97)00193-1
100. Sheehan, D. V., Lecrubier, Y., Sheehan, K. H., Amorim, P., Janavs, J., Weiller, E., Dunbar, G. C. (1998). The Mini-International Neuropsychiatric Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. The Journal of Clinical Psychiatry, 59, 22-33.
101. Simmonds, M. C., Higgins, J. P. T., Stewart, L. A., Tierney, J. F., Clarke, M. J., & Thompson, S. G. (2005). Meta-analysis of individual patient data from randomized trials: A review of methods used in practice. Clinical Trials, 2, 209-217. http://dx.doi.org/10.1191/1740774505cn087oa
102. Speer, D. C. (1992). Clinically significant change: Jacobson and Truax (1991) revisited. Journal of Consulting and Clinical Psychology, 60, 402-408. http://dx.doi.org/10.1037/0022-006X.60.3.402
103. Spitzer, R. L., Kroenke, K., Williams, J. W., & Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166, 1092-1097.
104. Stawski, R. S., Sliwinski, M. J., Almeida, D. M., & Smyth, J. M. (2008). Reported exposure and emotional reactivity to daily stressors: The roles of adult age and global perceived stress. Psychology and Aging, 23, 52-61. http://dx.doi.org/10.1037/0882-7974.23.1.52
105. Stewart, G., Altman, D. G., Askie, L. M., Duley, L., Simmonds, M. C., & Stewart, L. A. (2012). Statistical analysis of individual participant data meta-analyses: A comparison of methods and recommendations for practice. PLoS ONE, 7, e46042
106. Stewart, L. A., & Tierney, J. F. (2002). To IPD or not to IPD Advantages and disadvantages of systematic reviews using individual patient data. Evaluation & the Health Professions, 25, 76-97. http://dx.doi.org/10 .1177/0163278702025001006
107. Ström, M., Uckelstam, C.-J., Andersson, G., Hassmén, P., Umefjord, G., & Carlbring, P. (2013). Internet-delivered therapist-guided physical activity for mild to moderate depression: A randomized controlled trial. PeerJ, 1, e178. http://dx.doi.org/10.7717/peerj.178
108. Strupp, H. H., & Hadley, S. W. (1977). A tripartite model of mental health and therapeutic outcomes: With special reference to negative effects in psychotherapy. American Psychologist, 32, 187-196. http://dx.doi.org/10.1037/0003-066X.32.3.187
109. Sundin, E. C., & Horowitz, M. J. (2002). Impact of Event Scale: Psychometric properties. British Journal of Psychiatry, 180, 205-209.
110. Swift, J. K., & Greenberg, R. P. (2012). Premature discontinuation in adult psychotherapy: A meta-analysis. Journal of Consulting and Clinical Psychology, 80, 547-559. http://dx.doi.org/10.1037/a0028226
111. Tillfors, M., Carlbring, P., Furmark, T., Lewenhaupt, S., Spak, M., Eriksson, A., Andersson, G. (2008). Treating university students with social phobia and public speaking fears: Internet delivered self-help with or without live group exposure sessions. Depression and Anxiety, 25, 708-717. http://dx.doi.org/10.1002/da.20416
112. Tingey, R., Lambert, M., Burlingame, G., & Hansen, N. (1996). Assessing clinical significance: Proposed extensions to method. Psychotherapy Research, 6, 109-123. http://dx.doi.org/10.1080/10503309612331331638
113. Titov, N., Andrews, G., Kemp, A., & Robinson, E. (2010). Characteristics of adults with anxiety or depression treated at an Internet clinic: Comparison with a national survey and an outpatient clinic. PLoS ONE, 5, e10885.
114. van Buuren, S., & Groothuis-Oudshoorn, K. (2011). MICE: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45, 1-67.
115. Vaughan, B., Goldstein, M. H., Alikakos, M., Cohen, L. J., & Serby, M. J. (2014). Frequency of reporting of adverse events in randomized controlled trials of psychotherapy vs. psychopharmacotherapy. Comprehensive Psychiatry, 55, 849-855. http://dx.doi.org/10.1016/j.comppsych .2014.01.001
116. Vernmark, K., Lenndin, J., Bjärehed, J., Carlsson, M., Karlsson, J., Öberg, J., Andersson, G. (2010). Internet administered guided self-help versus individualized e-mail therapy: A randomized trial of two versions of CBT for major depression. Behaviour Research and Therapy, 48, 368-376. http://dx.doi.org/10.1016/j.brat.2010.01.005
117. Vessey, J. T., & Howard, K. I. (1993). Who seeks psychotherapy Psychotherapy: Theory, Research, & Practice, 30, 546-553. http://dx.doi .org/10.1037/0033-3204.30.4.546
118. Walker, S., Liberti, L., McAuslane, N., & Levitan, B. S. (2011). Refining the benefit-risk framework for the assessment of medicines: Valuing and weighting benefit and risk parameters. Clinical Pharmacology and Therapeutics, 89, 179-182. http://dx.doi.org/10.1038/clpt.2010.290
119. Wallace, M. L., Frank, E., & Kraemer, H. C. (2013). A novel approach for developing and interpreting treatment moderator profiles in randomized clinical trials. Journal of the American Medical Association Psychiatry, 70, 1241-1247. http://dx.doi.org/10.1001/jamapsychiatry.2013.1960
120. Waller, R., & Gilbody, S. (2009). Barriers to the uptake of computerized cognitive behavioural therapy: A systematic review of the quantitative and qualitative evidence. Psychological Medicine, 39, 705-712. http://dx.doi.org/10.1017/S0033291708004224
121. Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S. L., & Hawkins, E. J. (2003). Improving the effects of psychotherapy: The use of early identification of treatment failure and problem-solving strategies in routine practice. Journal of Counseling Psychology, 50, 59-68. http://dx.doi.org/10.1037/0022-0167.50.1.59
122. Wise, E. A. (2004). Methods for analyzing psychotherapy outcomes: A review of clinical significance, reliable change, and recommendations for future directions. Journal of Personality Assessment, 82, 50-59. http://dx.doi.org/10.1207/s15327752jpa8201-10
123. Young, C. (2006). What happens when people wait for therapy Assessing the clinical significance of the changes observed over the waiting period for clients referred to a primary care psychology service. Primary Care Mental Health, 4, 113-119.
124. Zuithoff, N. P. A., Vergouwe, Y., Kling, M., Nazareth, I., van Wezep, M. J., Moons, K. G. M., & Geerlings, M. I. (2010). The Patient Health Questionnaire-9 for detection of major depressive disorder in primary care: Consequences of current thresholds in a crossectional study. BMC Family Practice, 11, 98.