Machine learning methods for developing precision treatment rules with observational data

Behaviour Research and Therapy

Volumn 120, Issue , 2019, Pages

  Kessler, Ronald C ^a   Bossarte, Robert M ^b,c   Luedtke, Alex ^d,e   Zaslavsky, Alan M ^a   Zubizarreta, Jose R ^a,f  

^a HARVARD MEDICAL SCHOOL   (United States)

^b WEST VIRGINIA UNIVERSITY   (United States)

^c VETERANS AFFAIRS MEDICAL CENTER   (United States)

^d UNIVERSITY OF WASHINGTON   (United States)

^e Fred Hutchinson Cancer Research Center   (United States)

^f HARVARD UNIVERSITY   (United States)

Author keywords

Clinical decision support; Ensemble machine learning; Personalized treatment; Precision treatment; Super learner

Indexed keywords

ARTICLE; CLINICAL DECISION MAKING; CLINICAL TRIAL PROTOCOL; DATA ANALYSIS; ELECTRONIC MEDICAL RECORD; EVIDENCE BASED MEDICINE; HUMAN; MACHINE LEARNING; MENTAL DISEASE; PERSONALIZED MEDICINE; SAMPLE SIZE; TREATMENT OUTCOME; VALIDATION STUDY; CLINICAL DECISION SUPPORT SYSTEM; FACTUAL DATABASE; INFORMATION RETRIEVAL; OBSERVATIONAL STUDY; SUICIDE; SUICIDE ATTEMPT;

CLINICAL DECISION RULES; DATABASES, FACTUAL; DECISION SUPPORT SYSTEMS, CLINICAL; HUMANS; INFORMATION STORAGE AND RETRIEVAL; MACHINE LEARNING; OBSERVATIONAL STUDIES AS TOPIC; PRECISION MEDICINE; SUICIDE; SUICIDE, ATTEMPTED;

EID: 85067560414     PISSN: 00057967     EISSN: 1873622X     Source Type: Journal    
DOI: 10.1016/j.brat.2019.103412     Document Type: Article

References (134)

1. Abadie, A., Chingos, M.M., West, M.R., Endogenous stratification in randomized experiments. The Review of Economics and Statistics C:4 (2018), 567–580.
2. American Community Survey (ACS), https://www.census.gov/programs-surveys/acs, 2019. (Accessed 15 May 2019)
3. Anestis, M.D., Houtsma, C., The association between gun ownership and statewide overall suicide rates. Suicide and Life-Threatening Behavior 48:2 (2018), 204–217 https://doi.org/10.1111/sltb.12346.
4. Anglemyer, A., Horvath, H.T., Bero, L., Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database of Systematic Reviews, 4, 2014, MR000034 https://doi.org/10.1002/14651858.MR000034.pub2.
5. Angrist, J.D., Imbens, G.W., Rubin, D.B., Identification of causal effects using instrumental variables. Journal of the American Statistical Association 91:434 (1996), 444–455.
6. Baiocchi, M., Cheng, J., Small, D.S., Instrumental variable methods for causal inference. Statistics in Medicine 33:13 (2014), 2297–2340 https://doi.org/10.1002/sim.6128.
7. Barbui, C., Esposito, E., Cipriani, A., Selective serotonin reuptake inhibitors and risk of suicide: A systematic review of observational studies. Canadian Medical Associaiton Journal 180:3 (2009), 291–297 https://doi.org/10.1503/cmaj.081514.
8. Berger, M.L., Sox, H., Willke, R.J., Brixner, D.L., Eichler, H.G., Goettsch, W., et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE special task force on real-world evidence in health care decision making. Pharmacoepidemiology and Drug Safety 26:9 (2017), 1033–1039 https://doi.org/10.1002/pds.4297.
9. Bernert, R.A., Hom, M.A., Roberts, L.W., A review of multidisciplinary clinical practice guidelines in suicide prevention: Toward an emerging standard in suicide risk assessment and management, training and practice. Academic Psychiatry 38:5 (2014), 585–592 https://doi.org/10.1007/s40596-014-0180-1.
10. Bousman, C.A., Arandjelovic, K., Mancuso, S.G., Eyre, H.A., Dunlop, B.W., Pharmacogenetic tests and depressive symptom remission: A meta-analysis of randomized controlled trials. Pharmacogenomics 20:1 (2019), 37–47 https://doi.org/10.2217/pgs-2018-0142.
11. Bryan, C.J., Peterson, A.L., Rudd, M.D., Differential effects of brief CBT versus treatment as usual on posttreatment suicide attempts among groups of suicidal patients. Psychiatric Services 69:6 (2018), 703–709 https://doi.org/10.1176/appi.ps.201700452.
12. Bzdok, D., Meyer-Lindenberg, A., Machine learning for precision psychiatry: Opportunities and challenges. Biological Psychiatry. Cognitive Neuroscience and Neuroimaging 3:3 (2018), 223–230 https://doi.org/10.1016/j.bpsc.2017.11.007.
13. Carter, G., Milner, A., McGill, K., Pirkis, J., Kapur, N., Spittal, M.J., Predicting suicidal behaviours using clinical instruments: Systematic review and meta-analysis of positive predictive values for risk scales. The British Journal of Psychiatry: Journal of Mental Science 210:6 (2017), 387–395 https://doi.org/10.1192/bjp.bp.116.182717.
14. Cattaneo, M.D., Frandsen, B.R., Titiunik, R., Randomization inference in the regression discontinuity design: An application to party advantages in the US Senate. Journal of Causal Inference 3:1 (2015), 1–24 https://doi.org/10.1515/jci-2013-0010.
15. Chan, K.C., Yam, S.C., Zhang, Z., Globally efficient non-parametric inference of average treatment effects by empirical balancing calibration weighting. Journal of the Royal Statistical Society: Series B 78:3 (2016), 673–700 https://doi.org/10.1111/rssb.12129.
16. Chesin, M.S., Brodsky, B.S., Beeler, B., Benjamin-Phillips, C.A., Taghavi, I., Stanley, B., Perceptions of adjunctive mindfulness-based cognitive therapy to prevent suicidal behavior among high suicide-risk outpatient participants. Crisis 39:6 (2018), 451–460 https://doi.org/10.1027/0227-5910/a000519.
17. Chung, D.T., Ryan, C.J., Hadzi-Pavlovic, D., Singh, S.P., Stanton, C., Large, M.M., Suicide rates after discharge from psychiatric facilities: A systematic review and meta-analysis. JAMA Psychiatry 74:7 (2017), 694–702 https://doi.org/10.1001/jamapsychiatry.2017.1044.
18. Cole, S.R., Hernán, M.A., Constructing inverse probability weights for marginal structural models. American Journal of Epidemiology 168:6 (2008), 656–664.
19. Cohen, Z.D., DeRubeis, R.J., Treatment selection in depression. Annual Review of Clinical Psychology 14 (2018), 209–236 https://doi.org/10.1146/annurev-clinpsy-050817-084746.
20. Cooper, J., Steeg, S., Bennewith, O., Lowe, M., Gunnell, D., House, A., et al. Are hospital services for self-harm getting better? An observational study examining management, service provision and temporal trends in england. BMJ Open, 3(11), 2013, e003444 https://doi.org/10.1136/bmjopen-2013-003444.
21. Costello, E.J., Erkanli, A., Copeland, W., Angold, A., Association of family income supplements in adolescence with development of psychiatric and substance use disorders in adulthood among an American Indian population. Journal of the American Medical Association 303:19 (2010), 1954–1960 https://doi.org/10.1001/jama.2010.621.
22. Dahabreh, I.J., Sheldrick, R.C., Paulus, J.K., Chung, M., Varvarigou, V., Jafri, H., Kitsios, G.D., Do observational studies using propensity score methods agree with randomized trials? A systematic comparison of studies on acute coronary syndromes. European Heart Journal 33:15 (2012), 1893–1901 https://doi.org/10.1093/eurheartj/ehs114.
23. DeRubeis, R.J., Cohen, Z.D., Forand, N.R., Fournier, J.C., Gelfand, L.A., Lorenzo-Luaces, L., The personalized advantage index: Translating research on prediction into individualized treatment recommendations. A demonstration. PLoS One, 9(1), 2014, e83875 https://doi.org/10.1371/journal.pone.0083875.
24. Diamond, A., Sekhon, J.S., Genetic matching for estimating causal effects: A general multivariate matching method for achieving balance in observational studies. The Review of Economics and Statistics 95:3 (2013), 932–945 https://doi.org/10.1162/REST_a_00318.
25. Dorresteijn, J.A., Visseren, F.L., Ridker, P.M., Wassink, A.M., Paynter, N.P., Steyerberg, E.W., Cook, N.R., Estimating treatment effects for individual patients based on the results of randomised clinical trials. BMJ, 343, 2011, d5888 https://doi.org/10.1136/bmj.d5888.
26. Durand, C.P., Does raising type 1 error rate improve power to detect interactions in linear regression models? A simulation study. PLoS One, 8(8), 2013, e71079 https://doi.org/10.1371/journal.pone.0071079.
27. Fabbri, C., Crisafulli, C., Calabro, M., Spina, E., Serretti, A., Progress and prospects in pharmacogenetics of antidepressant drugs. Expert Opinion on Drug Metabolism and Toxicology 12:10 (2016), 1157–1168 https://doi.org/10.1080/17425255.2016.1202237.
28. Forkmann, T., Brakemeier, E.L., Teismann, T., Schramm, E., Michalak, J., The effects of mindfulness-based cognitive therapy and cognitive behavioral analysis system of psychotherapy added to treatment as usual on suicidal ideation in chronic depression: Results of a randomized-clinical trial. Journal of Affective Disorders 200 (2016), 51–57 https://doi.org/10.1016/j.jad.2016.01.047.
29. Genetic Links to Anxiety & Depression (GLAD), https://gladstudy.org.uk/, 2019. (Accessed 15 May 2019)
30. Greenland, S., Tests for interaction in epidemiologic studies: A review and a study of power. Statistics in Medicine 2:2 (1983), 243–251.
31. Greist, J.H., Mundt, J.C., Gwaltney, C.J., Jefferson, J.W., Posner, K., Predictive value of baseline electronic Columbia-Suicide Severity Rating Scale (eC-SSRS) assessments for identifying risk of prospective reports of suicidal behavior during research participation. Innovations in Clinical Neuroscience 11:9–10 (2014), 23–31.
32. Gruber, S., Logan, R.W., Jarrin, I., Monge, S., Hernan, M.A., Ensemble learning of inverse probability weights for marginal structural modeling in large observational datasets. Statistics in Medicine 34:1 (2015), 106–117 https://doi.org/10.1002/sim.6322.
33. Hahn, J., Todd, P., Van der Klaauw, W., Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica 69:1 (2001), 201–209 https://doi.org/10.1111/1468-0262.00183.
34. Hainmueller, J., Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis 20:1 (2012), 25–46 https://doi.org/10.1093/pan/mpr025.
35. Handley, M.A., Lyles, C.R., McCulloch, C., Cattamanchi, A., Selecting and improving quasi-experimental designs in effectiveness and implementation research. Annual Review of Public Health 39 (2018), 5–25 https://doi.org/10.1146/annurev-publhealth-040617-014128.
36. Hansen, B.B., Full matching in an observational study of coaching for the SAT. Journal of the American Statistical Association 99:467 (2004), 609–618 https://doi.org/10.1198/016214504000000647.
37. Harned, M.S., Chapman, A.L., Dexter-Mazza, E.T., Murray, A., Comtois, K.A., Linehan, M.M., Treating co-occurring Axis I disorders in recurrently suicidal women with borderline personality disorder: A 2-year randomized trial of dialectical behavior therapy versus community treatment by experts. Journal of Consulting and Clinical Psychology 76:6 (2008), 1068–1075 https://doi.org/10.1037/a0014044.
38. Hirshberg, D.A., Maleki, A., Zubizarreta, J., Minimax linear estimation of the retargeted mean. 2019 arXiv.org, arXiv:190110296.
39. Hirshberg, D.A., Zubizarreta, J.R., On two approaches to weighting in causal inference. Epidemiology 28:6 (2017), 812–816 https://doi.org/10.1097/EDE.0000000000000735.
40. Hjorthoj, C.R., Madsen, T., Agerbo, E., Nordentoft, M., Risk of suicide according to level of psychiatric treatment: A nationwide nested case-control study. Social Psychiatry and Psychiatric Epidemiology 49:9 (2014), 1357–1365 https://doi.org/10.1007/s00127-014-0860-x.
41. Hoffmire, C., Stephens, B., Morley, S., Thompson, C., Kemp, J., Bossarte, R.M., VA Suicide Prevention Applications Network: A national health care system–based suicide event tracking system. Public Health Reports 131:6 (2016), 816–821 https://doi.org/10.1177/0033354916670133.
42. Huh, D., Jobes, D.A., Comtois, K.A., Kerbrat, A.H., Chalker, S.A., Gutierrez, P.M., et al. The collaborative assessment and management of suicidality (CAMS) versus enhanced care as usual (E-CAU) with suicidal soldiers: Moderator analyses from a randomized controlled trial. Military Psychology 30:6 (2018), 459–506 https://doi.org/10.1080/08995605.2018.1503001.
43. Imai, K., Ratkovic, M., Covariate balancing propensity score. Journal of the Royal Statistical Society: Series B 76:1 (2014), 243–263 https://doi.org/10.1111/rssb.12027.
44. Jobes, D.A., Comtois, K.A., Gutierrez, P.M., Brenner, L.A., Huh, D., Chalker, S.A., Crow, B., A randomized controlled trial of the collaborative assessment and management of suicidality versus enhanced care as usual with suicidal soldiers. Psychiatry 80:4 (2017), 339–356 https://doi.org/10.1080/00332747.2017.1354607.
45. Judd, C.M., Westfall, J., Kenny, D.A., Experiments with more than one random factor: Designs, analytic models, and statistical power. Annual Review of Psychology 68 (2017), 601–625 https://doi.org/10.1146/annurev-psych-122414-033702.
46. Kang, J.D.Y., Schafer, J.L., Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science 22:4 (2007), 523–539.
47. Kapur, N., Ibrahim, S., While, D., Baird, A., Rodway, C., Hunt, I.M., et al. Mental health service changes, organisational factors, and patient suicide in England in 1997-2012: A before-and-after study. Lancet Psychiatry 3:6 (2016), 526–534 https://doi.org/10.1016/S2215-0366(16)00063-8.
48. Keele, L., Titiunik, R., Zubizarreta, J.R., Enhancing a geographic regression discontinuity design through matching to estimate the effect of ballot initiatives on voter turnout. Journal of the Royal Statistical Society: Series A 178:1 (2015), 223–239.
49. Kessler, R.C., The potential of predictive analytics to provide clinical decision support in depression treatment planning. Current Opinion in Psychiatry 31:1 (2018), 32–39 https://doi.org/10.1097/YCO.0000000000000377.
50. Kessler, R.C., Bernecker, S.L., Bossarte, R.M., Luedtke, A.R., McCarthy, J.F., Nock, M.K., The role of big data analytics in predicting suicide. Passos, I.C., Mwangi, B., Kapaczinski, F., (eds.) Personalized and predictive psychiatry - big data analytics in mental health, 2019, Springer Nature, Switzerland, 77–98.
51. Kessler, R.C., van Loo, H.M., Wardenaar, K.J., Bossarte, R.M., Brenner, L.A., Ebert, D.D., Zaslavsky, A.M., Using patient self-reports to study heterogeneity of treatment effects in major depressive disorder. Epidemiology and Psychiatric Sciences 26:1 (2017), 22–36 https://doi.org/10.1017/S2045796016000020.
52. Kinard, E.M., Klerman, L.V., Changes in life style following mental hospitalization. The Journal of Nervous and Mental Disease 168:11 (1980), 666–672.
53. Kind, A.J.H., Buckingham, W.R., Making neighborhood-disadvantage metrics accessible - the neighborhood atlas. New England Journal of Medicine 378:26 (2018), 2456–2458 https://doi.org/10.1056/NEJMp1802313.
54. Kovalchik, S.A., Varadhan, R., Weiss, C.O., Assessing heterogeneity of treatment effect in a clinical trial with the proportional interactions model. Statistics in Medicine 32:28 (2013), 4906–4923 https://doi.org/10.1002/sim.5881.
55. van der Laan, M.J., Luedtke, A.R., Targeted learning of the mean outcome under an optimal dynamic treatment rule. Journal of Causal Inference 3:1 (2015), 61–95 https://doi.org/10.1515/jci-2013-0022.
56. van der Laan, M.J., Polley, E.C., Hubbard, A.E., Super learner. Statistical Applications in Genetics and Molecular Biology, 6(Article25), 2007 https://doi.org/10.2202/1544-6115.1309.
57. Large, M.M., Kapur, N., Psychiatric hospitalisation and the risk of suicide. The British Journal of Psychiatry: Journal of Mental Science 212:5 (2018), 269–273 https://doi.org/10.1192/bjp.2018.22.
58. Large, M.M., Ryan, C.J., Carter, G., Kapur, N., Can we usefully stratify patients according to suicide risk?. BMJ, 359, 2017, j4627 https://doi.org/10.1136/bmj.j4627.
59. LeDell, E., van der Laan, M.J., Petersen, M., AUC-maximizing ensembles through metalearning. International Journal of Biostatistics 12:1 (2016), 203–218 https://doi.org/10.1515/ijb-2015-0035.
60. LexisNexis, Solutions for professionals who shape the world. 2019 https://www.lexisnexis.com/en-us/gateway.page. (Accessed 2 January 2019)
61. Linehan, M.M., Korslund, K.E., Harned, M.S., Gallop, R.J., Lungu, A., Neacsiu, A.D., Murray-Gregory, A.M., Dialectical behavior therapy for high suicide risk in individuals with borderline personality disorder: A randomized clinical trial and component analysis. JAMA Psychiatry 72:5 (2015), 475–482 https://doi.org/10.1001/jamapsychiatry.2014.3039.
62. Lin, J.J., Lu, T.H., Trends in solids/liquids poisoning suicide rates in Taiwan: A test of the substitution hypothesis. BMC Public Health, 11, 2011, 712.
63. Lipkovich, I., Dmitrienko, A., D' Agostino, R.B., D'Agostiino, R.B. Sr., Tutorial in biostatistics: Data-driven subgroup identification and analysis in clinical trials. Statistics in Medicine 36:1 (2017), 136–196 https://doi.org/10.1002/sim.7064.
64. Lipschitz, J.M., Benzer, J.K., Miller, C., Easley, S.R., Leyson, J., Post, E.P., et al. Understanding collaborative care implementation in the Department of Veterans Affairs: Core functions and implementation challenges. BMC Health Services Research, 17(1), 2017, 691 https://doi.org/10.1186/s12913-017-2601-9.
65. Lubin, G., Werbeloff, N., Halperin, D., Shmushkevitch, M., Weiser, M., Knobler, H.Y., Decrease in suicide rates after a change of policy reducing access to firearms in adolescents: A naturalistic epidemiological study. Suicide and Life-Threatening Behavior, 40(5), 2010, 10.1521/suli.2010.40.5.421 424-424.
66. Luedtke, A., Chambaz, A., Faster rates for policy learning. 2017 arXiv.org, arXiv:1704.06431.
67. Luedtke, A., Sadikova, E., Kessler, R.C., Sample size requirements for multivariate models to predict between-patient differences in best treatments of major depressive disorder. Clinical Psychological Science, 7(3), 2019 https://doi.org/10.1177/2167702618815466.
68. Luedtke, A.R., van der Laan, M.J., Evaluating the impact of treating the optimal subgroup. Statistical Methods in Medical Research 26:4 (2017), 1630–1640 https://doi.org/10.1177/0962280217708664.
69. McClellan, M., McNeil, B.J., Newhouse, J.P., Does more intensive treatment of acute myocardial infarction in the elderly reduce mortality? Analysis using instrumental variables. Journal of the American Medical Association 272:11 (1994), 859–866.
70. McCoy, T.H. Jr., Castro, V.M., Roberson, A.M., Snapper, L.A., Perlis, R.H., Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing. JAMA Psychiatry 73:10 (2016), 1064–1071 https://doi.org/10.1001/jamapsychiatry.2016.2172.
71. McCoy, T.H. Jr., Pellegrini, A.M., Perlis, R.H., Research Domain Criteria scores estimated through natural language processing are associated with risk for suicide and accidental death. Depression and Anxiety 36:5 (2019), 392–399 https://doi.org/10.1002/da.22882.
72. McIntosh, A.M., Stewart, R., John, A., Smith, D.J., Davis, K., Sudlow, C., Group, M.Q.D.S., Data science for mental health: A UK perspective on a global challenge. Lancet Psychiatry 3:10 (2016), 993–998 https://doi.org/10.1016/S2215-0366(16)30089-X.
73. McMain, S.F., Fitzpatrick, S., Boritz, T., Barnhart, R., Links, P., Streiner, D.L., Outcome trajectories and prognostic factors for suicide and self-harm behaviors in patients with borderline personality disorder following one year of outpatient psychotherapy. Journal of Personality Disorders 32:4 (2018), 497–512 https://doi.org/10.1521/pedi_2017_31_309.
74. Menke, A., Precision pharmacotherapy: Psychiatry's future direction in preventing, diagnosing, and treating mental disorders. Pharmgenomics and Personalized Medicine 11 (2018), 211–222 https://doi.org/10.2147/PGPM.S146110.
75. Moscoe, E., Bor, J., Bärnighausen, T., Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: A review of current and best practice. Journal of Clinical Epidemiology 68:2 (2015), 122–133, 10.1016/j.jclinepi.2014.06.021.
76. Mulder, R., Newton-Howes, G., Coid, J.W., The futility of risk prediction in psychiatry. The British Journal of Psychiatry: Journal of Mental Science 209:4 (2016), 271–272 https://doi.org/10.1192/bjp.bp.116.184960.
77. Murphy, S.A., Optimal dynamic treatment regimes. Journal of the Royal Statistical Society - Series B: Statistical Methodology 65:2 (2003), 331–355.
78. National Institutes of Health, All of us research program. 2019 https://allofus.nih.gov/. (Accessed 15 May 2019)
79. Ning, J., Hong, C., Li, L., Huang, X., Shen, Y., Estimating treatment effects in observational studies with both prevalent and incident cohorts. Canadian Journal of Statistics 45:2 (2017), 202–219 https://doi.org/10.1002/cjs.11317.
80. Nock, M.K., Park, J.M., Finn, C.T., Deliberto, T.L., Dour, H.J., Banaji, M.R., Measuring the suicidal mind: Implicit cognition predicts suicidal behavior. Psychological Science 21:4 (2010), 511–517 https://doi.org/10.1177/0956797610364762.
81. Noma, H., Tanaka, S., Analysis of case-cohort designs with binary outcomes: Improving efficiency using whole-cohort auxiliary information. Statistical Methods in Medical Research 26:2 (2017), 691–706, 10.1177/0962280214556175.
82. Nordt, C., Warnke, I., Seifritz, E., Kawohl, W., Modelling suicide and unemployment: A longitudinal analysis covering 63 countries, 2000-11. Lancet Psychiatry 2:3 (2015), 239–245 https://doi.org/10.1016/S2215-0366(14)00118-7.
83. Olbrich, S., van Dinteren, R., Arns, M., Personalized medicine: Review and perspectives of promising baseline EEG biomarkers in major depressive disorder and attention deficit hyperactivity disorder. Neuropsychobiology 72:3–4 (2015), 229–240 https://doi.org/10.1159/000437435.
84. Ondra, T., Dmitrienko, A., Friede, T., Graf, A., Miller, F., Stallard, N., et al. Methods for identification and confirmation of targeted subgroups in clinical trials: A systematic review. Journal of Biopharmaceutical Statistics 26:1 (2016), 99–119 https://doi.org/10.1080/10543406.2015.1092034.
85. Paksarian, D., Mojtabai, R., Kotov, R., Cullen, B., Nugent, K.L., Bromet, E.J., Perceived trauma during hospitalization and treatment participation among individuals with psychotic disorders. Psychiatric Services 65:2 (2014), 266–269 https://doi.org/10.1176/appi.ps.201200556.
86. Penkower, L., Bromet, E.J., Dew, M.A., Husbands' layoff and wives' mental health. A prospective analysis. Archives of General Psychiatry 45:11 (1988), 994–1000.
87. Perel, P., Edwards, P., Wentz, R., Roberts, I., Systematic review of prognostic models in traumatic brain injury. BMC Medical Informatics and Decision Making, 6, 2006, 38 https://doi.org/10.1186/1472-6947-6-38.
88. Pestian, J.P., Sorter, M., Connolly, B., Bretonnel Cohen, K., McCullumsmith, C., Gee, J.T., … STM Research Group. A machine learning approach to identifying the thought markers of suicidal subjects: A prospective multicenter trial. Suicide and Life-Threatening Behavior 47:1 (2017), 112–121 https://doi.org/10.1111/sltb.12312.
89. Peterson, K., Anderson, J., Bourne, D., Evidence brief: Use of patient reported outcome measures for measurement-based care in mental health shared decision-making. VA evidence-based synthesis program reports, 2018, Department of Veterans Affairs (US), Washington, DC.
90. Petkova, E., Ogden, R.T., Tarpey, T., Ciarleglio, A., Jiang, B., Su, Z., Trivedi, M.H., Statistical analysis plan for stage 1 EMBARC (establishing moderators and biosignatures of antidepressant response for clinical care) study. Contemporary Clinical Trials Communications 6 (2017), 22–30 https://doi.org/10.1016/j.conctc.2017.02.007.
91. Prendes-Alvarez, S., Nemeroff, C.B., Personalized medicine: Prediction of disease vulnerability in mood disorders. Neuroscience Letters 669 (2018), 10–13 https://doi.org/10.1016/j.neulet.2016.09.049.
92. Prieto-Merino, D., Pocock, S.J., The science of risk models. European Journal of Preventive Cardiology 19:2 Suppl (2012), 7–13 https://doi.org/10.1177/2047487312448995.
93. Quinlivan, L., Cooper, J., Davies, L., Hawton, K., Gunnell, D., Kapur, N., Which are the most useful scales for predicting repeat self-harm? A systematic review evaluating risk scales using measures of diagnostic accuracy. BMJ Open, 6(2), 2016, e009297 https://doi.org/10.1136/bmjopen-2015-009297.
94. Robins, J.M., Optimal structural nested models for optimal sequential decisions. Lin, D.Y., Heagerty, P.J., (eds.) Proceedings of the second Seattle symposium in biostatistics: Analysis of correlated data, 2004, Springer, New York, 189–326.
95. Robins, J.M., Rotnitzky, A., Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association 90:429 (1995), 122–129.
96. Rosellini, A.J., Street, A.E., Ursano, R.J., Chiu, W.T., Heeringa, S.G., Monahan, J., Kessler, R.C., Sexual assault victimization and mental health treatment, suicide attempts, and career outcomes among women in the us army. American Journal of Public Health 107:5 (2017), 732–739 https://doi.org/10.2105/AJPH.2017.303693.
97. Rosenbaum, P.R., Observation and experiment: An introduction to causal inference. 2017, Harvard University Press, Cambridge, Massachusetts.
98. Rosenbaum, P.R., Rubin, D.B., The central role of the propensity score in observational studies for causal effects. Biometrika 70:1 (1983), 41–55.
99. Rubin, D.B., van der Laan, M.J., Statistical issues and limitations in personalized medicine research with clinical trials. The International Journal of Biostatics, 8(1), 2012, 18 https://doi.org/10.1515/1557-4679.1423.
100. Rudd, D.M., Brief cognitive behavioral therapy (BCBT) for suicidality in military populations. Military Psychology 24 (2012), 592–603.
101. Rudd, M.D., Core competencies, warning signs, and a framework for suicide risk assessment in clinical practice. Nock, M.K., (eds.) The Oxford handbook of suicide and self-injury, 1st ed., 2014, Oxford University Press, Cary, 323–336.
102. Rumshisky, A., Ghassemi, M., Naumann, T., Szolovits, P., Castro, V.M., McCoy, T.H., et al. Predicting early psychiatric readmission with natural language processing of narrative discharge summaries. Translational Psychiatry, 6(10), 2016, e921 https://doi.org/10.1038/tp.2015.182.
103. Runeson, B., Odeberg, J., Pettersson, A., Edbom, T., Jildevik Adamsson, I., Waern, M., Instruments for the assessment of suicide risk: A systematic review evaluating the certainty of the evidence. PLoS One, 12(7), 2017, e0180292 https://doi.org/10.1371/journal.pone.0180292.
104. Rupasingha, A., Goetz, S.J., Freshwater, D., The production of social capital in US counties. The Journal of Socio-Economics 35:1 (2006), 83–101 https://doi.org/10.1016/j.socec.2005.11.001.
105. Saltiel, P.F., Silvershein, D.I., Major depressive disorder: Mechanism-based prescribing for personalized medicine. Neuropsychiatric Disease and Treatment 2015:11 (2015), 875–888 https://doi.org/10.2147/NDT.S73261.
106. Silverman, J.J., Galanter, M., Jackson-Triche, M., Jacobs, D.G., Lomax, J.W. 2nd, Riba, M.B., American Psychiatric Association, The American Psychiatric Association practice guidelines for the psychiatric evaluation of adults. American Journal of Psychiatry 172:8 (2015), 798–802 https://doi.org/10.1176/appi.ajp.2015.1720501.
107. Smith, G.C., Seaman, S.R., Wood, A.M., Royston, P., White, I.R., Correcting for optimistic prediction in small data sets. American Journal of Epidemiology 180:3 (2014), 318–324 https://doi.org/10.1093/aje/kwu140.
108. Steyerberg, E.W., Vickers, A.J., Cook, N.R., Gerds, T., Gonen, M., Obuchowski, N., Kattan, M.W., Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology 21:1 (2010), 128–138 https://doi.org/10.1097/EDE.0b013e3181c30fb2.
109. Svindseth, M.F., Dahl, A.A., Hatling, T., Patients' experience of humiliation in the admission process to acute psychiatric wards. Nordic Journal of Psychiatry 61:1 (2007), 47–53 https://doi.org/10.1080/08039480601129382.
110. Swanson, S.A., Instrumental variable analyses in pharmacoepidemiology: What target trials do we emulate?. Current Epidemiology Reports 4:4 (2017), 281–287.
111. Tighe, J., Nicholas, J., Shand, F., Christensen, H., Efficacy of acceptance and commitment therapy in reducing suicidal ideation and deliberate self-harm: Systematic review. Journal of Medical Internet Research Mental Health, 5(2), 2018, e10732 https://doi.org/10.2196/10732.
112. Trivedi, M.H., McGrath, P.J., Fava, M., Parsey, R.V., Kurian, B.T., Phillips, M.L., Weissman, M.M., Establishing moderators and biosignatures of antidepressant response in clinical care (EMBARC): Rationale and design. Journal of Psychiatric Research 78 (2016), 11–23 https://doi.org/10.1016/j.jpsychires.2016.03.001.
113. U.K. Biobank, https://www.ukbiobank.ac.uk/, 2018. (Accessed 2 January 2019)
114. U.S. Department of Veterans Affairs, VA/DoD clinical practice guidelines: Assessment and management of patients at risk for suicide. 2013 https://www.healthquality.va.gov/guidelines/MH/srb/. (Accessed 2 January 2019)
115. VA Office of Public and Intergovernmental Affairs, VA REACH VET Initiative helps save veterans lives: Program signals when more help is needed for at-risk veterans. 2017 https://www.va.gov/opa/pressrel/pressrelease.cfm?id=2878. (Accessed 2 January 2019)
116. VanderWeele, T.J., Luedtke, A.R., van der Laan, M.J., Kessler, R.C., Selecting optimal subgroups for treatment using many covariates. Epidemiology 30:3 (2019), 334–341 https://doi.org/10.1097/EDE.0000000000000991.
117. Venkataramani, A.S., Bor, J., Jena, A.B., Regression discontinuity designs in healthcare research. BMJ, 352, 2016, i1216, 10.1136/bmj.i1216.
118. Vickers, A.J., Van Calster, B., Steyerberg, E.W., Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 352, 2016, i6, 10.1136/bmj.i6.
119. Visconti, G., Zubizarreta, J.R., Handling limited overlap in observational studies with cardinality matching. Observational Studies 4 (2018), 217–249.
120. Waldrop, J., McGuinness, T.M., Measurement-based care in psychiatry. Journal of Psychosocial Nursing and Mental Health Services 55:11 (2017), 30–35 https://doi.org/10.3928/02793695-20170818-01.
121. Wang, Y., Zubizarreta, J.R., Minimal dispersion approximately balancing weights: Asymptotic properties and practical considerations. Biometrika, 2019 in press.
122. While, D., Bickley, H., Roscoe, A., Windfuhr, K., Rahman, S., Shaw, J., et al. Implementation of mental health service recommendations in England and Wales and suicide rates, 1997-2006: A cross-sectional and before-and-after observational study. Lancet 379:9820 (2012), 1005–1012 https://doi.org/10.1016/S0140-6736(11)61712-1.
123. Wigmore, E.M., Hafferty, J.D., Hall, L.S., Howard, D.M., Clarke, T.K., Fabbri, C., et al. Genome-wide association study of antidepressant treatment resistance in a population-based cohort using health service prescription data and meta-analysis with GENDEP. The Pharmacogenomics Journal, 2019 https://doi.org/10.1038/s41397-019-0067-3.
124. Woodford, R., Spittal, M.J., Milner, A., McGill, K., Kapur, N., Pirkis, J., et al. Accuracy of clinician predictions of future self-harm: A systematic review and meta-analysis of predictive studies. Suicide and Life-Threatening Behavior, 2017 https://doi.org/10.1111/sltb.12395.
125. Yiu, S., Su, L., Covariate association eliminating weights: A unified weighting framework for causal effect estimation. Biometrika 105:3 (2018), 709–722 https://doi.org/10.1093/biomet/asy015.
126. Zalsman, G., Hawton, K., Wasserman, D., van Heeringen, K., Arensman, E., Sarchiapone, M., et al. Suicide prevention strategies revisited: 10-year systematic review. Lancet Psychiatry 3:7 (2016), 646–659 https://doi.org/10.1016/S2215-0366(16)30030-X.
127. Zhang, B., Tsiatis, A.A., Davidian, M., Zhang, M., Laber, E., Estimating optimal treatment regimes from a classification perspective. Stat 1:1 (2012), 103–114 https://doi.org/10.1002/sta.411.
128. Zhang, Y., Zhang, O.R., Li, R., Flores, A., Selek, S., Zhang, X.Y., et al. Psychiatric stressor recognition from clinical notes to reveal association with suicide. Health Informatics Journal, 1460458218796598, 2018 https://doi.org/10.1177/1460458218796598.
129. Zhao, Y., Zeng, D., Rush, A.J., Kosorok, M.R., Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association 107:449 (2012), 1106–1118 https://doi.org/10.1080/01621459.2012.695674.
130. Zhou, X., Mayer-Hamblett, N., Khan, U., Kosorok, M.R., Residual weighted learning for estimating individualized treatment rules. Journal of the American Statistical Association 112:517 (2017), 169–187 https://doi.org/10.1080/01621459.2015.1093947.
131. Zhu, R., Zhao, Y.Q., Chen, G., Ma, S., Zhao, H., Greedy outcome weighted tree learning of optimal personalized treatment rules. Biometrics 73:2 (2017), 391–400 https://doi.org/10.1111/biom.12593.
132. Zubizarreta, J.R., Using mixed integer programming for matching in an observational study of kidney failure after surgery. Journal of the American Statistical Association 107:500 (2012), 1360–1371.
133. Zubizarreta, J.R., Stable weights that balance covariates for estimation with incomplete outcome data. Journal of the American Statistical Association 110:511 (2015), 910–922 https://doi.org/10.1080/01621459.2015.1023805.
134. Zubizarreta, J.R., Small, D.S., Goyal, N.K., Lorch, S., Rosenbaum, P.R., Stronger instruments via integer programming in an observational study of late preterm birth outcomes. Annals of Applied Statistics 7:1 (2013), 25–50.