Predictive Validity Performance Indicators in Violence Risk Assessment: A Methodological Primer

Behavioral Sciences and the Law

Volumn 31, Issue 1, 2013, Pages 8-22

  Singh, Jay P ^a,b  

^a UNIVERSITY OF SOUTH FLORIDA   (United States)

^b MOLDE UNIVERSITY COLLEGE   (Norway)

Author keywords

[No Author keywords available]

Indexed keywords

AREA UNDER THE CURVE; ARTICLE; FORECASTING; FORENSIC PSYCHIATRY; HUMAN; MALE; METHODOLOGY; RISK; RISK ASSESSMENT; SENSITIVITY AND SPECIFICITY; STANDARD; STATISTICAL MODEL; VIOLENCE;

AREA UNDER CURVE; FORECASTING; FORENSIC PSYCHIATRY; HUMANS; LOGISTIC MODELS; MALE; ODDS RATIO; RISK ASSESSMENT; SENSITIVITY AND SPECIFICITY; VIOLENCE;

EID: 84874337530     PISSN: 07353936     EISSN: 10990798     Source Type: Journal    
DOI: 10.1002/bsl.2052     Document Type: Article

References (93)

1. Altman, D. G., & Bland, J. M. (1994a). Diagnostic tests 1. Sensitivity and specificity. British Medical Journal, 308, 1552. DOI:10.1136/bmj.308.6943.1552.
2. Altman, D. G., & Bland, J. M. (1994b). Diagnostic tests 2: Predictive values. British Medical Journal, 309, 102. DOI:10.1136/bmj.309.6947.102.
3. Altman, D. G., & Bland, J. M. (1994c). Diagnostic tests 3: Receiver operating characteristic plots. British Medical Journal, 309, 188. DOI:10.1136/bmj.309.6948.188.
4. Andrews, D. A., & Bonta, J. (2010). Rehabilitating criminal justice policy and practice. Psychology, Public Policy, & Law, 16, 39-55. DOI:10.1037/a0018362.
5. Andrews, D. A., & Bonta, J. (1995). LSI-R: The Level of Service Inventory - Revised. Toronto, ON: Multi-Health Systems.
6. Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005). Researchers misunderstand confidence intervals and standard error bars. Psychological Methods, 10, 389-396. DOI:10.1037/1082-989X.10.4.389.
7. Bennett, B. M. (1972). On comparisons of sensitivity, specificity and predictive value of a number of diagnostic procedures. Biometrics, 28, 793-800.
8. Böhning, D., Holling, H., & Patilea, V. (2011). A limitation of the diagnostic-odds ratio in determining an optimal cut-off value for a continuous diagnostic test. Statistical Methods in Medical Research, 20, 541-550. DOI:10.1177/0962280210374532.
9. Breaugh, J. A. (2003). Effect size estimation: Factors to consider and mistakes to avoid. Journal of Management, 29, 79-97. DOI:10.1177/014920630302900106.
10. Brenner, H., & Gefeller, O. (1997). Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Statistics in Medicine, 16, 981-991. DOI:10.1002/(SICI)1097-0258(19970515)16:9<981::AID-SIM510>3.0.CO;2-N.
11. Breslow, N., & Day, N. (1987). Statistical methods in cancer research. Lyon: International Agency for Research on Cancer.
12. Chambless, L. E., & Diao, G. (2006). Estimation of time-dependent area under the ROC curve for long-term risk prediction. Statistics in Medicine, 25, 3474-3486. DOI:10.1002/sim.2299.
13. Cook, N. R. (2008). Statistical evaluation of prognostic versus diagnostic models: Beyond the ROC curve. Clinical Chemistry, 54, 17-23. DOI:10.1373/clinchem.2007.096529.
14. Cook, N. R. (2007). Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation, 115, 928-935. DOI:10.1161/CIRCULATIONAHA.106.672402.
15. Cook, R. J., & Sackett, D. L. (1995). The number needed to treat: A clinically useful measure of treatment effect. British Medical Journal, 310, 452-454. DOI:10.1136/bmj.310.6977.452.
16. Das Gupta, S. (1960). Point biserial correlation coefficient and its generalization. Psychometrika, 25, 393-408. DOI:10.1007/BF02289756.
17. Davies, H. T. O., Crombie, I. K., & Tavakoli, M. (1998). When can odds ratios mislead? British Medical Journal, 316, 989. DOI:10.1136/bmj.316.7136.989.
18. DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44, 837-845.
19. Douglas, K. S., Otto, R. K., Desmarais, S. L., & Borum, R. (2012). Clinical forensic psychology. In I. B. Weiner, J. A. Schinka, & W. F. Velicer (Eds.), Handbook of psychology, volume 2: Research methods in psychology. (pp. 213-244) Hoboken, NJ: John Wiley & Sons.
20. Farrington, D. P., & Loeber, R. (2000). Some benefits of dichotomization in psychiatric and criminological research. Criminal Behaviour & Mental Health, 10, 100-122. DOI:10.1002/cbm.349.
21. Fazel, S., Singh, J. P., Doll, H., & Grann, M. (2012). Use of risk assessment instruments to predict violence and antisocial behaviour in 73 samples involving 24, 827 people: Systematic review and meta-analysis. British Medical Journal, 345, e4692. DOI:10.1136/bmj.e4692.
22. Fisher, R. A. (1924). On a distribution yielding the error functions of several well known statistics. Proceedings of the International Congress of Mathematics, 2, 805-813.
23. Fleminger, S. (1997). Number needed to detain. British Journal of Psychiatry, 171, 287. DOI:10.1192/bjp.171.3.287a.
24. Furukawa, T. A. (1999). From effect size into number needed to treat. Lancet, 353, 1680.
25. Glas, A. S., Lijmer, J. G., Prins, M. H., Bonsel, G. J., & Bossuyt, P. M. M. (2003). The diagnostic odds ratio: A single indicator of test performance. Journal of Clinical Epidemiology, 56, 1129-1135. DOI:10.1016/S0895-4356(03)00177-X.
26. Gradstein, M. (1986). Maximal correlation between normal and dichotomous variables. Journal of Educational Statistics, 11, 259-261. DOI:10.3102/10769986011004259.
27. Grimes, D. A. & Schulz, K. F. (2008). Making sense of odds and odds ratios. Obstetrics & Gynecology, 111, 423-426. DOI:10.1097/01.AOG.0000297304.32187.5d.
28. Grimes, D. A., & Schulz, K. F. (2002). Bias and causal associations in observational research. Lancet, 359, 248-252. DOI:10.1016/S0140-6736(02)07451-2.
29. Guggenmoos-Holzmann, I., & van Houwelingen, H. C. (2000). The (in)validity of sensitivity and specificity. Statistics in Medicine, 19, 1783-1792. DOI:10.1002/1097-0258(20000715)19:13<1783::AID-SIM497>3.0.CO;2-B.
30. Hanczar, B., Hua, J., Sima, C., Weinstein, J., Bittner, M., & Dougherty, E. R. (2010). Small-sample precision of ROC-related estimates. Bioinformatics, 26, 822-830. DOI:10.1093/bioinformatics/btq037.
31. Hanley, J. A., & Hajian-Tilaki, K. O. (1997). Sampling variability of nonparametric estimates of areas under receiver operating characteristic curves: An update. Academic Radiology, 4, 49-58. DOI:10.1016/S1076-6332(97)80161-4.
32. Hanley, J. A., & McNeil, B. J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148, 839-843.
33. Hanson, R. K., & Thornton, D. (1999). Static-99: Improving actuarial risk assessments for sex offenders (User Report 99-02). Ottawa, ON: Department of the Solicitor General of Canada.
34. Hart, S. D., Webster, C. D., & Menzies, R. J. (1993). A note on portraying the accuracy of violence predictions. Law & Human Behavior, 17, 695-700. DOI:10.1007/BF01044690.
35. Heagerty, P. J., & Zheng, Y. (2005). Survival model predictive accuracy and ROC curves. Biometrics, 61, 92-105. DOI:10.1111/j.0006-341X.2005.030814.x.
36. Helmus, L., & Hanson, R. K. (2007). Predictive validity of the Static-99 and Static-2002 for sex offenders on community supervision. Sexual Offender Treatment, 2, 1-14.
37. Higgins, J., Deeks, J., & Altman, D. G. (2008). Special topics in statistics. In Higgins J. & Green S. (Eds.), Cochrane handbook for systematic reviews of interventions 5.0.0. London: Wiley.
38. Hilton, D. J., Reid, C. M., & Paratz, J. (2006). An under-used yet easily understood statistic: The number needed to treat (NNT). Physiotherapy, 92, 240-246.
39. Kraemer, H. C., & Gibbons, R. D. (2009). Where do we go wrong in assessing risk factors, diagnostic and prognostic tests? The problems of two-by-two association. Psychiatric Annals, 39, 711-718. DOI:10.3928/00485713-20090625-05.
40. Kraemer, H. C., & Kupfer, D. J. (2006). Size of treatment effects and their importance to clinical research and practice. Biological Psychiatry, 59, 990-996. DOI:10.1016/j.biopsych.2005.09.014.
41. Kroner, D. G. (2007). Issues in violent risk assessment: Lessons learned and future directions. Journal of Interpersonal Violence, 20, 231-235. DOI:10.1177/0886260504267743.
42. Leisenring, W., Alonzo, T. A., & Pepe, M. S. (2000). Comparison of predictive values of binary medical diagnostic tests for paired designs. Biometrics, 56, 345-351. DOI:10.1111/j.0006-341X.2000.00345.x.
43. Li, J., & Fine, J. P. (2011). Assessing the dependence of sensitivity and specificity on prevalence in meta-analysis. Biostatistics, 4, 1-13. DOI:10.1093/biostatistics/kxr008.
44. Long, J. S. (1997). Regression models for categorical and limited dependent variables. Thousand Oaks, CA: Sage.
45. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
46. Marzban, C. (2004). The ROC curve and the area under it as performance measures. Weather & Forecasting, 19, 1106-1114. DOI:10.1175/825.1.
47. McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12, 153-157.
48. McQuay, H. J., & Moore, A. (1997). Using numerical results from systematic reviews in clinical practice. Annals of Internal Medicine, 126, 712-720.
49. Metz, C. E., Herman, B. A., & Shen, J. H. (1998). Maximum-likelihood estimation of ROC curves from continuously-distributed data. Statistics in Medicine, 17, 1033-1053. DOI:10.1002/(SICI)1097-0258(19980515)17:9<1033::AID-SIM784>3.0.CO;2-Z.
50. Metz, C. E., Herman, B. A., & Roe, C. A. (1998). Statistical comparison of two ROC curve estimates obtained from partially-paired datasets. Medical Decision Making, 18, 110-121. DOI:10.1177/0272989X9801800118.
51. Monahan, D. (1981). The clinical prediction of violent behavior. Rockville, MD: US Department of Health and Human Services.
52. Moons, K. G., & Harrell, F. E. (2003). Sensitivity and specificity should be de-emphasized in diagnostic accuracy studies. Academic Radiology, 10, 670-672.
53. Moskowitz, C. S., & Pepe, M. S. (2004). Quantifying and comparing the predictive accuracy of continuous prognostic factors for binary outcomes. Biostatistics, 5, 113-127.
54. Mossman, D. (1994a). Assessing predictions of violence: Being accurate about accuracy. Journal of Consulting & Clinical Psychology, 62, 783-792. DOI:10.1037/0022-006X.62.4.783.
55. Mossman, D. (1994b). Further comments on portraying the accuracy of violence predictions. Law & Human Behavior, 18, 587-593. DOI:10.1007/BF01499177.
56. Mossman, D. (2013). Evaluating risk assessments using receiver operating characteristic analysis: Rationale, advantages, insights, and limitations. Behavioral Sciences & the Law, 31, 23-39.
57. Neller, D. J., & Frederick, R. I. (2013). Classification accuracy of actuarial risk assessment instruments. Behavioral Sciences & the Law, 31, 141-153.
58. Nunnally, J. C. (1978). Psychometric theory (2nd ed). New York: McGraw-Hill.
59. Obuchowski, N. A. (2005). Estimating and comparing diagnostic tests' accuracy when the gold standard is not binary. Academic Radiology, 12, 1198-1204. DOI:10.1016/j.acrs.2005.05.013.
60. Pampel, F. C. (2000). Logistic regression: A primer. Thousand Oaks, CA: Sage.
61. Pan, X., & Metz, C. E. (1997). The "proper" binormal model: Parametric ROC curve estimation with degenerate data. Academic Radiology, 4, 380-389. DOI:10.1016/S1076-6332(97)80121-3.
62. Pencina, M. J., D'Agostino, R. B., D'Agostino, R. B., & Vasan, R. S. (2008). Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Statistics in Medicine, 27, 157-172.
63. Pepe, M. S., Feng, Z., Huang, Y., Longton, G., Prentice, R., Thompson, I. M., & Zheng, Y. (2008). Integrating the predictiveness of a marker with its performance as a classifier. American Journal of Epidemiology, 167, 362-368. DOI:10.1093/aje/kwm305.
64. Pepe, M. S., Janes, H., Longton, G., Leisenring, W., & Newcomb, P. (2004). Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. American Journal of Epidemiology, 159, 882-890. DOI:10.1093/aje/kwh101.
65. Perkins, N. J., & Schisterman, E. F. (2006). The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. American Journal of Epidemiology, 7, 670-675. DOI:10.1093/aje/kwj063.
66. Pinson, L., & Gray, G. (2003). Number needed to treat: An underused measure of treatment effect. Psychiatric Services, 54, 145.
67. Quinsey, V. L., Harris, G. T., Rice, M. E., & Cormier, C. A. (2006). Violent offenders: Appraising and managing risk (2nd ed). Washington, DC: American Psychological Association.
68. Raghunathan, T. E., Rosenthal, R., & Rubin, D. B. (1996). Comparing correlated but nonoverlapping correlations. Psychological Methods, 1, 178-183. DOI:10.1037/1082-989X.1.2.178.
69. Rice, M. E., & Harris, G. T. (1995). Violent recidivism: Assessing predictive validity. Journal of Consulting & Clinical Psychology, 63, 737-748. DOI:10.1037/0022-006X.63.5.737.
70. Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC, Cohen's d and r. Law & Human Behavior, 29, 615-620. DOI:10.1007/s10979-005-6832-7.
71. Robbins, A. S., Chao, S. Y. & Fonseca, V. P. (2002). What's the relative risk? A method to directly estimate risk ratios in cohort studies of common outcomes. Annals of Epidemiology, 12, 452-454. DOI:10.1016/S1047-2797(01)00278-2.
72. Royston, P., Moons, K. G., Altman, D. G., & Vergouwe, Y. (2009). Prognosis and prognostic research: Developing a prognostic model. BMJ, 338, 1373-1377. DOI:10.1136/bmj.b604.
73. Rugge, T. (2006). Risk assessment of male aboriginal offenders: A 2006 perspective. (User Report 2006-01). Ottawa: Public Safety Canada.
74. SAS Institute Inc. (2012). SAS: Version 12.0. Cary, NC: Author.
75. Shiu, S. Y., & Gatsonis, C. (2008). The predictive receiver operating characteristic curve for the joint assessment of the positive and negative predictive values. Philosophical Transactions of the Royal Society, 366, 2313-2333. DOI:10.1098/rsta.2008.0043.
76. Singh, J. P., Desmarais, S. L., & Van Dorn, R. A. (2013). Measurement of predictive validity in violence risk assessment studies: a second-order systematic review. Behavioral Sciences & the Law, 31, 55-73.
77. Singh, J. P., Grann, M., & Fazel, S. (2011). A comparative study of violence risk assessment tools: A systematic review and metaregression analysis of 68 studies involving 25, 980 participants. Clinical Psychology Review, 499-513. DOI:10.1016/j.cpr.2010.11.
78. Singh, J. P., Grann, M., Lichtenstein, P., Långström, N., & Fazel, S. (2012). A novel approach to determining violence risk in schizophrenia: Developing a stepped strategy in 13, 806 discharged patients. PLoS ONE, 7, e31727. DOI:10.1371/journal.pone.0031727.
79. Sjöstedt, G., & Grann, M. (2002). Risk assessment: What is being predicted by actuarial prediction instruments? International Journal of Forensic Mental Health, 1, 179-183. DOI:10.1080/14999013.2002.10471172.
80. Smith, C. A. B. (1951). A test for heterogeneity of proportions. Annals of Eugenics, 16, 16-25. DOI:10.1111/j.1469-1809.1951.tb02455.x.
81. SPSS Inc. (2012). SPSS for Windows: Release 21.0. Chicago: Author.
82. StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Author.
83. Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245-251.
84. Stephan, C., Wesseling, S., Schink, T., & Jung, K. (2003). Comparison of eight computer programs for receiver-operating characteristic analysis. Clinical Chemistry, 49, 433-439. DOI:10.1373/49.3.433.
85. Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010). Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology, 21, 128-138. DOI:10.1097/EDE.0b013e3181c30fb2.
86. Szmukler, G. (2012). Risk assessment for suicide and violence is of extremely limited value in general psychiatric practice. Australian & New Zealand Journal of Psychiatry, 46, 173-174. DOI:10.1177/0004867411432214.
87. Urbaniok, F., Rinne, T., Held, L., Rossegger, A., & Endrass, J. (2008). Forensische Risikokalkulationen: Beurteilung der Anwendbarkeit und Validitat verschiedener Verfahren. [Forensic risk assessment: Assessing the applicability and validity of different methods]. Fortschritte der Neurologie-Psychiatrie, 76, 470-477. DOI:10.1055/s-2008-1038228.
88. Vickers, A. J., Cronin, A. M., & Begg, C. B. (2011). One statistical test is sufficient for assessing new predictive markers. BMC Medical Research Methodology, 11, 1-7. DOI:10.1186/1471-2288-11-13.
89. Vrieze, S. I., & Grove, W. M. (2008). Predicting sex offender recidivism. I. Correcting for item overselection and accuracy overestimation in scale development. II. Sampling error-induced attenuation of predictive validity over base rate information. Law & Human Behavior, 32, 266-278. DOI:10.1007/s10979-007-9092-x.
90. Wang, W., Davis, C. S., & Soong, S. J. (2006). Comparison of predictive values of two diagnostic tests from the same sample of subjects using weighted least squares. Statistics in Medicine, 25, 2215-2229. DOI:10.1002/sim.2332.
91. Ware, J. H. (2006). The limitations of risk factors as prognostic tools. New England Journal of Medicine, 355, 2615-2617. DOI:10.1056/NEJMp068249.
92. Webster, C. D., Douglas, K. S., Eaves, D., & Hart, S. D. (1997). HCR- 20: Assessing risk for violence. Version 2. Burnaby, BC: Simon Fraser University, Mental Health, Law, and Policy Institute.
93. Youden, D. (1950). Index for rating diagnostic tests. Cancer, 3, 32-35. DOI:10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.