Evaluating Risk Assessments Using Receiver Operating Characteristic Analysis: Rationale, Advantages, Insights, and Limitations

Behavioral Sciences and the Law

Volumn 31, Issue 1, 2013, Pages 23-39

  Mossman, Douglas ^a  

^a UC Department of Psychiatry   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; EVALUATION; FORENSIC PSYCHIATRY; HUMAN; RECEIVER OPERATING CHARACTERISTIC; RESEARCH; RISK ASSESSMENT; STANDARD; UNITED STATES; VIOLENCE;

ALGORITHMS; EVALUATION STUDIES AS TOPIC; FORENSIC PSYCHIATRY; HUMANS; RESEARCH; RISK ASSESSMENT; ROC CURVE; UNITED STATES; VIOLENCE;

EID: 84874305617     PISSN: 07353936     EISSN: 10990798     Source Type: Journal    
DOI: 10.1002/bsl.2050     Document Type: Article

References (47)

1. Anderson, D., & Hanson, R. K. (2010). Static-99: An actuarial tool to assess risk of sexual and violent recidivism among sexual offenders. In Otto R. K., & Douglas K. S. (Eds.), Handbook of Violence Risk Assessment (pp. 251-267). New York: Routledge/Taylor & Francis Group.
2. Barzman, D., Brackenbury, L., Sonnier, L., Schnell, B., Cassedy, A., Salisbury, Sorter M; & Mossman, D. (2011). Brief Rating of Aggression by Children and Adolescents (BRACHA): Development of a tool to assess risk of inpatients' aggressive behavior. The Journal of the American Academy of Psychiatry and the Law, 39, 170-179.
3. Baxtrom v. Herold. (1966). 383 U.S. 107.
4. Cohen, J. (1988). Statistical power analysis for the behavioral sciences 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates.
5. Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1674.
6. Dolan, M., & Doyle, M. (2000). Violence risk prediction. Clinical and actuarial measures and the role of the Psychopathy Checklist. The British Journal of Psychiatry, 177, 303-311.
7. Dorfman, D. D., & Alf, E., Jr. (1969). Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals-rating method data, Journal of Mathematical Psychology, 6, 487-496.
8. Douglas, K. S., & Reeves, K. A. (2010). Historical-Clinical-Risk Management-20 (HCR-20) violence risk assessment scheme: Rationale, application, and empirical overview. In Otto R. K., & Douglas K. S. (Eds.), Handbook of violence risk assessment (pp. 147-185). New York, NY: Routledge/Taylor & Francis.
9. Douglas, K. S., Ogloff, J. R. P., Nicholls, T. L., & Grant, I. (1999). Assessing risk for violence among psychiatric patients: The HCR-20 risk assessment scheme and the Psychopathy Checklist: Screening Version. Journal of Consulting and Clinical Psychology, 67, 917-930.
10. Ennis, B. J., & Litwack, T. R. (1974). Psychiatry and the presumption of expertise: Flipping coins in the courtroom. California Law Review, 62, 693-752.
11. Erdman, H. P., Greist, J. H., Gustafson, D. H., Taves, J. E., & Klein, M. H. (1987). Suicide risk prediction by computer interview: a prospective study. The Journal of Clinical Psychiatry, 48, 464-467.
12. Faust, D., & Ahern, D. C. (2011). Clinical judgment and prediction. In Faust D. (Ed.), Coping with psychiatric and psychological testimony 6th ed. (pp. 147-208). New York: Oxford University Press.
13. Gardner, W., Lidz, C. W., Mulvey, E. P., & Shaw, E. C. (1996). A comparison of actuarial methods for identifying repetitively violent patients with mental illnesses. Law and Human Behavior, 20, 35-48.
14. Goodenough, D. J., Rossmann, K., & Lusted, L. B. (1974). Radiographic applications of receiver operating characteristic (ROC) analysis. Radiology, 110, 89-95.
15. Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical-statistical controversy. Psychology, Public Policy, and Law, 2, 293-323.
16. Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19-30.
17. Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29-36.
18. Hanson, R. K., & Morton-Bourgon, K. E. (2009). The accuracy of recidivism risk assessments for sexual offenders: A meta-analysis of 118 prediction studies. Psychological Assessment, 21, 1-21.
19. Hart, S. D., & Logan, C. (2011). Formulation of violence risk using evidence- based assessments: the structured professional judgment approach. In Sturmey P., & McMurran M. (Eds.), Forensic case formulation (pp. 83-106). Chichester, West Sussex, UK: John Wiley & Sons.
20. Kans. Stat. Annot. § 59-29a02. (2009).
21. Lunn, D., Spiegelhalter, D., Thomas, A., & Best, N. (2009). The BUGS project: Evolution, critique and future directions. Statistics in Medicine, 28, 3049-3067.
22. McGraw, K. O., & Wong, S. P. (1992). A common language effect size statistic. Psychological Bulletin, 111, 361-365.
23. McNeil, B. J., Varady, P. D., Burrows, B. A., & Adelstein, S. J. (1975). Measures of clinical efficacy-cost-effectiveness calculations in the diagnosis and treatment of hypertensive renovascular disease. The New England Journal of Medicine, 293, 216-221.
24. Metz, C. E., & Kronman, H. B. (1980). Statistical significance tests for binormal ROC curves. Journal of Mathematical Psychology 22, 218-243.
25. Metz, C. E., Herman, B. A., & Shen, J. H. (1998). Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Statistics in Medicine, 17, 1033-53.
26. Monahan, J. (1981). The Clinical Prediction of Violent Behavior. Rockville, MD: National Institute of Mental Health.
27. Monahan, J. (1984). The prediction of violent behavior: toward a second generation of theory and policy. The American Journal of Psychiatry, 141, 1-11.
28. Monahan, J. (1997). Clinical and actuarial predictions of violence. In Faigman D., Kaye D., Saks M., & Sanders J. (Eds.), Modern scientific evidence: The law of science and expert testimony (Vol. 1 pp. 300-318). St. Paul, MN: West Group.
29. Monahan, J. (2002). Clinical and actuarial predictions of violence. B. Scientific status. In Faigman D. L., Kaye D. H., Saks M. J., & Sanders J. (Eds.), Science and the law: Social and behavioral sciences issues (pp. 90-112). St. Paul, MN: West Group.
30. Morse, S. J. (1978). Crazy behavior, morals, and science: An analysis of mental health law. California Law Review, 51, 527-654.
31. Mossman, D. (1994a) Assessing predictions of violence: being accurate about accuracy. Journal of Consulting and Clinical Psychology, 62, 783-792.
32. Mossman, D. (1994b). Further comments on portraying the accuracy of violence prediction. Law and Human Behavior, 18, 587-593.
33. Mossman, D. (2000). Assessing the risk of violence - are "accurate" predictions useful? The Journal of the American Academy of Psychiatry and the Law, 28, 272-281.
34. Mossman, D. (2006). Critique of pure risk assessment or, Kant meets Tarasoff. University of Cincinnati Law Review, 75, 523-609.
35. Mossman, D., & Somoza, E. (1989). Maximizing diagnostic information from the dexamethasone suppression test: an approach to criterion selection using receiver operating characteristic analysis. Archives of General Psychiatry, 44, 653-660.
36. Murphy, J. M., Berwick, D. M., Weinstein, M. C., Borus, J. F., Budman, S. H., & Klerman, G. L. (1987). Performance of screening and diagnostic tests. Application of receiver operating characteristic analysis. Archives of General Psychiatry, 44, 550-555.
37. Rice, M. E., & Harris, G. T. (1995). Violent recidivism: Assessing predictive validity. Journal of Consulting and Clinical Psychology, 63, 737-748.
38. Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen's d, and r. Law and Human Behavior, 29, 615-620.
39. Ruscio, J. (2008). A probability-based measure of effect size: Robustness to base rates and other factors. Psychological Methods, 13, 19-30.
40. Simpson, A. J., & Fitter, M. J. (1973). What is the best index of detectability? Psychology Bulletin, 80, 481-488.
41. Singh, J. P., Grann, M., & Fazel, S. (2011). A comparative study of violence risk assessment tools: a systematic review and metaregression analysis of 68 studies involving 25, 980 participants. Clinical Psychology Review, 31, 499-513.
42. Somoza, E., & Mossman, D. (1991). ROC curves and the binormal assumption. The Journal of Neuropsychiatry and Clinical Neurosciences, 3, 436-439.
43. Sreenivasan, S., Weinberger, L. E., Frances, A., & Cusworth-Walker, S. (2010). Alice in actuarial land: through the looking glass of changing Static-99 norms. The Journal of the American Academy of Psychiatry and the Law, 38, 400-406.
44. Steadman, H., Mulvey, E., Monahan, J., Robbins, P., Appelbaum, P., Grisso, T., ... Silver, E. (1998). Violence by people discharged from acute psychiatric inpatient facilities and by others in the same neighborhoods. Archives of General Psychiatry, 55, 393-401.
45. Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Sciences in the Public Interest (Supplement to Psychological Science), 1, 1-26.
46. Yang, M., Wong, S. C. P., & Coid, J. (2010). The efficacy of violence prediction: A meta-analytic comparison of nine risk assessment tools. Psychological Bulletin, 136, 740-767.
47. Zweig, M. H., & Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39, 561-577.