Analyzing the performance of risk assessment instruments: A response to vrieze and grove (2007)

Law and Human Behavior

Volumn 32, Issue 3, 2008, Pages 279-291

  Mossman, Douglas ^a,b  

^a WRIGHT STATE UNIVERSITY   (United States)

^b UNIVERSITY OF CINCINNATI   (United States)

Author keywords

Actuarial judgment; Cut off selection; Receiver operating characteristic; Recidivism; Risk assessment; ROC area; Sex offender statutes; Test validation

Indexed keywords

ACCURACY; AREA UNDER THE CURVE; HUMAN; LEGAL ASPECT; NOTE; PREDICTIVE VALIDITY; RECIDIVISM; RISK ASSESSMENT; SEXUAL CRIME; STATISTICAL ANALYSIS;

HUMANS; RECURRENCE; RISK ASSESSMENT; ROC CURVE; SEX OFFENSES; UNITED STATES;

EID: 43749114332     PISSN: 01477307     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10979-007-9123-7     Document Type: Note

References (57)

1. American Heritage Dictionary of the English Language: Fourth Edition. (2000). Accessible at http://www.bartleby.com/61/ (site visited July 9, 2007).
2. Babyak, M. A. (2004). What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66, 411-421.
3. Banks, S., Robbins, P., Silver, E., Vesselinov, R., Steadman, H., Monahan, J., Mulvey, E., Appelbaum, P., Grisso, T., & Roth, L. (2004). A multiple models approach to violence risk assessment among people with mental disorder. Criminal Justice and Behavior, 31, 324-340.
4. Bartosh, D. L., Garby, T., Lewis, D., & Gray, S. (2003). Differences in the predictive validity of actuarial risk assessments in relation to sex offender type. International Journal of Offender Therapy and Comparative Criminology, 47, 422-438.
5. Brown, L. D., Cai, T., & DasGupta, A. (2001). Interval estimation for a binomial proportion (with discussion). Statistical Science, 16, 101-133.
6. Commonwealth v. Boucher, 438 Mass. 274, 780 N.E.2d 47 (2002).
7. Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1674.
8. Doren, D. M., & Dow, E. A. (2003). What "shrinkage" of the MnSOST-R? A response to Wollert (2002b). Journal of Threat Assessment, 2, 49-64.
9. Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. London: Chapman & Hall.
10. Epperson, D. L., Kaul, J. D., Huot, S., Goldman, R., & Alexander, W. (December 2003). Minnesota Sex Offender Screening Tool-Revised (MnSOST-R) technical paper: Development, validation, and recommended risk level cut scores. Minnesota Department of Corrections. Available at http://www.psychology. iastate.edu/∼dle/mnsost_download.htm (accessed June 4, 2007).
11. Fitch, L. W. (2007). State hospitals and the future of forensic services. Presentation at the NASMHPD Research Institute, May 7, 2007, viewable at http://www.nasmhpd.org/HospSummit07.cfm (site visited July 9, 2007).
12. Gunderson, D. (April 19, 2004). Critics charge sex offender screening tool doesn't work. Minnesota Public Radio broadcast, text available at http://news.minnesota.publicradio.org/features/2004/04/19_gundersond_mnsostr/ (last accessed June 12, 2007).
13. Hanley, J. A. (1988). The robustness of the 'binormal' assumptions used in fitting ROC curves. Medical Decision Making, 8, 197-203.
14. Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29-36.
15. Hanson, R. K., & Morton-Bourgon, K. E. (2004). Predictors of sexual recidivism: An updated meta-analysis. (User report 2004-02.). Ottawa: Public Safety and Emergency Preparedness Canada.
16. Hanson, R. K., & Morton-Bourgon, K. E. (2007). The accuracy of recidivism risk assessments for sexual offenders: A meta-analysis (2007-01). Ottawa: Public Safety Canada, available at http://www.publicsafety.gc.ca/res/ cor/rep/_fl/crp2007-01-en.pdf (accessed June 15, 2007).
17. Hanson, R. K., & Thornton, D. (2000). Improving risk assessments for sex offenders: A comparison of three actuarial scales. Law and Human Behavior, 24, 119-136.
18. Harrell, F. E., Lee, K. L., & Mark, D. B. (1996). Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine, 15, 361-387.
19. Harris, G. T., & Rice, M. E. (in press). Characterizing the value of actuarial violence risk assessment. Criminal Justice and Behavior.
20. Hart, S. D., Michie, C., & Cooke, D. J. (2007). Precision of actuarial risk assessment instruments: Evaluating the 'margins of error' of group v. individual predictions of violence. British Journal of Psychiatry, 49, s60-s65.
21. Human Rights Watch. (September 2007). No easy answers: Sex offender laws in the US. Available at http://hrw.org/reports/2007/us0907/ (site visited October 11, 2007).
22. In re Detention of Brooks, 36 P.3d 1034 (Wash. 2001).
23. In the Matter of the Commitment of W. Z., 773 A.2d 97 (N. J. Super. Ct. App. Div. 2001).
24. Janus, E. S. (2004). Sexually violent predator laws: Psychiatry in service to a morally dubious enterprise. Lancet, 364, s50-s51.
25. Janus, E. S., & Prentky, R. A. (2003). Forensic use of actuarial risk assessment with sex offenders: Accuracy, admissibility and accountability. American Criminal Law Review, 40, 1443-1499.
26. Langton, C. M., Barbaree, H. E., Seto, M. C., Peacock, E. J., Harkins, L., & Hansen, K. T. (2007). Actuarial assessment of risk for reoffense among adult sex offenders: Evaluating the predictive accuracy of the Static-2002 and five other instruments. Criminal Justice and Behavior, 34, 37-59.
27. Levenson, J. S., & D'Amora, D. A. (2007). Social policies designed to prevent sexual violence: The emperor's new clothes? Criminal Justice Policy Review, 18, 168-199.
28. Meehl, P. E., & Rosen, A. (1955). Antecedent probability and the efficiency of psychometric signs, patterns or cutting scores. Psychological Bulletin, 52, 194-216.
29. Merriam-Webster Online Dictionary. (2007). Accessible at http://www.m-w.com/ (site visited July 9, 2007).
30. Miconnet, N., Cornu, M., Beaufort, A., Rosso, L., & Denis, J.-B. (2005). Uncertainty distribution associated with estimating a proportion in microbial risk assessment. Risk Analysis, 25, 39-48.
31. Miller, H. A., Amenta, A., & Conroy, M. (2005). Sexually violent predator evaluations: Empirical evidence, strategies for professionals, and research directions. Law and Human Behavior, 29, 29-54.
32. Monahan, J. (2006). A jurisprudence of risk assessment: Forecasting harm among prisoners, predators, and patients. Virginia Law Review, 92, 391-435.
33. Monahan, J., & Silver, E. (2003). Judicial decision thresholds for violence risk management. International Journal of Forensic Mental Health, 2, 1-6.
34. Monahan, J., & Steadman, H. J. (1996). Violent storms and violent people: How meteorology can inform risk communication in mental health law. American Psychologist, 51, 931-938.
35. Mossman, D. (1994a). Assessing predictions of violence: Being accurate about accuracy. Journal of Consulting and Clinical Psychology, 62, 783-792.
36. Mossman, D. (1994b). Further comments on portraying the accuracy of violence predictions. Law and Human Behavior, 18, 587-593.
37. Mossman, D. (1995). Dangerousness decisions: An essay on the mathematics of clinical violence predictions and involuntary hospitalization. University of Chicago Law School Roundtable, 2, 95-138.
38. Mossman, D. (1999). Plotting ROC curves: Giving us fits. Medical Decision Making, 19, 214-215.
39. Mossman, D. (2000). Commentary: Assessing the risk of violence - are "accurate" predictions useful? Journal of the American Academy of Psychiatry and the Law, 28, 272-281.
40. Mossman, D. (2006a). Another look at interpreting risk categories. Sexual Abuse: A Journal of Research and Treatment, 18, 41-63.
41. Mossman, D. (2006b). Critique of pure risk assessment, or Kant meets Tarasoff. University of Cincinnati Law Review, 75, 523-609.
42. Mossman, D., & Hart, K. J. (1993). How bad is civil commitment? A study of attitudes toward violence and involuntary hospitalization. Bulletin of the American Academy of Psychiatry and the Law, 21, 181-194.
43. Mossman, D., & Somoza, E. (1992). Balancing risks and benefits: Another approach to optimizing diagnostic tests. Journal of Neuropsychiatry and Clinical Neurosciences, 4, 331-335.
44. National Climatic Data Center. (2006). Comparative Climatic Data - 2006. Available at http://www.ncdc.noaa.gov/oa/ncdc.html (site visited July 13, 2007).
45. People vs. Ghilotti, 44 P.3d 949 (Cal. 2002).
46. Prentky, R. A., Janus, E., Barbaree, H., Schwartz, B. K., & Kafka, M. P. (2006). Sexually violent predators in the courtroom: Science on trial. Psychology, Public Policy, and Law, 12, 357-393.
47. Quinsey, V. L., Harris, G. T., Rice, M. E., & Cormier, C. A. (2006). Violent offenders: Appraising and managing risk (2nd ed.). Washington, DC: American Psychological Association.
48. Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC, Cohen's d and r. Law and Human Behavior, 29, 615-620.
49. Somoza, E., & Mossman, D. (1991). ROC curves and the binormal assumption. Journal of Neuropsychiatry and Clinical Neurosciences, 3, 436-439.
50. Somoza, E., & Somoza, J. R. (1993). A neural-network approach to predicting admission decisions in a psychiatric emergency room. Medical Decision Making, 13, 273-280.
51. Sreenivasan, S., Weinberger, L. E., & Garrick, T. (2003). Expert testimony in sexually violent predator commitments: Conceptualizing legal standards of "mental disorder" and "likely to reoffend". Journal of the American Academy of Psychiatry and the Law, 31, 471-485.
52. State v. Curiel, 227 Wis. 2d 389, 597 N.W.2d 712 (1999).
53. Swets, J. A. (1986). Indices of discrimination and diagnostic accuracy: Their ROCs and implied models. Psychological Bulletin, 99, 100-117.
54. Vrieze, S. I., & Grove, W. M. (published online 3 July 2007). Predicting sex offender recidivism. I. Correcting for item overselection and accuracy overestimation in scale development. II. Sampling error-induced attenuation of predictive validity over base rate information. Law and Human Behavior, 10.1007/s10979-007-9092-x.
55. Wollert, R. (2003). The importance of cross-validation in actuarial test construction: Shrinkage in the risk estimates for the Minnesota Sex Offender Screening Tool-Revised. Journal of Threat Assessment, 2, 87-102.
56. Wollert, R. (2006). Low base rates limit expert certainty when current actuarials are used to identify sexually violent predators: An application of Bayes's Theorem. Psychology, Public Policy & Law, 12, 56-85.
57. World Health Organization, Department of Essential Health Technologies. (2004). HIV assays: Operational characteristics (Phase 1), Report 14, Simple/Rapid Tests.