Quantifying the accuracy of forensic examiners in the absence of a "gold standard"

Law and Human Behavior

Volumn 34, Issue 5, 2010, Pages 402-417

  Mossman, Douglas ^a,b,c   Bowen, Michael D ^c   Vanness, David J ^d,e   Bienenfeld, David ^c   Correll, Terry ^c   Kay, Jerald ^c   Klykylo, William M ^c   Lehrer, Douglas S ^c  

^a UNIVERSITY OF CINCINNATI   (United States)

^b UNIVERSITY OF CINCINNATI COLLEGE OF MEDICINE   (United States)

^c WRIGHT STATE UNIVERSITY   (United States)

^d UNIVERSITY OF WISCONSIN SCHOOL OF MEDICINE AND PUBLIC HEALTH   (United States)

^e UNITED BIOSOURCE CORPORATION   (United States)

Author keywords

Adjudicative competence; Bayesian; Competence to stand trial; Diagnostic accuracy; Gold standard; Maximum likelihood; ROC analysis

Indexed keywords

ADULT; ARTICLE; BAYES THEOREM; DIAGNOSTIC ERROR; FEMALE; FORENSIC PSYCHIATRY; HUMAN; LEGAL ASPECT; MALE; MENTAL CAPACITY; MIDDLE AGED; PSYCHOLOGICAL ASPECT; RECEIVER OPERATING CHARACTERISTIC; STANDARD; UNITED STATES;

ADULT; BAYES THEOREM; DIAGNOSTIC ERRORS; FEMALE; FORENSIC PSYCHIATRY; HUMANS; MALE; MENTAL COMPETENCY; MIDDLE AGED; ROC CURVE; UNITED STATES;

EID: 77957567697     PISSN: 01477307     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10979-009-9197-5     Document Type: Article

References (72)

1. H Akaike 1974 A new look at the statistical model identification IEEE Transactions on Automatic Control 19 716 723 10.1109/TAC.1974.1100705
2. AA Akinkunmi 2002 The MacArthur Competence Assessment Tool Fitness to Plead: A preliminary evaluation of a research instrument for assessing fitness to plead in England and Wales Journal of the American Academy of Psychiatry and the Law 30 476 482 12539898
3. PS Albert 2007 Random effects modeling approaches for estimating ROC curves from repeated ordinal tests without a gold standard Biometrics 63 593 602 10.1111/j.1541-0420.2006.00712.x 17688512
4. American Academy of Psychiatry and the Law. (May 2005). Ethics guidelines for the practice of forensic psychiatry. http://www.aapl.org/ethics.htm. Accessed 19 Sept 2008.
5. G Bennett 1985 A guided tour through selected ABA standards relating to incompetence to stand trial: Incompetence to stand trial George Washington Law Review 53 375 413
6. WA Berg JD Blume JB Cormack EB Mendelson D Lehrer M Böhm- Vélez ED Pisano RA Jong WP Evans MJ Morton MC Mahoney LH Larsen RG Barr DM Farria HS Marques K Boparai ACRIN 6666 Investigators 2008 Combined screening with ultrasound and mammography vs mammography alone in women at elevated risk of breast cancer Journal of the American Medical Association 299 2151 2163 10.1001/jama.299.18.2151 18477782
7. MT Boccaccini D Turner DC Murrie 2008 Do some evaluators report consistently higher or lower psychopathy scores than others? Findings from a statewide sample of sexually violent predator evaluations Psychology, Public Policy, and Law 14 262 283 10.1037/a0014523
8. RJ Bonnie 1990 The competence of criminal defendants with mental retardation to participate in their own defense Journal of Criminal Law and Criminology 81 419 446 10.2307/1143845
9. SP Brooks A Gelman 1998 Alternative methods for monitoring convergence of iterative simulations Journal of Computational and Graphical Statistics 7 434 455 10.2307/1390675
10. A Buchanan 2006 Competency to stand trial and the seriousness of the charge Journal of the American Academy of Psychiatry and the Law 34 458 465 17185474
11. DM Cain AS Detsky 2008 Everyone's a little bit biased (even physicians) Journal of the American Medical Association 299 2893 2895 10.1001/jama.299.24. 2893 18577735
12. DM Cain G Loewenstein DA Moore 2005 The dirt on coming clean: Perverse effects of disclosing conflicts of interest Journal of Legal Studies 34 1 25 10.1086/426699
13. Carlin, B. P., & Louis, T. A. (2000). Bayes and empirical Bayes methods for data analysis (2nd ed.). London: Chapman & Hall.
14. YK Choi WO Johnson MT Collins IA Gardner 2006 Bayesian inferences for receiver operating characteristic curves in the absence of a gold standard Journal of Agricultural, Biological, and Environmental Statistics 11 210 229 10.1198/108571106X110883
15. Committee on the Revision of the Specialty Guidelines for Forensic Psychology. (11 January 2006). Specialty guidelines for forensic psychology, second official draft. http://www.ap-ls.org/links/. Accessed 19 Sept 2008.
16. VG Cooper PA Zapf 2003 Predictor variables in competency to stand trial decisions Law and Human Behavior 27 423 436 10.1023/A:1024089117535 12916229
17. LJ Cronbach PE Meehl 1955 Construct validity in psychological tests Psychological Bulletin 52 281 302 10.1037/h0040957 13245896
18. Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993).
19. RM Dawes D Faust PE Meehl 1989 Clinical versus actuarial judgment Science 243 1668 1674 10.1126/science.2648573 2648573
20. KS Douglas JR Ogloff TL Nicholls I Grant 1999 Assessing risk for violence among psychiatric patients: The HCR-20 violence risk assessment scheme and the Psychopathy Checklist: Screening Version Journal of Consulting and Clinical Psychology 67 917 930 10.1037/0022-006X.67.6.917 10596513
21. Dusky v. United States, 362 U.S. 402 (1960).
22. Faigman, D. L., Saks, M. J., Sanders, J., & Cheng, E. K. (2008). Modern scientific evidence: Standards, statistics, and research methods, student ed. Eagan, MN: Thomson West.
23. SV Faraone MT Tsuang 1994 Measuring diagnostic accuracy in the absence of a "gold standard" American Journal of Psychiatry 151 650 657 8166304
24. W Gardner CW Lidz EP Mulvey EC Shaw 1996 Clinical versus actuarial predictions of violence of patients with mental illnesses Journal of Consulting and Clinical Psychology 64 602 609 10.1037/0022-006X.64.3.602 8698955
25. AE Gelfand AFM Smith 1990 Sampling-based approaches to calculating marginal densities Journal of the American Statistical Association 85 389 409
26. S Geman D Geman 1984 Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images IEEE Transactions on Pattern Analysis and Machine Intelligence 6 721 741 10.1109/TPAMI.1984.4767596
27. SL Golding R Roesch J Schreiber 1984 Assessment and conceptualization of competency to stand trial: Preliminary data on the Interdisciplinary Fitness Interview Law and Human Behavior 8 321 334 10.1007/BF01044699
28. Grisso, T. (2003). Legally relevant assessments for legal competencies. In T. Grisso (Ed.), Evaluating competencies: Forensic assessments, instruments (2nd ed., pp. 21-40). New York: Kluwer Academic/Plenum Publishers.
29. Gutheil, T. G. (2004). The expert witness. In R. I. Simon & L. H. Gold (Eds.), The American Psychiatric Publishing textbook of forensic psychiatry (pp. 75-89). Arlington, VA: American Psychiatric Publishing.
30. Hagen, M. A. (1997). Whores of the court: The fraud of psychiatric testimony and the rape of American justice. New York: ReganBooks.
31. JA Hanley BJ McNeil 1982 The meaning and use of the area under the receiver operating characteristic (ROC) curve Radiology 143 29 36 7063747
32. GT Harris ME Rice CA Cormier 2002 Prospective replication of the Violence Risk Appraisal Guide in predicting violent recidivism among forensic patients Law and Human Behavior 26 377 394 10.1023/A:1016347320889 12182529
33. RM Henkelman I Kay MJ Bronskill 1990 Receiver operator characteristic (ROC) analysis without truth Medical Decision Making 10 24 29 10.1177/0272989X9001000105 2325524
34. Jackson v. Indiana, 406 U.S. 715 (1972).
35. MS Jacobs NL Ryba PA Zapf 2008 Competence-related abilities and psychiatric symptoms: An analysis of the underlying structure and correlates of the MacCAT-CA and the BPRS Law and Human Behavior 32 64 77 10.1007/s10979-007- 9086-8 17546483
36. SYH Kim PS Appelbaum J Swan TS Stroup JP McEvoy DC Goff DV Jeste JS Lamberti A Leibovici ED Caine 2007 Determining when impairment constitutes incapacity for informed consent in schizophrenia research British Journal of Psychiatry 191 38 43 10.1192/bjp.bp.106.033324 17602123
37. Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999).
38. CD Lehman C Gatsonis CK Kuhl RE Hendrick ED Pisano L Hanna S Peacock SF Smazal DD Maki TB Julian ER DePeri DA Bluemke MD Schnall ACRIN Trial 6667 Investigators Group 2007 MRI evaluation of the contralateral breast in women with recently diagnosed breast cancer New England Journal of Medicine 356 1295 1303 10.1056/NEJMoa065447 17392300
39. DJ Lunn A Thomas N Best D Spiegelhalter 2000 WinBUGS-A Bayesian modeling framework: Concepts, structure, and extensibility Statistics and Computing 10 325 337 10.1023/A:1008929526011
40. KO McGraw SP Wong 1996 Forming inferences about some intraclass correlations Psychological Methods 1 30 46 10.1037/1082-989X.1.1.30
41. Melton, G. B., Petrila, J., Poythress, N., Slobogin, C., Lyons, P., & Otto, R. K. (2007). Psychological evaluations for the courts: A handbook for mental health professionals and lawyers (3rd ed.). New York: Guilford.
42. N Metropolis A Rosenbluth M Rosenbluth A Teller E Teller 1953 Equations of state calculations by fast computing machines Journal of Chemical Physics 21 1087 1091 10.1063/1.1699114
43. HA Miller 2005 The Miller-Forensic Assessment of Symptoms Test (M-Fast): Test generalizability and utility across race literacy, and clinical opinion Criminal Justice and Behavior 32 591 611 10.1177/0093854805278805
44. J Monahan HJ Steadman PS Appelbaum T Grisso EP Mulvey LH Roth PC Robbins S Banks E Silver 2006 The classification of violence risk Behavioral Science and the Law 24 721 730 10.1002/bsl.725
45. D Mossman 1999 "Hired guns", "whores", and "prostitutes": Case law references to clinicians of ill repute Journal of the American Academy of Psychiatry and the Law 27 414 425 10509941
46. D Mossman 2005 Is prosecution "medically appropriate"? New England Journal on Criminal and Civil Confinement 31 15 80
47. D Mossman 2007 Predicting restorability of incompetent criminal defendants Journal of the American Academy of Psychiatry and the Law 35 34 43 17389343
48. D Mossman 2008 Conceptualizing and characterizing accuracy in assessments of competence to stand trial Journal of the American Academy of Psychiatry and the Law 36 340 351 18802182
49. D Mossman SG Noffsinger P Ash RL Frierson J Gerbasi M Hackett CF Lewis DA Pinals CL Scott KG Sieg BW Wall HV Zonana 2007 AAPL practice guideline for the forensic psychiatric evaluation of competence to stand trial Journal of the American Academy of Psychiatry and the Law 35 Suppl 4 S3 S72
50. D Mossman E Somoza 1991 ROC curves, test accuracy, and the description of diagnostic tests Journal of Neuropsychiatry and Clinical Neurosciences 3 330 333 1821250
51. DC Murrie MT Boccaccini D Turner M Meeks C Woods C Tussey 2009 Rater (dis)agreement on risk assessment measures in sexually violent predator proceedings: Evidence of adversarial allegiance in forensic evaluation? Psychology, Public Policy, and Law 15 19 53 10.1037/a0014897
52. DC Murrie M Boccaccini PA Zapf JI Warren CE Henderson 2008 Clinician variation in findings of competence to stand trial Psychology, Public Policy, and Law 14 177 193 10.1037/a0013578
53. NA Obuchowski 2003 Receiver operating characteristic curves and their use in radiology Radiology 229 3 8 10.1148/radiol.2291010898 14519861
54. Parry, J., & Drogin, E. Y. (2007). Mental disability law, evidence and testimony: A comprehensive reference manual for lawyers, judges, and mental disability professionals. Washington, DC: American Bar Association.
55. Pate v. Robinson, 383 U.S. 375 (1966).
56. Poythress, N., Monahan, J., Bonnie, R., Otto, R. K., & Hoge, S. K. (2002). Adjudicative competence: The MacArthur studies. New York: Kluwer/Plenum.
57. Y Qu M Tan MH Kutner 1996 Random effects models in latent class analysis for evaluating accuracy of diagnostic tests Biometrics 53 797 810 10.2307/2533043
58. ME Rice GT Harris 1995 Violent recidivism: Assessing predictive validity Journal of Consulting and Clinical Psychology 63 737 748 10.1037/0022-006X.63.5. 737 7593866
59. B Rosenfeld K Ritchie 1998 Competence to stand trial: Clinician reliability and the role of offense severity Journal of Forensic Sciences 43 151 159 9456535
60. J Skeem S Golding N Cohn G Berge 1998 The logic and reliability of expert opinion on competence to stand trial Law and Human Behavior 22 519 547 10.1023/A:1025787429972 9833565
61. GW Small V Kepe LM Ercoli P Siddarth SY Bookheimer KJ Miller H Lavretsky AC Burggren GM Cole HV Vinters PM Thompson SC Huang N Satyamurthy ME Phelps JRN Barrio 2006 PET of brain amyloid and tau in mild cognitive impairment New England Journal of Medicine 355 2652 2663 10.1056/NEJMoa054625 17182990
62. E Somoza D Mossman 1991 ROC curves and the binormal assumption Journal of Neuropsychiatry and Clinical Neurosciences 3 436 439 1821267
63. Spencer, B. D. (2008). When do latent class models overstate accuracy for binary classifiers? With applications to jury accuracy, survey response error, and diagnostic error. Institute for Policy Research, Northwestern University, Working Paper Series WP-08-10.
64. State v. Sullivan, 739 N.E.2d 788 (Ohio 2001).
65. HJ Steadman EP Mulvey J Monahan PC Robbins PS Appelbaum T Grisso LH Roth E Silver 1998 Violence by people discharged from acute psychiatric inpatient facilities and by others in the same neighborhoods Archives of General Psychiatry 55 393 401 10.1001/archpsyc.55.5.393 9596041
66. Swets, J. A. (1995). Signal detection theory and ROC analysis in psychology and diagnostics: Collected papers. Mahwah, NJ: Lawrence Erlbaum Associates.
67. JS Uebersax 1988 Validity inferences from interobserver agreement Psychological Bulletin 104 405 416 10.1037/0033-2909.104.3.405
68. JS Uebersax WM Grove 1990 Latent class analysis of diagnostic agreement Statistics in Medicine 9 559 572 10.1002/sim.4780090509 2190288
69. Weissman, H. N., & DeBow, D. M. (2003). Ethical principles and professional competencies. In I. B. Weiner (Series Ed.) & A. M. Goldstein (Vol. Ed.), Handbook of psychology: Vol. 11. Forensic psychology (pp. 33-53). New York: Wiley.
70. PA Zapf KL Hubbard VG Cooper MC Wheeles KA Ronan 2004 Have the courts abdicated their responsibility for determination of competency to stand trial to clinicians? Journal of Forensic Psychology Practice 4 27 44 10.1300/J158v04n01-02
71. XH Zhou P Castelluccio C Zhou 2005 Nonparametric estimation of ROC curves in the absence of a gold standard Biometrics 61 600 609 10.1111/j.1541-0420. 2005.00324.x 16011710
72. MH Zweig G Campbell 1993 Receiver operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine Clinical Chemistry 39 561 577 8472349