A comprehensive meta-reanalysis of the robustness of the experience-accuracy effect in clinical judgment

Journal of Counseling Psychology

Volumn 62, Issue 3, 2015, Pages 360-378

  Spengler, Paul M ^a   Pilipis, Lois A ^a,b  

^a BALL STATE UNIVERSITY   (United States)

^b Pilipis Behavioral Group   (United States)

Author keywords

Clinical judgment; Decision making; Judgment accuracy; Meta analysis

Indexed keywords

CLINICAL COMPETENCE; DECISION MAKING; FEMALE; HEALTH STUDENT; HUMAN; LIFE EVENT; MALE; META ANALYSIS; PROCEDURES; PSYCHOLOGY; STANDARDS;

CLINICAL COMPETENCE; DECISION MAKING; FEMALE; HUMANS; JUDGMENT; LIFE CHANGE EVENTS; MALE; PSYCHOLOGY; STUDENTS, HEALTH OCCUPATIONS;

EID: 84937073120     PISSN: 00220167     EISSN: None     Source Type: Journal    
DOI: 10.1037/cou0000065     Document Type: Article

References (196)

1. Ægisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., . . . Rush, J. D. (2006). The meta-analysis of clinical judgment project: Fifty-six years of accumulated research on clinical versus statistical prediction. The Counseling Psychologist, 34, 341-382. http://dx.doi.org/10.1177/0011000005285875
2. Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R., & Dalton, C. M. (2011). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14, 306-331. http://dx.doi.org/10.1177/1094428110375720
3. Akehurst, L., Bull, R., Vrij, A., & Köhnken, G. (2004). The effects of training professional groups and lay persons to use criteria-based content analysis to detect deception. Applied Cognitive Psychology, 18, 877-891. http://dx.doi.org/10.1002/acp.1057
4. Alford, J. D., & Locke, B. J. (1984). Clinical responses to psychopathology of mentally retarded persons. American Journal of Mental Deficiency, 89, 195-197.
5. Altmaier, E. M., & Hansen, J. C. (Eds.). (2012). The Oxford handbook of counseling psychology. New York, NY: New York: Oxford University Press.
6. American Educational Research Association. (2014). Standards for educational and psychological testing. Washington, DC: Author.
7. Amira, S., Abramowitz, S. I., & Gomes-Schwartz, B. (1977). Sociallycharged pupil and psychologist effects on psychoeducational decisions. The Journal of Special Education, 11, 433-440. http://dx.doi.org/10.1177/002246697701100405
8. Andersen, S. M., & Hawthorn, B. H. (1989). The recognition, diagnosis, and treatment of mental disorders by primary care physicians. Medical Care, 27, 869-886.
9. APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61, 271-285. http://dx.doi.org/10.1037/0003-066X.61.4.271
10. Arkell, R. N. (1976). Naïve prediction of pathology from human figure drawings. Journal of School Psychology, 14, 114-117. http://dx.doi.org/10.1016/0022-4405(76)90046-7
11. Barkin, S. L. (1991). Judgments of professional psychologists about patients with the acquired immunodeficiency disease (AIDS) virus and terminal cancer (Unpublished doctoral dissertation). Hofstra University, Hempstead, NY.
12. Batson, C. D., & Marz, B. (1979). Dispositional bias in trained therapists' diagnoses: Does it exist?. Journal of Applied Social Psychology, 9, 476-489. http://dx.doi.org/10.1111/j.1559-1816.1979.tb02720.x
13. Beck, K. A., & Ogloff, J. R. (1995). Child abuse reporting in British Columbia: Psychologists' knowledge of and compliance with the reporting law. Professional Psychology, Research and Practice, 26, 245-251. http://dx.doi.org/10.1037/0735-7028.26.3.245
14. Becker, B. J. (2005). FailsafeN or file-drawer number. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 111-126). West Sussex, UK: Wiley.
15. Benlifer, V. E., & Kiesler, S. B. (1972). Psychotherapists' perceptions of adjustment and attraction toward children described as in therapy. Journal of Experimental Research in Personality, 6, 169-177.
16. Berman, G., & Berman, D. (1984). In the eyes of the beholder: Effects of psychiatric labels and training on clinical judgments. Academic Psychology Bulletin, 6, 36-42.
17. Berven, N. L. (1985). Reliability and validity of standardized case management simulations. Journal of Counseling Psychology, 32, 397-409. http://dx.doi.org/10.1037/0022-0167.32.3.397
18. Blashfield, R., Sprock, J., Pinkston, K., & Hodgin, J. (1985). Exemplar prototypes of personality disorder diagnoses. Comprehensive Psychiatry, 26, 11-21. http://dx.doi.org/10.1016/0010-440X(85)90045-8
19. Blumetti, A. E. (1972). A test of clinical versus actuarial prediction: A consideration of accuracy and cognitive functioning (Unpublished doctoral dissertation). University of Florida, Gainesville, FL.
20. Boland, K. (2002). Ethical decision-making among hospital social workers (Unpublished doctoral dissertation). Marywood University, Scranton, PA.
21. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2005). Comprehensive meta-analysis (Version 2) [Computer software]. Englewood, NJ: Biostat.
22. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis West Sussex, UK: Wiley. http://dx.doi.org/10.1002/9780470743386
23. Borman, G. D., & Grigg, J. A. (2009). Visual and narrative interpretation. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 497-519). New York, NY: Russell Sage Foundation.
24. Brammer, R. (2002). Effects of experience and training on diagnostic accuracy. Psychological Assessment, 14, 110-113. http://dx.doi.org/10.1037/1040-3590.14.1.110
25. Brenneis, C. B. (1971). Factors affecting diagnostic judgments of manifest dream content in schizophrenia. Psychological Reports, 29, 811-818. http://dx.doi.org/10.2466/pr0.1971.29.3.811
26. Bruhn, A. R., & Reed, M. R. (1975). Simulation of brain damage on the Bender-Gestalt test by college subjects. Journal of Personality Assessment, 39, 244-255. http://dx.doi.org/10.1207/s15327752jpa3903_5
27. Butcher, J. E. (1983). Validation of a standardized simulation for the assessment of competence in mental health counselors (Unpublished doctoral dissertation). The Pennsylvania State University, University Park, PA.
28. Butte, D. G. (1998). Accuracy of nurses assessing patients using American Psychiatric Association criteria (Unpublished doctoral dissertation). University of Utah, Salt Lake City, UT.
29. Caetano, D. F. (1974). Labeling theory and the presumption of mental illness in diagnosis: An experimental design. Journal of Health and Social Behavior, 15, 253-260. http://dx.doi.org/10.2307/2137025
30. Carlin, A. S., & Hewitt, P. L. (1990). The discrimination of patientgenerated and randomly generated MMPIs. Journal of Personality Assessment, 54, 24-29. http://dx.doi.org/10.1207/s15327752jpa5401&2_3
31. Carroll, N., Rosenberg, H., & Funke, S. (1988). Recognition of intoxication by alcohol counselors. Journal of Substance Abuse Treatment, 5, 239-246. http://dx.doi.org/10.1016/0740-5472(88)90046-3
32. Casas, J. M., Brady, S., & Ponterotto, J. G. (1983). Sexual preference biases in counseling: An information processing approach. Journal of Counseling Psychology, 30, 139-145. http://dx.doi.org/10.1037/0022-0167.30.2.139
33. Chandler, M. J. (1970). Self-awareness and its relation to other parameters of the clinical inference process. Journal of Consulting and Clinical Psychology, 35, 258-264. http://dx.doi.org/10.1037/h0029881
34. Cohn, L. D., & Becker, B. J. (2003). How meta-analysis increases statistical power. Psychological Methods, 8, 243-253. http://dx.doi.org/10.1037/1082-989X.8.3.243
35. Cooper, H. (1998). Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage Publications.
36. Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis. New York, NY: Russell Sage Foundation.
37. Cooper, R. P., & Werner, P. D. (1990). Predicting violence in newly admitted inmates: A lens model analysis of staff decision-making. Criminal Justice and Behavior, 17, 431-447. http://dx.doi.org/10.1177/0093854890017004004
38. Cressen, R. (1975). Artistic quality of drawings and judges' evaluations of the DAP. Journal of Personality Assessment, 39, 132-137. http://dx.doi.org/10.1207/s15327752jpa3902_7
39. Cummings, A. L., Hallberg, E. T., Martin, J., Slemon, A., & Hiebert, B. (1990). Implications of counselor conceptualizations for counselor education. Counselor Education and Supervision, 30, 120-134. http://dx.doi.org/10.1002/j.1556-6978.1990.tb01189.x
40. Dawes, R. R. (1994). House of cards: Psychology and psychotherapy built on myth. New York, NY: Free Press.
41. Dawson, V. L., Zeitz, C. M., & Wright, J. C. (1989). Expert-novice differences in person perception: Evidence of experts' sensitivities to the organization of behavior. Social Cognition, 7, 1-30. http://dx.doi.org/10.1521/soco.1989.7.1.1
42. DeHaven, C. L. (1992). Educational diagnostic decision-making practices demonstrated by school psychologists, and a computer-based test interpretation system (Unpublished doctoral dissertation). University of Southern California, Los Angeles, CA.
43. de Mesquita, P. B. (1992). Diagnostic problem solving of school psychologists: Scientific method or guesswork?. Journal of School Psychology, 30, 269-291. http://dx.doi.org/10.1016/0022-4405(92)90011-S
44. di Nardo, P. A. (1975). Social class and diagnostic suggestion as variables in clinical judgment. Journal of Consulting and Clinical Psychology, 43, 363-368. http://dx.doi.org/10.1037/h0076728
45. Ebling, R., & Levenson, R. (2003). Who are the marital experts?. Journal of Marriage and Family, 65, 130-142. http://dx.doi.org/10.1111/j.1741-3737.2003.00130.x
46. Eells, T. D., Lombart, K. G., Kendjelic, E. M., Turner, L. C., & Lucas, C. P. (2005). The quality of psychotherapy case formulations: A comparison of expert, experienced, and novice cognitive-behavioral and psychodynamic therapists. Journal of Consulting and Clinical Psychology, 73, 579-589. http://dx.doi.org/10.1037/0022-006X.73.4.579
47. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634. http://dx.doi.org/10.1136/bmj.315.7109.629
48. Ekman, P., O'Sullivan, M., & Frank, M. G. (1999). A few can catch a liar. Psychological Science, 10, 263-266. http://dx.doi.org/10.1111/1467-9280.00147
49. Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406. http://dx.doi.org/10.1037/0033-295X.100.3.363
50. Evans, C. (1983). "Draw a person . . . a whole person": Drawings from psychiatric patients and well-adjusted adults as judged by six traditional DAP indicators, licensed psychologists, and the general public (Unpublished doctoral dissertation). Temple University, Philadelphia, PA.
51. Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33, 517. http://dx.doi.org/10.1037/0003-066X.33.5.517.a
52. Falvey, J. E., Bray, T. E., & Hebert, D. J. (2005). Case conceptualization and treatment planning: Investigation of problem-solving and clinical judgment. Journal of Mental Health Counseling, 27, 348-372.
53. Faust, D., & Ahern, D. C. (2012). Clinical judgment and prediction. In D. Faust (Ed.), Coping with psychiatric and psychological testimony: Based on the original work by Jay Ziskin (6th ed., pp. 147-208). New York, NY: Oxford University Press.
54. Faust, D., & Faust, K. A. (2012). Experts' experience and diagnostic and predictive accuracy. In D. Faust (Ed.), Coping with psychiatric and psychological testimony: Based on the original work by Jay Ziskin (6th ed., pp. 131-146). New York, NY: Oxford University Press.
55. Faust, D., Guilmette, T. J., Hart, K., Arkes, H. R., Fishburne, F. J., & Davey, L. (1988). Neuropsychologists' training, experience, and judgment accuracy. Archives of Clinical Neuropsychology, 3, 145-163. http://dx.doi.org/10.1093/arclin/3.2.145
56. Finlayson, L. M., & Koocher, G. P. (1991). Professional judgment and child abuse reporting in sexual abuse cases. Professional Psychology: Research and Practice, 22, 464-472. http://dx.doi.org/10.1037/0735-7028.22.6.464
57. Fouad, N. A., Carter, J. A., & Subich, L. M. (Eds.). (2012a). APA handbook of counseling psychology: Vol. 1. Theories, research, and methods. Washington, DC: American Psychological Association.
58. Fouad, N. A., Carter, J. A., & Subich, L. M. (Eds.). (2012b). APA handbook of counseling psychology: Vol. 2. Practice, interventions, and applications. Washington, DC: American Psychological Association.
59. Gambrill, E. (2005). Critical thinking in clinical practice: Improving the quality of judgments and decisions. Hoboken, NJ: Wiley.
60. Garb, H. N. (1998). Studying the clinician: Judgment research and psychological assessment Washington, DC: American Psychological Association. http://dx.doi.org/10.1037/10299-000
61. Garb, H. N. (2006). The conjunction effect and clinical judgment. Journal of Social and Clinical Psychology, 25, 1048-1056. http://dx.doi.org/10.1521/jscp.2006.25.9.1048
62. Garb, H. N., & Boyle, P. A. (2003). The diagnosis of neurological disorders in older adults. Assessment, 10, 129-134. http://dx.doi.org/10.1177/1073191103010002003
63. Garcia, S. K. (1993). Development of a methodology to differentiate between the physiological and psychological basis of panic attacks (Unpublished doctoral dissertation). St. Mary's University, San Antonio, TX.
64. Gaudette, M. D. (1992). Clinical decision making in neuropsychology: Bootstrapping the neuropsychologist utilizing the Brunswik's lens model (Unpublished doctoral dissertation). Indiana University of Pennsylvania, Indiana, PA.
65. Gerbe, N. (2007). School psychologists' knowledge of Asperger's disorder, its differential diagnosis, and treatment recommendations (Unpublished doctoral dissertation). St. John's University, New York, NY.
66. Goldberg, L. R. (1959). The effectiveness of clinicians' judgments; the diagnosis of organic brain damage from the Bender-Gestalt test. Journal of Consulting Psychology, 23, 25-33. http://dx.doi.org/10.1037/h0048736
67. Goldsmith, L., & Schloss, P. J. (1986). Diagnostic overshadowing among school psychologists working with hearing-impaired learners. American Annals of the Deaf, 131, 288-293. http://dx.doi.org/10.1353/aad.2012.0832
68. Goldstein, S. G., Kleinknecht, R. A., & Deysach, R. E. (1973). Effect of experience and amount of information on identification of cerebral impairment. Journal of Consulting and Clinical Psychology, 41, 30-34. http://dx.doi.org/10.1037/h0035614
69. Graham, J. R. (1971). Feedback and accuracy of clinical judgment from the MMPI. Journal of Consulting and Clinical Psychology, 36, 286-291. http://dx.doi.org/10.1037/h0030741
70. Graham, J. R., & Naglieri, J. A. (2013). Handbook of psychology, Vol. 10: Assessment psychology (2nd ed.). Hoboken, NJ: Wiley.
71. Greenhouse, J. B., & Iyengar, S. (2009). Sensitivity analysis and diagnostics. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 417-434). New York, NY: Russell Sage Foundation.
72. Gregory, J. M. (1980). Clinical judgment research tasks and the psychotherapist's ability to understand patients (Unpublished doctoral dissertation). Duke University, Durham, NC.
73. Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19-30. http://dx.doi.org/10.1037/1040-3590.12.1.19
74. Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61, 155-163. http://dx.doi.org/10.1002/jclp.20108
75. Hansen, D. J., Bumby, K. M., Lundquist, L. M., Chandler, R. M., Le, P. T., & Futa, K. T. (1997). The influence of case and professional variables on the identification and reporting of child maltreatment: A case study of licensed psychologists and certified masters social workers. Journal of Family Violence, 12, 313-332. http://dx.doi.org/10.1023/A:1022852921883
76. Harding, T. P. (2007). Clinical decision-making: How prepared are we?. Training and Education in Professional Psychology, 1, 95-104. http://dx.doi.org/10.1037/1931-3918.1.2.95
77. Heaton, R. K., Smith, H. H., Jr., Lehman, R. A. W., & Vogt, A. T. (1978). Prospects for faking believable deficits on neuropsychological testing. Journal of Consulting and Clinical Psychology, 46, 892-900. http://dx.doi.org/10.1037/0022-006X.46.5.892
78. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
79. Hickling, E. J., Blanchard, E. B., Mundy, E., & Galovski, T. (2002). Detection of malingered MVA related posttraumatic stress disorder: An investigation of the ability to detect professional actors by experienced clinicians, psychological tests and psychophysiological assessment. Journal of Forensic Psychology Practice, 2, 33-53. http://dx.doi.org/10.1300/J158v02n01_02
80. Higgins, J. P. T., & Green, S. (Eds.). (2011). Cochrane handbook for systematic reviews of interventions Version 5.1.0 Retrieved from http://www.cochrane-handbook.org
81. Hill, C., Thompson, B. J., & Corbett, M. M. (1992). The impact of therapist ability to perceive displayed and hidden client reaction on immediate outcome in first sessions of brief therapy. Psychotherapy Research, 2, 143-155. http://dx.doi.org/10.1080/10503309212331332914
82. Hill, C. E. (2014). Helping skills: Facilitating exploration, insight, and action (4th ed.). Washington, DC: American Psychological Association. http://dx.doi.org/10.1037/14345-000
83. Hillerbrand, E., & Claiborn, C. D. (1990). Examining reasoning skill differences between expert and novice counselors. Journal of Counseling & Development, 68, 684-691. http://dx.doi.org/10.1002/j.1556-6676.1990.tb01437.x
84. Hillman, J. L., Stricker, G., & Zweig, R. A. (1997). Clinical psychologists' judgments of older adult patients with character pathology: Implications for practice. Professional Psychology: Research and Practice, 28, 179-183. http://dx.doi.org/10.1037/0735-7028.28.2.179
85. Hoaglin, D. C., & Iglewicz, B. (1987). Fine tuning some resistant rules for outlier labeling. Journal of the American Statistical Association, 82, 1147-1149. http://dx.doi.org/10.1080/01621459.1987.10478551
86. Holt, R. R. (1958). Clinical and statistical prediction; a reformulation and some new data. The Journal of Abnormal and Social Psychology, 56, 1-12. http://dx.doi.org/10.1037/h0041045
87. Homant, R. J., & Kennedy, D. B. (1985). The effect of prior experience on expert witnesses' opinions. Criminal Justice Review, 10, 18-21. http://dx.doi.org/10.1177/073401688501000203
88. Horner, T. M., Guyer, M. J., & Kalter, N. M. (1993). Clinical expertise and the assessment of child sexual abuse. Journal of the American Academy of Child & Adolescent Psychiatry, 32, 925-933. http://dx.doi.org/10.1097/00004583-199309000-00006
89. Horowitz, L. M., Inouye, D., & Siegelman, E. Y. (1979). On averaging judges' ratings to increase their correlation with an external criterion. Journal of Consulting and Clinical Psychology, 47, 453-458. http://dx.doi.org/10.1037/0022-006X.47.3.453
90. Howitt, P. S. (1984). Kinetic family drawings and clinical judgment: An evaluation of judges' ability to differentiate between the K-F-D's of abusing, control, and concerned mothers (Unpublished doctoral dissertation). University of Windsor, Ontario, Canada.
91. Huffaker, S. (2008). Factors predicting case formulation proficiency in traumatic brain injury: Experience and knowledge (Unpublished doctoral dissertation). Alliant International University, San Diego, CA.
92. Ioannidis, J. P. A., & Trikalinos, T. A. (2005). Early extreme contradictory estimates may appear in published research: The Proteus phenomenon in molecular genetics research and randomized trials. Journal of Clinical Epidemiology, 58, 543-549. http://dx.doi.org/10.1016/j.jclinepi.2004.10.019
93. Jackson, R. L., Rogers, R., & Shuman, D. W. (2004). The adequacy and accuracy of sexually violent predator evaluations: Contextualized risk assessment in clinical practice. The International Journal of Forensic Mental Health, 3, 115-129. http://dx.doi.org/10.1080/14999013.2004.10471201
94. Jopp, D. (2001). An examination of the diagnostic overshadowing bias (Unpublished doctoral dissertation). University of Illinois, Chicago, IL.
95. Kahneman, D., Slovic, P., & Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases Cambridge, London: New York: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511809477
96. Kellner, L. J. (2001). Are school psychologists knowledgeable about adolescent suicide? (Unpublished doctoral dissertation). Fairleigh Dickinson University, Teaneck, NJ.
97. Kendell, R. E. (1973). Psychiatric diagnoses: A study of how they are made. The British Journal of Psychiatry, 122, 437-445. http://dx.doi.org/10.1192/bjp.122.4.437
98. Kennel, R. G., & Agresti, A. A. (1995). Effects of gender and age on psychologists' reporting of child sexual abuse. Professional Psychology: Research and Practice, 26, 612-615. http://dx.doi.org/10.1037/0735-7028.26.6.612
99. Kim, N. S., & Ahn, W. K. (2002). Clinical psychologists' theory-based representations of mental disorders predict their diagnostic reasoning and memory. Journal of Experimental Psychology: General, 131, 451-476. http://dx.doi.org/10.1037/0096-3445.131.4.451
100. Kitamura, T., Kitamura, F., Ito, A., Okazaki, Y., Okuda, N., Mitsuhashi, T., & Katoh, H. (1999). Image of psychiatric patients' competency to give informed consent to treatment in Japan. II. A case vignette study of competency judgements. International Journal of Law and Psychiatry, 22, 133-142. http://dx.doi.org/10.1016/S0160-2527(98)00038-7
101. Kivlighan, D. M., Jr., & Quigley, S. T. (1991). Dimensions used by experienced and novice group therapists to conceptualize group process. Journal of Counseling Psychology, 38, 415-423. http://dx.doi.org/10.1037/0022-0167.38.4.415
102. Kocsis, R. N. (2003). Criminal psychological profiling: Validities and abilities. International Journal of Offender Therapy and Comparative Criminology, 47, 126-144. http://dx.doi.org/10.1177/0306624X03251092
103. Lacks, P. B., & Newport, K. (1980). A comparison of scoring systems and level of scorer experience on the Bender-Gestalt test. Journal of Personality Assessment, 44, 351-357. http://dx.doi.org/10.1207/s15327752jpa4404_3
104. Lambert, L., & Wertheimer, M. (1988). Is diagnostic ability related to relevant training and experience?. Professional Psychology: Research and Practice, 19, 50-52. http://dx.doi.org/10.1037/0735-7028.19.1.50
105. Lambert, M. J., & Ogles, B. M. (2004). The efficacy and effectiveness of psychotherapy. In M. J. Lambert (Ed.), Bergin and Garfield's handbook of psychotherapy and behavior change (5th ed., pp. 139-193). New York, NY: Wiley.
106. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174. http://dx.doi.org/10.2307/2529310
107. Lefkowitz, M. B. (1973). Statistical and clinical approaches to the identification of couples at risk in marriage. Unpublished doctoral dissertation, University of Florida, Gainesville.
108. Leli, D. A., & Filskov, S. B. (1981). Clinical-actuarial detection and description of brain impairment with the W-B Form 1. Journal of Clinical Psychology, 37, 623-629.
109. Leli, D. A., & Filskov, S. B. (1984). Clinical detection of intellectual deterioration associated with brain damage. Journal of Clinical Psychology, 40, 1435-1441.
110. Leon, J. A., & Perez, O. (2001). The influence of prior knowledge on the time course of clinical diagnosis inferences: A comparison of experts and novices. Discourse Processes, 31, 187-213. http://dx.doi.org/10.1207/S15326950DP3102_04
111. Levenberg, S. B. (1975). Professional training, psychodiagnostic skill, and kinetic family drawings. Journal of Personality Assessment, 39, 389-393. http://dx.doi.org/10.1207/s15327752jpa3904_11
112. Lichtenberg, J. W. (1997). Expertise in counseling psychology: A concept in search of support. Educational Psychology Review, 9, 221-238. http://dx.doi.org/10.1023/A:1024783107643
113. Lichtenberg, J. W. (2009). Comment: Effects of experience on judgment accuracy. The Counseling Psychologist, 37, 410-415. http://dx.doi.org/10.1177/0011000008330829
114. Loewy, M. I. (1994). Size bias by mental health professionals: Use of the illusory correlation paradigm (Unpublished doctoral dissertation). University of California, Santa Barbara, CA.
115. Loganbill, C., Hardy, E., & Delworth, U. (1982). Supervision: A conceptual model. The Counseling Psychologist, 10, 3-42. http://dx.doi.org/10.1177/0011000082101002
116. Lowell, E. S. (1995). Cognitive strategies in psychodiagnosis (Unpublished doctoral dissertation). University of Colorado at Boulder, Boulder, CO.
117. Lubman, D. I., Hides, L., Jorm, A. F., & Morgan, A. J. (2007). Health professionals' recognition of co-occurring alcohol and depressive disorders in youth: A survey of Australian general practitioners, psychiatrists, psychologists and mental health nurses using case vignettes. Australian and New Zealand Journal of Psychiatry, 41, 830-835. http://dx.doi.org/10.1080/00048670701579090
118. Martin, J., Slemon, A. G., Hiebert, B., Hallberg, E. T., & Cummings, A. L. (1989). Conceptualizations of novice and experienced counselors. Journal of Counseling Psychology, 36, 395-400. http://dx.doi.org/10.1037/0022-0167.36.4.395
119. Mayfield, W. A., Kardash, C. M., & Kivlighan, D. M. (1999). Differences in experienced and novice counselors' knowledge structures about clients: Implications for case conceptualization. Journal of Counseling Psychology, 46, 504-514. http://dx.doi.org/10.1037/0022-0167.46.4.504
120. Meyerson, A. T., Moss, J. Z., Belville, R., & Smith, H. (1979). Influence of experience on major clinical decisions: Training implications. Archives of General Psychiatry, 36, 423-427. http://dx.doi.org/10.1001/archpsyc.1979.01780040065007
121. Millard, R. W., & Evans, I. M. (1983). Clinical decision processes and criteria for social validity. Psychological Reports, 53, 775-778. http://dx.doi.org/10.2466/pr0.1983.53.3.775
122. Mohr, J. J., Weiner, J. L., Chopp, R. M., & Wong, S. J. (2009). Effects of client bisexuality on clinical judgment: When is bias most likely to occur?. Journal of Counseling Psychology, 56, 164-175. http://dx.doi.org/10.1037/a0012816
123. Morran, D. K. (1986). Relationship of counselor self-talk and hypothesis formulation to performance level. Journal of Counseling Psychology, 33, 395-400. http://dx.doi.org/10.1037/0022-0167.33.4.395
124. Moxley, A. W. (1973). Clinical judgment: The effects of statistical information. Journal of Personality Assessment, 37, 86-91. http://dx.doi.org/10.1080/00223891.1973.10119834
125. Mumma, G. H., & Mooney, S. R. (2007). Comparing the validity of alternative cognitive case formulations: A latent variable, multivariate time series approach. Cognitive Therapy and Research, 31, 451-481. http://dx.doi.org/10.1007/s10608-006-9062-5
126. Nisbett, R. E., Krantz, D. H., Jepson, C., & Kunda, Z. (1983). The use of statistical heuristics in everyday inductive reasoning. Psychological Review, 90, 339-363.
127. Nisbett, R., & Ross, L. (1980). Human inference: Strategies and shortcomings of human judgment. Englewood Cliffs, NJ: Prentice Hall.
128. O'Byrne, K. R., & Goodyear, R. K. (1997). Client assessment by novice and expert psychologists: A comparison of strategies. Educational Psychology Review, 9, 267-278. http://dx.doi.org/10.1023/A:1024739325390
129. Parker, G., Mahendran, R., Yeo, S. G., Loh, M. I., & Jorm, A. F. (1999). Diagnosis and treatment of mental disorders: A survey of Singapore mental health professionals. Social Psychiatry and Psychiatric Epidemiology, 34, 555-563. http://dx.doi.org/10.1007/s001270050175
130. Parmley, M. C. (2006). The effects of the confirmation bias on diagnostic decision-making (Unpublished doctoral dissertation). Drexel University, Philadelphia, PA.
131. Patterson, D. R. (1982). Social-cognitive biases in clinical judgment (Unpublished doctoral dissertation). Florida State University, Tallahassee, FL.
132. Perez, F. I. (1976). Behavioral analysis of clinical judgment. Perceptual and Motor Skills, 43, 711-718. http://dx.doi.org/10.2466/pms.1976.43.3.711
133. Persons, J. B., & Bertagnolli, A. (1999). Inter-rater reliability of cognitivebehavioral case formulations of depression: A replication. Cognitive Therapy and Research, 23, 271-283. http://dx.doi.org/10.1023/A:1018791531158
134. Pfeiffer, A. M., Whelan, J. P., & Martin, J. M. (2000). Decision-making bias in psychotherapy: Effects of hypothesis source and accountability. Journal of Counseling Psychology, 47, 429-436. http://dx.doi.org/10.1037/0022-0167.47.4.429
135. Platt, S. (Ed.). (1989). Respectfully quoted: A dictionary of quotations requested from the Congressional Research Service. Washington, DC: Library of Congress.
136. Quinsey, V. L., & Cyr, M. (1986). Perceived dangerousness and treatability of offenders: The effects of internal versus external attributions of crime casualty. Journal of Interpersonal Violence, 1, 458-471. http://dx.doi.org/10.1177/088626086001004005
137. Reidy, K. (1987). The effectiveness of the DSM-3 in reducing diagnostic errors with dual diagnosis cases (Unpublished doctoral dissertation). State University of New York at Buffalo, Buffalo, NY.
138. Reiss, S., & Szyszko, J. (1983). Diagnostic overshadowing and professional experience with mentally retarded persons. American Journal of Mental Deficiency, 87, 396-402.
139. Rerick, K. E. (1999). Improving counselors' attitudes and clinical judgments towards dual diagnosis (Unpublished doctoral dissertation). University of South Dakota, Vermillion.
140. Ridley, C. R., Li, L. C., & Hill, C. L. (1998). Multicultural assessment: Reexamination, reconceptualization, and practical application. The Counseling Psychologist, 26, 827-910. http://dx.doi.org/10.1177/0011000098266001
141. Ridley, C. R., & Shaw-Ridley, M. (2009). Clinical judgment accuracy: From meta-analysis to metatheory. The Counseling Psychologist, 37, 400-409. http://dx.doi.org/10.1177/0011000008330830
142. Rieffel, L. M. (2005). The moderating effects of clinician cognitive complexity on the accuracy of neuropsychological clinical judgments for Hispanic patients (Unpublished doctoral dissertation). Fielding Graduate University, Santa Barbara, CA.
143. Rock, D. L., & Bransford, J. D. (1992). An empirical evaluation of three components of the tetrahedron model of clinical judgment. The Journal of Nervous and Mental Disease, 180, 560-565.
144. Rock, D. L., Bransford, J. D., Maisto, S. A., & Morey, L. C. (1987). The study of clinical judgment: An ecological approach. Clinical Psychology Review, 7, 645-661. http://dx.doi.org/10.1016/0272-7358(87)90011-0
145. Rodriguez, C. M. (2002). Professionals' attitudes and accuracy on child abuse reporting decisions in New Zealand. Journal of Interpersonal Violence, 17, 320-342. http://dx.doi.org/10.1177/0886260502017003006
146. Rønnestad, M. H., & Skovholt, T. M. (2013). The developing practitioner: Growth and stagnation of therapists and counselors. New York, NY: Taylor & Francis.
147. Rosenthal, R. (1991). Meta-analytic procedures for social research (Rev. Ed.). Newbury Park, CA: Sage. http://dx.doi.org/10.4135/9781412984997
148. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments West Sussex, UK: Wiley. http://dx.doi.org/10.1002/0470870168
149. Ruscio, J. (2006). The clinician as subject: Practitioners are prone to the same judgment errors as everyone else. In S. Lilienfeld & W. O'Donohue (Eds.), The great ideas of clinical science: 17 principles that every mental health professional should understand (pp. 29-48). New York, NJ: Routledge.
150. Ruscio, J., & Stern, A. (2005). The consistency and accuracy of holistic judgment: Clinical decision making with a minimally complex task. The Scientific Review of Mental Health Practice: Objective Investigations of Controversial and Unorthodox Claims in Clinical Psychology, Psychiatry, and Social Work, 4, 52-65.
151. Sandell, R. (1991). A closer look at the ability to predict psychotherapeutic outcome. Counselling Psychology Quarterly, 4, 127-134. http://dx.doi.org/10.1080/09515079108256716
152. Schinka, J. A., & Sines, J. O. (1974). Correlates of accuracy in personality assessment. Journal of Clinical Psychology, 30, 374-377. http://dx.doi.org/10.1002/1097-4679(197407)30:3_374::AID-JCLP2270300347_3.0.CO;2-F
153. Schmidt, F. L., Oh, I. S., & Hayes, T. L. (2009). Fixed-versus randomeffects models in meta-analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62, 97-128. http://dx.doi.org/10.1348/000711007X255327
154. Schwarz, S., & Stahlberg, D. (2003). Strength of hindsight bias as a consequence of meta-cognitions. Memory (Hove, England), 11, 395-410. http://dx.doi.org/10.1080/09658210244000496
155. Seay, O. J. (1991). Major depression and mental retardation: Effects of psychologist workplace and level of mental retardation on diagnostic overshadowing (Unpublished doctoral dissertation). The University of Texas at Austin, Austin, TX.
156. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for causal inference. Boston, MA: Houghton Mifflin.
157. Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78, 298-311. http://dx.doi.org/10.1037/a0019247
158. Shumaker, K. R. (1999). Measured professional competence between and among different mental health disciplines when evaluating and making recommendations in cases of suspected child sexual abuse (Unpublished doctoral dissertation). United States International University, San Diego, CA.
159. Silverberg, J. J. (1975). Theoretical models of clinical decision-making: The discrimination of significant parameters of the judgmental process (Unpublished doctoral dissertation). University of Iowa, Iowa City, IA.
160. Skovholt, T. M., Rønnestad, M. H., & Jennings, L. (1997). Searching for expertise in counseling, psychotherapy, and professional psychology. Educational Psychology Review, 9, 361-369. http://dx.doi.org/10.1023/A:1024798723295
161. Slavney, P. R., & Chase, G. A. (1985). Clinical judgments of selfdramatisation. British Journal of Psychiatry, 146, 614-617.
162. Spengler, P. M. (2000). Does vocational overshadowing even exist? A test of the robustness of the vocational overshadowing bias. Journal of Counseling Psychology, 47, 342-351.
163. Spengler, P. M. (2013). Clinical versus mechanical prediction. In J. R. Graham & J. A. Nagkieri (Eds.), Handbook of psychology: Vol. 10. Assessment psychology (2nd ed., pp. 26-49). Hoboken, NJ: Wiley.
164. Spengler, P. M., Blustein, D. L., & Strohmer, D. C. (1990). Diagnostic and treatment overshadowing of vocational problems by personal problems. Journal of Counseling Psychology, 37, 372-381. http://dx.doi.org/10.1037/0022-0167.37.4.372
165. Spengler, P. M., Strohmer, D. C., Dixon, D. N., & Shivy, V. A. (1995). A scientist-practitioner model of psychological assessment: Implications for training, practice and research. The Counseling Psychologist, 23, 506-534. http://dx.doi.org/10.1177/0011000095233009
166. Spengler, P. M., Strohmer, D. C., & Prout, H. T. (1990). Testing the robustness of the diagnostic overshadowing bias. American Journal on Mental Retardation, 95, 204-214.
167. Spengler, P. M., White, M. J., Ægisdóttir, S., & Maugherman, A. S. (2009). Time keeps on ticking: The experience of clinical judgment. The Counseling Psychologist, 37, 416-423. http://dx.doi.org/10.1177/0011000009332008
168. Spengler, P. M., White, M. J., Ægisdóttir, S., Maugherman, A. S., Anderson, L. A., Cook, R. S., . . . Rush, J. D. (2009). The Meta-Analysis of Clinical Judgment Project: Effects of experience on judgment accuracy. The Counseling Psychologist, 37, 350-399. http://dx.doi.org/10.1177/0011000006295149
169. Starr, R. H., Jr. (1987). Clinical judgment of abuse-proneness based on parent-child interactions. Child Abuse & Neglect, 11, 87-92. http://dx.doi.org/10.1016/0145-2134(87)90036-6
170. Steiner, C. (1977). The effect of experience, training and cue-availability on clinical judgment (Unpublished doctoral dissertation). Adelphi University, Garden City, NY.
171. Stoltenberg, C. D., McNeill, B. W., & Delworth, U. (1998). IDM supervision: An integrated developmental model for supervising counselors and therapists. San Francisco, CA: Jossey-Bass.
172. Strain, B. A. (2002). Influence of gender bias on the diagnosis of borderline personality disorder (Unpublished doctoral dissertation). Alliant International University, Fresno, CA.
173. Strohmer, D. C., Shivy, V. A., & Chiodo, A. L. (1990). Information processing strategies in counselor hypothesis testing: The role of selective memory and expectancy. Journal of Counseling Psychology, 37, 465-472. http://dx.doi.org/10.1037/0022-0167.37.4.465
174. Temerlin, M. K. (1970). Diagnostic bias in community mental health. Community Mental Health Journal, 6, 110-117. http://dx.doi.org/10.1007/BF01434656
175. Thompson, B. J., & Hill, C. E. (1991). Therapist perceptions of client reactions. Journal of Counseling & Development, 69, 261-265. http://dx.doi.org/10.1002/j.1556-6676.1991.tb01500.x
176. Tracey, T. J., Hays, K. A., Malone, J., & Herman, B. (1988). Changes in counselor response as a function of experience. Journal of Counseling Psychology, 35, 119-126. http://dx.doi.org/10.1037/0022-0167.35.2.119
177. Tracey, T. J., Wampold, B. E., Lichtenberg, J. W., & Goodyear, R. K. (2014). Expertise in psychotherapy: An elusive goal?. American Psychologist, 69, 218-229. http://dx.doi.org/10.1037/a0035099
178. Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison Wesley.
179. Twaites, T. N. (1974). The relationship of confidence to accuracy in clinical prediction (Unpublished doctoral dissertation). University of Minnesota, Minneapolis, MN.
180. van Ijzendoorn, W. J. E., & Bus, A. G. (1993). How valid are experts' prognoses on children with learning problems?. Journal of School Psychology, 31, 317-325. http://dx.doi.org/10.1016/0022-4405(93)90014-A
181. Walker, B. S. (1999). The diagnosis and treatment of major depression in Aids patients: Effect of counselor experience and attitude toward people with AIDS (Unpublished doctoral dissertation). Ball State University, Muncie, IN.
182. Walker, E., & Lewine, R. J. (1990). Prediction of adult-onset schizophrenia from childhood home movies of the patients. The American Journal of Psychiatry, 147, 1052-1056. http://dx.doi.org/10.1176/ajp.147.8.1052
183. Walters, G. D., White, T. W., & Greene, R. L. (1988). Use of the MMPI to identify malingering and exaggeration of psychiatric symptomatology in male prison inmates. Journal of Consulting and Clinical Psychology, 56, 111-117. http://dx.doi.org/10.1037/0022-006X.56.1.111
184. Wampold, B. E., Casas, J. M., & Atkinson, D. R. (1981). Ethic bias in counseling: An information processing approach. Journal of Counseling Psychology, 28, 498-503. http://dx.doi.org/10.1037/0022-0167.28.6.498
185. Wedding, D. (1983). Clinical and statistical prediction in neuropsychology. Clinical Neuropsychology, 5, 49-55.
186. Werner, P. D., Rose, T. L., & Yesavage, J. A. (1983). Reliability, accuracy, and decision-making strategy in clinical predictions of imminent dangerousness. Journal of Consulting and Clinical Psychology, 51, 815-825. http://dx.doi.org/10.1037/0022-006X.51.6.815
187. White, M. J., Nichols, C. N., Cook, R. S., Spengler, P. M., Walker, B. S., & Look, K. K. (1995). Diagnostic overshadowing and mental retardation: A meta-analysis. American Journal on Mental Retardation, 100, 293-298.
188. Wiggins, J. (1973). Personality and prediction: Principles of personality assessment. Reading, MA: Addison Wesley.
189. Wilkin, B. S. (2001). The diagnosis of attention-deficit/hyperactivity disorder (ADHD) by psychologists, pediatricians, and general practitioners (Unpublished doctoral dissertation). University of Windsor, Windsor, Canada.
190. Wilson, C. A., & Gettinger, M. (1989). Determinants of child-abuse reporting among Wisconsin school psychologists. Professional School Psychology, 4, 91-102.
191. Wisch, A. F., & Mahalik, J. R. (1999). Male therapists' clinical bias: Influence of client gender roles and therapist gender role conflict. Journal of Counseling Psychology, 46, 51-60. http://dx.doi.org/10.1037/0022-0167.46.1.51
192. Witteman, C. L. M., & van den Bercken, J. H. L. (2007). Intermediate effects in psychodiagnostic classification. European Journal of Psychological Assessment, 23, 56-61. http://dx.doi.org/10.1027/1015-5759.23.1.56
193. Wood, D. S. (2004). An intervention for diagnostic overshadowing (Unpublished doctoral dissertation). Arizona State University, Tempe, AZ.
194. Wood, W., & Eagly, A. H. (2009). Advantages of certainty and uncertainty. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The Handbook of Research Synthesis and Meta-Analysis (pp. 455-472). New York, NY: The Russell Sage Foundation.
195. Yeo, S. G., Parker, G., Mahendran, R., Jorm, A. F., Yap, H. L., Lee, C., & Loh, M. I. (2001). Mental health literacy survey of psychiatrically and generally trained nurses employed in a Singapore psychiatric hospital. International Journal of Nursing Practice, 7, 414-421. http://dx.doi.org/10.1046/j.1440-172X.2001.00321.x
196. Zozula, L. J. (2001). The assessment process of psychologists as a function of clinical experience (Unpublished doctoral dissertation). McGill University, Montreal, Canada.