A meta-analysis of confidence and judgment accuracy in clinical decision making

Journal of Counseling Psychology

Volumn 62, Issue 4, 2015, Pages 553-567

  Miller, Deborah J ^a   Spengler, Elliot S ^a   Spengler, Paul M ^a  

^a BALL STATE UNIVERSITY   (United States)

Author keywords

Assessment; Clinical judgment; Meta analysis; Overconfidence bias

Indexed keywords

CLINICAL DECISION MAKING; DECISION MAKING; EMOTION; HUMAN; MALE; META ANALYSIS; PROCEDURES; PSYCHOLOGY; RISK ASSESSMENT; SELF CONCEPT; STANDARDS;

CLINICAL DECISION-MAKING; EMOTIONS; HUMANS; JUDGMENT; MALE; PSYCHOLOGY; RISK ASSESSMENT; SELF CONCEPT;

EID: 84973161408     PISSN: 00220167     EISSN: None     Source Type: Journal    
DOI: 10.1037/cou0000105     Document Type: Article

References (117)

1. Aamodt, M. G., & Custer, H. (2006). Who can best catch a liar? A meta-analysis of individual differences in detecting deception. Forensic Examiner, 15, 6-11.
2. Ægisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., ... Rush, J. D. (2006). The Meta-Analysis of Clinical Judgment project: Fifty-six years of accumulated research on clinical versus statistical prediction. The Counseling Psychologist, 34, 341-382. http://dx.doi.org/10.1177/0011000005285875.
3. American Psychological Association. (2015). Guidelines for clinical supervision in health service psychology. American Psychologist, 70, 33-46. http://dx.doi.org/10.1037/a0038112.
4. Arkes, H. R. (1981). Impediments to accurate clinical judgment and possible ways to minimize their impact. Journal of Consulting and Clinical Psychology, 49, 323-330. http://dx.doi.org/10.1037/0022-006X.49.3.323.
5. Becker, B. J. (2005). The failsafeN or file-drawer number. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 111-126). West Sussex, UK: Wiley.
6. Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. (2005). Comprehensive Meta-Analysis Version 2. Engelwood, NJ: Biostat.
7. Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. (2009). Introduction to meta-analysis. Chichester, UK: Wiley. http://dx.doi.org/10.1002/9780470743386.
8. Borman, G. D., & Grigg, J. A. (2009). Visual and narrative interpretation. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 497-520). New York, NY: Russell Sage Foundation.
9. Borum, R., Otto, R., & Golding, S. (1993). Improving clinical judgment and decision making in forensic evaluation. The Journal of Psychiatry & Law, 21, 35-76.
10. *Boyle, P. A. (2000). Decision-making strategies used by neuropsychologists in the differential diagnosis of dementia (Unpublished doctoral dissertation). Amherst, MA: University of Massachusetts.
11. *Cantor, N., Smith, E. E., French, R. S., & Mezzich, J. (1980). Psychiatric diagnosis as prototype categorization. Journal of Abnormal Psychology, 89, 181-193. http://dx.doi.org/10.1037/0021-843X.89.2.181.
12. *Carlin, A. S., & Hewitt, P. L. (1990). The discrimination of patient-generated and randomly generated MMPIs. Journal of Personality Assessment, 54, 24-29. http://dx.doi.org/10.1207/s15327752jpa5401&2_3.
13. *Carroll, N., Rosenberg, H., & Funke, S. (1988). Recognition of intoxication by alcohol counselors. Journal of Substance Abuse Treatment, 5, 239-246. http://dx.doi.org/10.1016/0740-5472(88)90046-3.
14. Cochrane Library. (2015). The Cochrane Collaboration. Chichester, UK: Wiley. Retrieved from http://www.cochranelibrary.com.
15. Cooper, H. (1998). Synthesizing research: A guide for literature reviews (3rd ed.). Beverly Hills, CA: Sage.
16. Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.
17. *Cooper, R. P., & Werner, P. D. (1990). Predicting violence in newly admitted inmates. Criminal Justice and Behavior, 17, 417-431. http://dx.doi.org/10.1177/0093854890017004004.
18. Corey, D. M., Dunlap, W. P., & Burke, M. J. (1998). Averaging correlations: Expected values and bias in combined Pearson rs and Fisher's z transformations. The Journal of General Psychology, 125, 245-261. http://dx.doi.org/10.1080/00221309809595548.
19. Cramer, R. J., DeCoster, J., Harris, P. B., Fletcher, L. M., & Brodsky, S. L. (2011). A confidence-credibility model of expert witness persuasion: Mediating effects and implications for trial consultation. Consulting Psychology Journal: Practice and Research, 63, 129-137. http://dx.doi.org/10.1037/a0024591.
20. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302. http://dx.doi.org/10.1037/h0040957.
21. Dauber, L. M. (1980). An evaluation of the process of clinical judgment based on case history information. (Unpublished doctoral dissertation). Buffalo, NY: State University of New York at Buffalo.
22. *Desmarais, S. L., Nicholls, T. L., Read, J. D., & Brink, J. (2010). Confidence and accuracy in assessments in short-term risks presented by forensic psychiatric patients. Journal of Forensic Psychiatry & Psychology, 21, 1-22. http://dx.doi.org/10.1080/14789940903183932.
23. *Douglas, K. S., & Ogloff, J. R. (2003). The impact of confidence on the accuracy of structured professional and actuarial violence risk judgments in a sample of forensic psychiatric patients. Law and Human Behavior, 27, 573-587. http://dx.doi.org/10.1023/B:LAHU.0000004887.50905.f7.
24. Dunning, D., Heath, C., & Suls, J. M. (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5, 69-106. http://dx.doi.org/10.1111/j.1529-1006.2004.00018.x.
25. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634. http://dx.doi.org/10.1136/bmj.315.7109.629.
26. Einhorn, H. J. (1980). Overconfidence in judgment. New Directions for Methodology of Social and Behavioral Sciences, 4, 1-15.
27. *Ekman, P., & O'Sullivan, M. (1991). Who can catch a liar? American Psychologist, 46, 913-920. http://dx.doi.org/10.1037/0003-066X.46.9.913.
28. *Elkovitch, N., Viljoen, J. L., Scalora, M. J., & Ullman, D. (2008). Assessing risk of reoffending in adolescents who have committed a sexual offense: The accuracy of clinical judgments after completion of risk assessment instruments. Behavioral Sciences & the Law, 26, 511-528. http://dx.doi.org/10.1002/bsl.832.
29. Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406. http://dx.doi.org/10.1037/0033-295X.100.3.363.
30. Faust, D., & Faust, K. A. (2012). Experts' experience and diagnostic and predictive accuracy. In D. Faust (Ed.), Coping with psychiatric and psychological testimony: Based on the original work by Jay Ziskin (6th ed., pp. 131-146). New York, NY: Oxford University Press.
31. *Fero, D. D. (1975). A lens model analysis of the effects of amount of information and mechanical decision-making aid on clinical judgment and confidence (Unpublished doctoral dissertation). Bowling Green, OH: Bowling Green State University.
32. Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977). Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance, 3, 552-564. http://dx.doi.org/10.1037/0096-1523.3.4.552.
33. Garb, H. N. (1986). The appropriateness of confidence ratings in clinical judgment. Journal of Clinical Psychology, 42, 190-197. http://dx.doi.org/10.1002/1097-4679(198601)42:1<190::AID-JCLP2270420133>3.0.CO;2-6.
34. Glidewell, J. C., & Livert, D. E. (1992). Confidence in the practice of clinical psychology. Professional Psychology, Research and Practice, 23, 362-368. http://dx.doi.org/10.1037/0735-7028.23.5.362.
35. Goldberg, L. R. (1959). The effectiveness of clinicians' judgments; the diagnosis of organic brain damage from the Bender-Gestalt test. Journal of Consulting Psychology, 23, 25-33. http://dx.doi.org/10.1037/h0048736.
36. *Greenfield, M. F., & Haaga, D. A. F. (2011). Accuracy and confidence of training therapists' recognition of sessions before sudden gains. Journal of Cognitive and Behavioral Psychotherapies, 11, 157-172.
37. Greenhouse, J. B., & Iyengar, S. (2009). Sensitivity analysis and diagnostics. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 417-434). New York, NY: Russell Sage Foundation.
38. Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19-30. http://dx.doi.org/10.1037/1040-3590.12.1.19.
39. *Haderlie, M. M. (2011). Enhancing therapists' clinical judgments of client progress subsequent to objective feedback (Unpublished doctoral dissertation). Las Vegas, NV: University of Nevada.
40. Hafdahl, A. R. (2010). Random-effects meta-analysis of correlations: Monte Carlo evaluation of mean estimators. The British Journal of Mathematical and Statistical Psychology, 63, 227-254. http://dx.doi.org/10.1348/000711009X431914.
41. Harris, G. T., Rice, M. E., Quinsey, V. L., & Cormier, C. A. (2015). Violent offenders: Appraising and managing risk (3rd ed.). Washington, DC: American Psychological Association. http://dx.doi.org/10.1037/14572-000.
42. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
43. Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9, 426-445.
44. Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557-560. http://dx.doi.org/10.1136/bmj.327.7414.557.
45. Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557-560. http://dx.doi.org/10.1136/bmj.327.7414.557.
46. Hirsch, P. A., & Stone, G. L. (1983). Cognitive strategies and the client conceptualization process. Journal of Counseling Psychology, 30, 566-572. http://dx.doi.org/10.1037/0022-0167.30.4.566.
47. Hoaglin, D. C., & Iglewicz, B. (1987). Fine tuning some resistant rules for outlier labeling. Journal of the American Statistical Association, 82, 1147-1149. http://dx.doi.org/10.1080/01621459.1987.10478551.
48. Holsopple, J. Q., & Phelan, J. G. (1954). The skills of clinicians in analysis of projective tests. Journal of Clinical Psychology, 10, 307-320. http://dx.doi.org/10.1002/1097-4679(195410)10:4<307::AID-JCLP2270100402>3.0.CO;2-3.
49. *Hopwood, C. J., & Richard, D. C. (2005). Graduate student WAIS-III scoring accuracy is a function of full scale IQ and complexity of examiner tasks. Assessment, 12, 445-454. http://dx.doi.org/10.1177/1073191105281072.
50. Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar, Straus & Giroux.
51. Kahneman, D., Slovic, P., & Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases. Cambridge, UK: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511809477.
52. Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80, 237-251. http://dx.doi.org/10.1037/h0034747.
53. *Kalichman, S. C., & Craig, M. E. (1991). Professional psychologists' decisions to report suspected child abuse: Clinician and situation influences. Professional Psychology, Research and Practice, 22, 84-89. http://dx.doi.org/10.1037/0735-7028.22.1.84.
54. *Kalichman, S. C., Craig, M. E., & Follingstad, D. R. (1989). Factors influencing the reporting of father-child sexual abuse: Study of licensed practicing psychologists. Professional Psychology, Research and Practice, 20, 84-89. http://dx.doi.org/10.1037/0735-7028.20.2.84.
55. Keillor, G. (2014). The Keillor reader. New York, NY: Penguin Group.
56. *Kendell, R. E. (1973). Psychiatric diagnoses: A study of how they are made. The British Journal of Psychiatry, 122, 437-445. http://dx.doi.org/10.1192/bjp.122.4.437.
57. Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980). Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory, 6, 107-118. http://dx.doi.org/10.1037/0278-7393.6.2.107.
58. Lambert, J., Bessière, V., & Ngoala, G. (2012). Does expertise influence the impact of overconfidence on judgment, valuation and investment decision? Journal of Economic Psychology, 33, 1115-1128. http://dx.doi.org/10.1016/j.joep.2012.07.007.
59. Lambert, M. J., Hansen, N., Umphress, V., Lunnen, K., Okiishi, J., Burlingame, G., ... Reisinger, C. (1996). Administration and scoring manual for the Outcome Questionnaire (OQ 45.2). Wilmington, DL: American Professional Credentialing.
60. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174. http://dx.doi.org/10.2307/2529310.
61. *Lefkowitz, M. B. (1973). Statistical and clinical approaches to the identification of couples at risk in marriage (Unpublished doctoral dissertation). Gainesville, FL: University of Florida.
62. *Leli, D. A., & Filskov, S. B. (1981). Clinical-actuarial detection and description of brain impairment with the W-B form I. Journal of Clinical Psychology, 37, 623-629. http://dx.doi.org/10.1002/1097-4679(198107)37:3<623::AID-JCLP2270370330>3.0.CO;2-V.
63. *Leli, D. A., & Filskov, S. B. (1984). Clinical detection of intellectual deterioration associated with brain damage. Journal of Clinical Psychology, 40, 1435-1441. http://dx.doi.org/10.1002/1097-4679(198411)40:6<1435::AID-JCLP2270400629>3.0.CO;2-0.
64. *Levenberg, S. B. (1975). Professional training, psychodiagnostic skill, and kinetic family drawings. Journal of Personality Assessment, 39, 389-393. http://dx.doi.org/10.1207/s15327752jpa3904_11.
65. Lichtenstein, S., & Fischhoff, B. (1977). Do those who know more also know more about how much they know? The calibration of probability judgments. Organizational Behavior and Human Performance, 10, 159-183. http://dx.doi.org/10.1016/0030-5073(77)90001-0.
66. Lilienfeld, S. O., Lynn, S. J., & Lohr, J. M. (2015). Science and pseudo-science in clinical psychology (2nd ed.). New York, NY: Guilfrod Press.
67. Lipko, A. R., Dunlosky, J., & Merriman, W. E. (2009). Persistent over-confidence despite practice: The role of task experience in preschoolers' recall predictions. Journal of Experimental Child Psychology, 103, 152-166. http://dx.doi.org/10.1016/j.jecp.2008.10.002.
68. Martin, J. M. (2001). Confirmation bias in the therapy session: The effects of expertise, external validity, instruction set, confidence and diagnostic accuracy (Unpublished doctoral dissertation). Memphis, TN: The University of Memphis.
69. Matt, G. E., & Cook, T. D. (2009). Threats to the validity of generalized inferences. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 537-560). New York, NY: Russell Sage Foundation.
70. *McNiel, D. E., Sandberg, D. A., & Binder, R. L. (1998). The relationship between confidence and accuracy in clinical assessment of psychiatric patients' potential for violence. Law and Human Behavior, 22, 655-669. http://dx.doi.org/10.1023/A:1025754706716.
71. Meehl, P. E. (1957). When shall we use our heads instead of the formula? Journal of Counseling Psychology, 4, 268-273. http://dx.doi.org/10.1037/h0047554.
72. *Moxley, A. W. (1973). Clinical judgment: The effects of statistical information. Journal of Personality Assessment, 37, 86-91. http://dx.doi.org/10.1080/00223891.1973.10119834.
73. *Nadler, J. D., Mittenberg, W., DePiano, F. A., & Schneider, B. A. (1994). Effects of patient age on neuropsychological test interpretation. Professional Psychology, Research and Practice, 3, 288-295. http://dx.doi.org/10.1037/0735-7028.25.3.288.
74. Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 175-220. http://dx.doi.org/10.1037/1089-2680.2.2.175.
75. Nisbett, R., & Ross, L. (1980). Human inference: Strategies and shortcomings of social judgment. Englewood Cliffs, NJ: Prentice Hall.
76. Norcross, J. C., & Lambert, M. J. (2011). Evidenced-based therapy relationships. In J. C. Norcross (Ed.), Psychotherapy relationships that work: Evidence-based responsiveness (2nd ed., pp. 3-24). New York, NY: Oxford University Press. http://dx.doi.org/10.1093/acprof:oso/9780199737208.003.0001.
77. Olsson, N., & Juslin, P. (2003). Calibration of confidence among eyewitnesses and earwitnesses. In P. Chambres, M. Izaute, & P. Marescaux (Eds.), Metacognition: Process, function and use (pp. 203-218). Dordrecht, the Netherlands: Kluwer Academic Publishers.
78. Oskamp, S. (1965). Overconfidence in case-study judgments. Journal of Consulting Psychology, 29, 261-265. http://dx.doi.org/10.1037/h0022125.
79. Overton, R. C. (1998). A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects. Psychological Methods, 3, 354-379. http://dx.doi.org/10.1037/1082-989X.3.3.354.
80. Owen, J. (2008). The nature of confirmatory strategies in the initial assessment process. Journal of Mental Health Counseling, 30, 362-374. http://dx.doi.org/10.17744/mehc.30.4.55281735v473x055.
81. *Pedley, M. (1994). The influence of client race on therapists' judgment making and memory for session-related material: An investigation of information processing among psychologists in training (Unpublished doctoral dissertation). Binghamton, NY: State University of New York.
82. Price, P. C., & Stone, E. R. (2004). Intuitive evaluation of likelihood judgment producers: Evidence for a confidence heuristic. Journal of Behavioral Decision Making, 17, 39-57. http://dx.doi.org/10.1002/bdm.460.
83. *Rabinowitz, J., & Garelik-Wyler, R. (1999). Accuracy and confidence in clinical assessment of psychiatric inpatients risk of violence. International Journal of Law and Psychiatry, 22, 99-106. http://dx.doi.org/10.1016/S0160-2527(98)00032-6.
84. Regehr, C., Bogo, M., Shlonsky, A., & LeBlanc, V. (2010). Confidence and professional judgment in assessing children's risk of abuse. Research on Social Work Practice, 20, 621-628. http://dx.doi.org/10.1177/1049731510368050.
85. Ridley, C. R., Li, L. C., & Hill, C. L. (1998). Multicultural assessment: Reexamination, reconceptualization, and practical application. The Counseling Psychologist, 26, 827-910. http://dx.doi.org/10.1177/0011000098266001.
86. *Rodriguez, C. M. (2002). Professionals' attitudes and accuracy on child abuse reporting decisions in New Zealand. Journal of Interpersonal Violence, 17, 320-342. http://dx.doi.org/10.1177/0886260502017003006.
87. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641. http://dx.doi.org/10.1037/0033-2909.86.3.638.
88. Rosenthal, R. (1984). Meta analytic procedures for social science. Applied Social Science Research Methods (Vol. 6). Beverly Hills, CA: Sage.
89. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. West Sussex, UK: Wiley. http://dx.doi.org/10.1002/0470870168.
90. *Ruscio, J., & Stern, A. R. (2005). The consistency and accuracy of holistic judgment: Clinical decision making with a minimally complex task. The Scientific Review of Mental Health Practice, 4, 52-65.
91. *Ryan, J. J., & Schnakenberg-Ott, S. D. (2003). Scoring reliability on the Wechsler Adult Intelligence Scale (3rd ed.). Assessment, 10, 151-159. http://dx.doi.org/10.1177/1073191103010002006.
92. Ryback, D. (1967). Confidence and accuracy as a function of experience in judgment-making in the absence of systematic feedback. Perceptual and Motor Skills, 24, 331-334. http://dx.doi.org/10.2466/pms.1967.24.1.331.
93. Sackett, P. R., Harris, M. M., & Orr, J. M. (1986). On seeking moderator variables in the meta-analysis of correlational data: A Monte Carlo investigation of statistical power and resistance to Type I error. Journal of Applied Psychology, 71, 302-310. http://dx.doi.org/10.1037/0021-9010.71.2.302.
94. Semmler, C., Brewer, N., & Douglass, A. B. (2012). Jurors believe eyewitnesses. In B. L. Cutler (Ed.), Convictions of the innocent: Lessons from psychological research (pp. 185-209). Washington, DC: American Psychological Association. http://dx.doi.org/10.1037/13085-009.
95. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin and Company.
96. Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78, 298-311. http://dx.doi.org/10.1037/a0019247.
97. Smith, J., & Agate, J. (2004). Solutions for overconfidence: Evaluation of an instructional module for counselor trainees. Counselor Education and Supervision, 44, 31-43. http://dx.doi.org/10.1002/j.1556-6978.2004.tb01858.x.
98. Smith, J., & Dumont, F. (1997). Eliminating overconfidence in psychodiagnosis: Strategies for training and practice. Clinical Psychology: Science and Practice, 4, 335-345. http://dx.doi.org/10.1111/j.1468-2850.1997.tb00125.x.
99. Smith, J., & Dumont, F. (2002). Confidence in psychodiagnosis: What makes us so sure? Clinical Psychology & Psychotherapy, 9, 292-298. http://dx.doi.org/10.1002/cpp.336.
100. Spengler, P. M. (2013). Clinical versus mechanical prediction. In J. Graham & J. Naglieri (Eds.), Handbook of psychology (2nd ed.). Assessment Psychology (Vol. 10, pp. 26-49). Hoboken, NJ: Wiley.
101. Spengler, P. M., & Pilipis, L. A. (2015, March 23). A comprehensive meta-reanalysis of the robustness of the experience-accuracy effect in clinical judgment. [Advance online publication]. Journal of Counseling Psychology. http://dx.doi.org/10.1037/cou0000065.
102. Spengler, P. M., Strohmer, D. C., Dixon, D. N., & Shivy, V. A. (1995). A scientist-practitioner model of psychological assessment: Implications for training, practice and research. The Counseling Psychologist, 23, 506-534. http://dx.doi.org/10.1177/0011000095233009.
103. Spengler, P. M., White, M. J., Ægisdóttir, S., Maugherman, A. S., Anderson, L. A., Cook, R. S., ... Rush, J. D. (2009). The Meta-Analysis of Clinical Judgment project: Effects of experience on judgment accuracy. The Counseling Psychologist, 37, 350-399. http://dx.doi.org/10.1177/0011000006295149.
104. Sporer, S. L., Penrod, S., Read, D., & Cutler, B. L. (1995). Choosing, confidence, and accuracy: A meta-analysis of the confidence-accuracy relation in eyewitness identification studies. Psychological Bulletin, 118, 315-327. http://dx.doi.org/10.1037/0033-2909.118.3.315.
105. *Stemple, D. M. (1985). The clinical judgment process of prediction of behavior problems in hospitalized adolescents using the MMPI and the Rorschach (Unpublished doctoral dissertation). New York, NY: Fordham University.
106. Strohmer, D. C., Shivy, V. A., & Chiodo, A. L. (1990). Information processing strategies in counselor hypothesis testing: The role of selective memory and expectancy. Journal of Counseling Psychology, 37, 465-472. http://dx.doi.org/10.1037/0022-0167.37.4.465.
107. Tenney, E. R., Small, J. E., Kondrad, R. L., Jaswal, V. K., & Spellman, B. A. (2011). Accuracy, confidence, and calibration: How young children and adults assess credibility. Developmental Psychology, 47, 1065-1077. http://dx.doi.org/10.1037/a0023273.
108. Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison Wesley.
109. *Twaites, T. N. (1974). The relationship of confidence to accuracy in clinical prediction (Unpublished doctoral dissertation). Minneapolis, MN: University of Minnesota.
110. Walfish, S., McAlister, B., O'Donnell, P., & Lambert, M. J. (2012). An investigation of self-assessment bias in mental health providers. Psychological Reports, 110, 639-644. http://dx.doi.org/10.2466/02.07.17.PR0.110.2.639-644.
111. *Walker, E., & Lewine, R. J. (1990). Prediction of adult-onset schizophrenia from childhood home movies of the patients. The American Journal of Psychiatry, 147, 1052-1056. http://dx.doi.org/10.1176/ajp.147.8.1052.
112. *Walters, G. D., White, T. W., & Greene, R. L. (1988). Use of the MMPI to identify malingering and exaggeration of psychiatric symptomatology in male prison inmates. Journal of Counseling and Clinical Psychology, 56, 111-117. http://dx.doi.org/10.1037/0022-006X.56.1.111.
113. White, M. J., Nichols, C. N., Cook, R. S., Spengler, P. M., Walker, B. S., & Look, K. K. (1995). Diagnostic overshadowing and mental retardation: A meta-analysis. American Journal on Mental Retardation, 100, 293-298.
114. *Wittemann, C. L., & van den Bercken, J. H. (2007). Intermediate effects in psychodiagnostic classification. European Journal of Psychological Assessment, 23, 56-61. http://dx.doi.org/10.1027/1015-5759.23.1.56.
115. Wood, W., & Eagly, A. H. (2009). Advantages of certainty and uncertainty. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 455-472). New York, NY: Russell Sage Foundation.
116. Woodman, T., & Hardy, L. (2003). The relative impact of cognitive anxiety and self-confidence upon sport performance: A meta-analysis. Journal of Sports Sciences, 21, 443-457. http://dx.doi.org/10.1080/0264041031000101809.
117. *Young, R. C. (1972). Clinical judgment as a means of improving actuarial prediction from the MMPI. Journal of Consulting and Clinical Psychology, 38, 457-459. http://dx.doi.org/10.1037/h0032915.