Training to Improve Calibration and Discrimination: The Effects of Performance and Environmental Feedback

Organizational Behavior and Human Decision Processes

Volumn 83, Issue 2, 2000, Pages 282-309

  Stone, Eric R ^a   Opel, Ryan B ^a  

^a WAKE FOREST UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0345856407     PISSN: 07495978     EISSN: None     Source Type: Journal    
DOI: 10.1006/obhd.2000.2910     Document Type: Article

References (66)

1. Adams P. A., Adams J. K. Training in confidence judgments. American Journal of Psychology. 71:1958;747-751.
2. Alpert, M, &, Raiffa, H. (, 1969,), A progress report on the training of probability assessors. Unpublished manuscript.
3. Arkes H. R., Christensen C., Lai C., Blumer C. Two methods of reducing overconfidence. Organizational Behavior and Human Decision Processes. 39:1987;133-144.
4. Ayton P. On the competence and incompetence of experts. Wright G., Bolger F. Expertise and decision support. 1992;77-105 Plenum Press, New York.
5. Balzer W. K., Doherty M. E., O'Connor R. Jr. Effects of cognitive feedback on performance. Psychological Bulletin. 106:1989;410-433.
6. Balzer W. K., Hammer L. B., Sumner K. E., Birchenough T. R., Martens S. P., Raymark P. H. Effects of cognitive feedback components, display format, and elaboration on performance. Organizational Behavior and Human Decision Processes. 58:1994;369-385.
7. Balzer W. K., Sulsky L. M., Hammer L. B., Sumner K. E. Task information, cognitive information, or functional validity information: Which components of cognitive feedback affect performance? Organizational Behavior and Human Decision Processes. 53:1992;35-54.
8. Benson P. G., Curley S. P., Smith G. F. Belief assessment: An underdeveloped phase of probability elicitation. Management Science. 41:1995;1639-1653.
9. Benson P. G., Önkal D. The effects of feedback and training on the performance of probability forecasters. International Journal of Forecasting. 8:1992;559-573.
10. Bolger, F, &, Wright, G. (, 1992,), Reliability and validity in expert judgment. In, G. Wright and F. Bolger, (Eds.), Expertise and decision support, (pp, 47, - 76,), New York, Plenum Press.
11. Bornstein B. H., Zickafoose D. J. "I know I know it, I know I saw it": The stability of the confidence-accuracy relationship across domains. Journal of Experimental Psychology: Applied. 5:1999;76-88.
12. Braun P. A., Yaniv I. A case study of expert judgment: Economists' probabilities versus base-rate model forecasts. Journal of Behavioral Decision Making. 5:1992;217-231.
13. Brenner L. A., Koehler D. J., Liberman V., Tversky A. Overconfidence in probability and frequency judgments: A critical examination. Organizational Behavior and Human Decision Processes. 65:1996;212-219.
14. Brier G. W. Verification of forecasts expressed in terms of probability. Monthly Weather Review. 78:1950;1-3.
15. Chan S. Expert judgments under uncertainty: Some evidence and suggestions. Social Science Quarterly. 63:1982;428-444.
16. Christensen-Szalanski J. J. J., Bushyhead J. B. Physicians' use of probabilistic information in a real clinical setting. Journal of Experimental Psychology: Human Perception and Performance. 7:1981;928-935.
17. Dawes R. M., Mulford M. The false consensus effect and overconfidence: Flaws in judgment or flaws in how we study judgment. Organizational Behavior and Human Decision Processes. 65:1996;201-211.
18. Eddy D. M. Probabilistic reasoning in clinical medicine: Problems and opportunities. Kahneman D., Slovic P., Tversky A. Judgment under uncertainty: Heuristics and biases. 1982;249-267 Cambridge University Press, Cambridge.
19. Einhorn H. J. Learning from experience and suboptimal rules in decision making. Kahneman D., Slovic P., Tversky A. Judgment under uncertainty: Heuristics and biases. 1982;268-283 Cambridge University Press, Cambridge.
20. Erev I., Wallsten T. S., Budescu D. V. Simultaneous over- and underconfidence: The role of error in judgment processes. Psychological Review. 101:1994;519-527.
21. Ferrell W. R., McGoey P. J. A model of calibration for subjective probabilities. Organizational Behavior and Human Decision Performance. 26:1980;32-53.
22. Fischer G. W. Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting. Organizational Behavior and Human Performance. 29:1982;352-369.
23. Fischhoff B. Debiasing. Kahneman D., Slovic P., Tversky A. Judgment under uncertainty: Heuristics and biases. 1982;422-444 Cambridge University Press, Cambridge.
24. Garb H. N. Clinical judgment, clinical training, and professional experience. Psychological Bulletin. 105:1989;387-396.
25. Gardner H. Tansey R. G., Kleiner F. S. Gardner's art through the ages. 1996;Harcourt Brace College, New York.
26. Kahneman D., Lovallo D. Timid choices and bold forecasts: A cognitive perspective on risk taking. Management Science. 39:1993;17-31.
27. Keren G. Facing uncertainty in the game of bridge: A calibration study. Organizational Behavior and Human Decision Processes. 39:1987;98-114.
28. Keren G. Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica. 77:1991;217-273.
29. Liberman V., Tversky A. On the evaluation of probability judgments: Calibration, resolution, and monotonicity. Psychological Bulletin. 114:1993;162-173.
30. Lichtenstein S., Fischhoff B. Do those who know more also know more about how much they know? The calibration of probability judgments. Organizational Behavior and Human Performance. 20:1977;159-183.
31. Lichtenstein S., Fischhoff B. Training for calibration. Organizational Behavior and Human Performance. 26:1980;149-171.
32. Lichtenstein S., Fischhoff B., Phillips L. D. Calibration of probabilities: The state of the art to 1980. Kahneman D., Slovic P., Tversky A. Judgment under uncertainty: Heuristics and biases. 1982;306-334 Cambridge University Press, Cambridge.
33. McClelland A. G. R., Bolger F. The calibration of subjective probabilities: Theories and models 1980-1994. Wright G., Ayton P. Subjective probability. 1994;453-482 Wiley, Chichester.
34. Murphy A. H. A new vector partition of the probability score. Journal of Applied Meteorology. 12:1973;595-600.
35. Murphy A. H., Winkler R. L. Probability forecasting in meteorology. Journal of the American Statistical Association. 79:1984;489-500.
36. Oskamp S. The relationship of clinical experience and training methods to several criteria of clinical prediction. Psychological Monographs. 76:1962;1-25.
37. Oskamp S. Overconfidence in case-study judgments. Journal of Consulting Psychology. 29:1965;261-265.
38. Parker, A. M, Downs, J. S, Fischhoff, B, Bruine de Bruin, W, &, Dawes, R. M. (, 1999,), The appropriateness of adolescents' confidence in their knowledge: AIDS-related and general. Manuscript submitted for publication.
39. Payne D. G., Wenger J. J. Cognitive psychology. 1998;Houghton Mifflin, Boston.
40. Petrusic W. M., Baranski J. V. Context, feedback, and the calibration and resolution of confidence in perceptual judgments. American Journal of Psychology. 110:1997;543-572.
41. Plous, S. (, 1993,), The psychology of judgment and decision making, New York, McGraw-Hill.
42. Pulford B. D., Colman A. M. Overconfidence: Feedback and item difficulty effects. Personality and Individual Differences. 23:1997;125-133.
43. Remus W., O'Connor M., Griggs K. Does feedback improve the accuracy of recurrent judgmental forecasts. Organizational Behavior and Human Decision Processes. 66:1996;22-30.
44. Roth P. L. Research trends in judgment and their implications for the Schmidt-Hunter global estimation procedure. Organizational Behavior and Human Decision Processes. 54:1993;299-319.
45. Schneider S. Item difficulty, discrimination, and the confidence-frequency effect in a categorical judgment task. Organizational Behavior and Human Decision Processes. 61:1995;148-167.
46. Shanteau J. Psychological characteristics and strategies of expert decision makers. Acta Psychologica. 68:1988;203-215.
47. Sharp G. L., Cutler B. L., Penrod S. D. Performance feedback improves the resolution of confidence judgments. Organizational Behavior and Human Decision Processes. 42:1988;271-283.
48. Smith D., Dumont F. Eliminating overconfidence in psychodiagnosis: Strategies for training and practice. Clinical Psychology: Science and Practice. 4:1997;335-345.
49. Stankov L., Crawford J. D. Self-confidence and performance on tests of cognitive abilities. Intelligence. 25:1997;93-109.
50. Stone, E. R, &, Hoffman, R. R. (, 1999,), The distinction between calibration and substantive expertise in the evaluation of judgment tasks. Unpublished manuscript.
51. Suantak L., Bolger F., Ferrell W. R. The hard-easy effect in subjective probability calibration. Organizational Behavior and Human Decision Processes. 67:1996;201-221.
52. Subbotin V. Outcome feedback effects on under- and overconfident judgments (general knowledge tasks). Organizational Behavior and Human Decision Processes. 66:1996;268-276.
53. Wallace H. A. What is in the corn judge's mind. Journal of the American Society of Agronomy. 15:1923;300-304.
54. Wallsten T. S., Bender R. H., Li Y. Dissociating judgment from response processes in statement verification: The effects of experience on each component. Journal of Experimental Psychology: Learning, Memory, and Cognition. 25:1999;96-115.
55. Wallsten T. S., Budescu D. V. Encoding subjective probabilities: A psychological and psychometric review. Management Science. 29:1983;151-173.
56. Wallsten T. S., González-Vallejo C. Statement verification: A stochastic model of judgment and response. Psychological Review. 101:1994;490-504.
57. Wigton R. S. Applications of judgment analysis and cognitive feedback to medicine. Brehmer B., Joyce R. B. Human judgment: The SJT approach. 1988;North-Holland, Amsterdam.
58. Winkler R. L., Murphy A. H. "Good" probability assessors. Journal of Applied Meteorology. 7:1968;751-758.
59. Winman A., Juslin P. Calibration of sensory and cognitive judgments: Two different accounts. Scandinavian Journal of Psychology. 34:1993;135-148.
60. Yaniv I., Yates J. F., Smith J. E. K. Measures of discrimination skill in probabilistic judgment. Psychological Bulletin. 110:1991;611-617.
61. Yates J. F. External correspondence: Decompositions of the mean probability score. Organizational Behavior and Human Performance. 30:1982;132-156.
62. Yates J. F. Judgment and decision making. 1990;Prentice-Hall, Englewood Cliffs.
63. Yates J. F. Subjective probability accuracy analysis. Wright G., Ayton P. Subjective probability. 1994;381-410 Wiley, Chichester.
64. Yates J. F., Lee J.-W., Shinotsuka H., Patalano A. L., Sieck W. R. Cross-cultural variations in probability judgment accuracy: Beyond general knowledge overconfidence? Organizational Behavior and Human Decision Processes. 74:1998;89-117.
65. Yates J. F., Purdy G. N., Potts P. R. Probability analyzer [computer program]. 1994;University of MichiganDepartment of Psychology & Office of Instructional Technology, Ann Arbor.
66. Yates J. F., Zhu Y., Ronis D. L., Wang D.-F., Shinotsuka H., Toda M. Probability judgment accuracy: China, Japan, and the United States. Organizational Behavior and Human Decision Processes. 43:1989;147-171.