Improving calibration without training: The role of task information

Applied Cognitive Psychology

Volumn 17, Issue 4, 2003, Pages 419-441

  Rakow, Tim ^a   Harvey, Nigel ^b   Finer, Sarah ^b  

^a UNIVERSITY OF ESSEX   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; CALIBRATION; COMPUTER GRAPHICS; CONTROLLED STUDY; DECISION MAKING; EXPERIMENTAL DESIGN; FEMALE; FORECASTING; HUMAN; HUMAN EXPERIMENT; MALE; MATHEMATICAL COMPUTING; MEDICAL SCHOOL; MEDICAL STUDENT; POPULATION; PRIORITY JOURNAL; PROBABILITY; STATISTICAL MODEL; TASK PERFORMANCE; TRAINING;

EID: 8744229928     PISSN: 08884080     EISSN: None     Source Type: Journal    
DOI: 10.1002/acp.878     Document Type: Article

References (76)

1. Anderson NH. 1974. Information integration theory: a brief survey. In Contemporary Developments in Mathematical Psychology (Vol. 2), Krantz D, Atkinson RC, Luce RD, Suppes P (eds). WH Freeman: San Francisco; 237-305.
2. Angus-Leppan P, Fatseas V. 1986. The forecasting accuracy of trainee accountants using judge-mental and statistical techniques. Accounting and Business Research 16: 179-188.
3. Argote L, Devadas A, Malone N. 1990. The base-rate fallacy: contrasting process and outcomes of groups and individual judgement. Organizational Behavior and Human Decision Processes 46: 296-310.
4. Argote L, Seabright MA, Dyer L. 1986. Individual versus group use of base-rate and individuating information. Organizational Behavior and Human Decision Processes 38: 65-75.
5. Balzer WK, Doherty ME, O'Connor R, Jr. 1989. Effects of cognitive feedback on performance. Psychological Bulletin 106: 410-433.
6. Balzer WK, Hammer LB, Sumner KE, Birchenough TR, Martens SP, Raymark PH. 1994. Effects of cognitive feedback components, display format, and elaboration of performance. Organizational Behavior and Human Decision Processes 58: 369-385.
7. Bar-Hillel M. 1980. The base-rate fallacy in probability judgements. Acta Psychologica 44: 211-213.
8. Bar-Hillel M. 1990. Back to base rates. In Insights in Decision Making: A Tribute to Hillel J. Einhorn, Hogarth RM (ed.). Chicago University Press: Chicago; 200-216.
9. Beach LR, Braun GP. 1994. Laboratory studies of subjective probability: a status report. In Subjective Probability, Wright G, Ayton P (eds). John Wiley: Chichester.
10. Benson PG, Önkal D. 1992. The effects of feedback and training on the performance of probability forecasters. International Journal of Forecasting 8: 559-573.
11. Björkman M. 1972. Feedforward and feedback as determiners of knowledge and policy: notes on a neglected issue. Scandinavian Journal of Psychology 13: 152-158.
12. Braun PA, Yaniv I. 1992. A case study of expert judgement: economists probabilities versus base-rate model forecasts. Journal of Behavioral Decision Making 5: 217-231.
13. Brenner L, Koehler D, Liberman V, Tversky A. 1996. Overconfidence in probability and frequency judgements: a critical examination. Organizational Behavior and Human Decision Processes 56: 212-219.
14. Brier GW. 1950. Verification of forecasts expressed in terms of a probability. Monthly Weather Review 78(1): 1-3.
15. Brunswik E. 1952. The Conceptual Framework of Psychology University of Chicago Press: Chicago.
16. Christensen-Szalanski JJJ, Busheyhead JB. 1981. Physicians' use of probabilistic information in a real clinical setting. Journal of Experimental Psychology: Human Perception and Performance 7 4): 928-935.
17. Coll R, Thyagarajan A, Chopra S. 1991. An experimental study comparing the effectiveness of computer graphics data versus computer tabular data. IEEE Transactions in Systems, Man and Cybernetics 21: 897-900.
18. De Sanctis G. 1984. Computer graphics as design aids: directions for research. Decision Sciences 15: 463-487.
19. Deane DH, Hammond KR, Summers DA. 1972. Journal of Experimental Psychology 72: 20-26.
20. Dickson G. 1995. Principles of risk management. In Clinical Risk Management, Vincent C (ed.). BMA: London; 18-30.
21. Dickson GW, De Sanctis G, McBride DJ. 1986. Understanding the effectiveness of computer graphics for decision support: a cumulative experimental approach. Communications of the ACM 29: 40-47.
22. Doherty M, Balzer WK. 1988. Cognitive feedback. In Human Judgement: The SJT View, Brehmer B, Joyce CRB (eds). North-Holland: Amsterdam; 163-197.
23. Doise W. 1969. Intergroup relations and polarization in individual and collective judgements. Journal of Personality and Social Psychology 12: 136-143.
24. Ferrell WR. 1994. Discrete subjective probabilities and decision analysis: elicitation, calibration and combination. In Subjective Probability, Wright G, Ayton P (eds). John Wiley: Chichester; 411-451.
25. Ferrell WR, McGoey PJ. 1980. A model of calibration for subjective probabilities. Organizational Behavior and Human Decision Processes 26: 32-53.
26. Fischer GW. 1982. Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting. Organizational Behavior and Human Decision Processes 29: 352-369.
27. Fischhoff B. 1989. Eliciting knowledge for analytical representation. IEEE Transactions on Systems, Man and Cybernetics 19: 448-61.
28. Fischhoff B, Slovic P. 1980. A little learning ... : confidence in multicue judgement. In Attention and Performance VIII Nickerson R (ed.). Earlbaum: Hillsdale, NJ.
29. Fischhoff B, Slovic P, Lichtenstein S. 1977. Knowing with certainty: the appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance 3: 552-564.
30. Fischhoff B, Slovic P, Lichtenstein S. 2000. Weighing the risks: which risks are acceptable? In The Perception of Risk, Slovic P (ed.). Earthscan: London; 121-136.
31. Griffin D, Buehler R. 1999. Frequency, probability, and prediction: easy solutions to cognitive illusions? Cognitive Psychology 38: 48-78.
32. Griffin D, Tversky A. 1992. The weighting of evidence and the determinants of confidence. Cognitive Psychology 24: 411-435.
33. Hammond KR. 1955. Probabilistic functionalism and the clinical method. Psychological Review 62: 255-262.
34. Hammond KR. 1971. Computer graphics as an aid to learning. Science 172: 903-908.
35. Hammond KR, Boyle PJR. 1971. Quasi-rationality, quarrels, and new conceptions of feedback. Bulletin of the British Psychological Society 24: 103-113.
36. Harvey N. 1997. Confidence in judgement. Trends in Cognitive Sciences 1: 78-82.
37. Harvey N, Bolger F. 1996. Graphs versus tables: effects of data presentation format on judgemental forecasting. International Journal of Forecasting 12: 119-137.
38. Hastie R, Dawes RM. 2001. Rational Choice in an Uncertain World: The Psychology of Judgement and Decision Making (2nd edn). Sage: Thousand Oaks, CA.
39. Jarvenpaa SL. 1990. Graphic displays in decision making - the visual salience effect. Journal of Behavioral Decision Making 3: 247-262.
40. Keeney RL, Raiffa H. 1976. Decisions with Multiple Objectives: Preferences and Value Tradeoffs. John Wiley: New York.
41. Keren G. 1988. On the ability of monitoring non-veridical perceptions and uncertain knowledge: some calibration studies. Acta Psychologica 67: 95-119.
42. Keren G. 1991. Calibration and probability judgements: conceptual and methodological issues. Acta Psychologica 77: 217-273.
43. Knowles BA, Hammond KR, Stewart TR, Summers DA. 1971. Positive and negative redundancy in multiple cue probability tasks. Journal of Experimental Psychology 90: 157-159.
44. Koehler JJ. 1996. The base rate fallacy reconsidered: descriptive, normative, and methodological challenges. Behavioral and Brain Sciences 19: 1-53.
45. Koriat A, Lichtenstein S, Fischoff B. 1980. Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory 6: 107-18.
46. Lawrence MJ. 1983. An exploration of some practical issues in the use of quantitative forecasting models. Journal of Forecasting 2: 169-179.
47. Lawrence MJ, Edmundson RH, O'Connor M. 1985. An examination of the accuracy of judgemental extrapolation of time series. International Journal of Forecasting 1: 25-36.
48. Lichtenstein S, Fischhoff B. 1980. Training for calibration. Organizational Behavior and Human Decision Processes 26: 149-171.
49. Lichtenstein S, Fischhoff B, Phillips LD. 1977. Calibration of probabilities: the state-of-the-art. In Decision Making and Change in Human Affairs, Jungerman H, de Zeeuw G (eds). D. Reidel: Amsterdam.
50. Lichtenstein S, Fischhoff B, Phillips LD. 1982. Calibration of probabilities: the state of the art to 1980. In Judgement Under Uncertainty: Heuristics and Biases, Kahneman D, Slovic P, Tversky A (eds). Cambridge University Press: Cambridge.
51. McClelland AGR, Bolger F. 1994. The calibration of subjective probabilities: theories and models 1980-94. In Subjective Probability Wright G, Ayton P (eds). John Wiley: Chichester; 453-482.
52. McClish DK, Powell SH. 1989. How well can physicians estimate mortality in a medical intensive care unit? Medical Decision Making 9: 125-132.
53. Moscovici S, Zavalloni M. 1969. The group as a polariser of attitudes. Journal of Personality and Social Psychology 12: 125-135.
54. Murphy AH, Daan H. 1984. Impacts of feedback and experience on the quality of subjective probability forecasts: comparison of results from the first and second years of the Zierikzee experiment. Monthly Weather Review 112: 413-425.
55. Raiffa H. 1968. Decision Analysis: Introductory Lectures on Choice Under Uncertainty. Addison-Wesley: New York.
56. Remus W. 1987. A study of graphical and tabular displays and their interaction with environmental complexity. Management Science 33: 1200-1204.
57. Russo JE, Schoemaker PJH. 1992. Managing overconfidence. Sloan Management Review Winter: 7-17.
58. Schmidt RA. 1988. Motor Control and Learning (2nd edn). Human Kinetics Publishers: Champaign, IL.
59. Sieber JE. 1974. Effects of decision importance on ability to generate warranted subjective certainty. Journal of Personality and Social Psychology 30: 688-694.
60. Smith EE, Medin DL. 1981. Categories and Concepts. Harvard University Press: Cambridge, MA.
61. Smith M, Ferrell WR. 1983. The effect of base rate on calibration of subjective probability for true-false questions: model and experiment. In Analyzing and Aiding Decisions, Humphreys P, Svenson O, Vari A (eds). North-Holland: Amsterdam; 469-488.
62. Sniezek JA, Buckley T. 1993. Becoming more or less uncertain. In Individual and Group Decision Making: Current Issues, Castellan NJ, Jr (ed.). Erlbaum: Hillsdale, NJ; 87-108.
63. Sniezek JA, Henry RA. 1990. Revision, weighting, and commitment in consensus group judgement. Organizational Behavior and Human Decision Processes 45: 66-84.
64. Staël von Holstein C-AS. 1972. Probabilistic forecasting: an experiment related to the stock market. Organizational Behavior and Human Performance 8: 139-158.
65. Stewart TR, Moninger WR, Grassia J, Brady RH, Merrem FH. 1989. Analysis of expert judgement in a hail forecasting experiment. Weather and Forecasting 4: 24-34.
66. Swets JA. 1988. Measuring the accuracy of diagnostic systems. Science 240: 1285-1293.
67. Tindale RS. 1993. Decision errors made by individuals and groups. In Individual and Group Decision Making: Current Issues, Castellan NJ, Jr (ed.). Erlbaum: Hillsdale, NJ; 109-124.
68. Tomassini LA, Soloman I, Ramsey MB. 1982. Calibration of auditors' probabilistic judgements: some empirical evidence. Organizational Behavior and Human Performance 30: 391-406.
69. Tullis TS. 1988. Screen design. In Human-computer Interaction, Helander M (ed.). North-Holland: Amsterdam; 367-411.
70. Tversky A, Kahneman D. 1974. Judgement under uncertainty: heuristics and biases. Science 185: 1124-1131.
71. Wallsten TS, Bender RH, Li Y. 1999. Dissociating judgement from response processes in statement verification: the effects of experience on each component. Journal of Experimental Psychology: Learning, Memory and Cognition 25: 96-115.
72. Wilkie ME, Pollock AC. 1996. An application of probability judgement accuracy measures to currency forecasting. International Journal of Forecasting 12: 25-40.
73. Yates JF. 1982. External correspondence: decompositions of the mean probability score.Organizational Behavior and Human Performance 30: 132-156.
74. Yates JF. 1990. Judgement and Decision Making. Prentice Hall: Englewood Cliffs, NJ.
75. Yates JF. 1994. Subjective probability accuracy analysis. In Subjective Probability, Wright G, Ayton P (eds). John Wiley: Chichester; 381-410.
76. Zahay D. 1992. The influence of computerized feedback on overconfidence in knowledge. Behavior and Information Technology 11: 324-333.