Calibration of subjective probability judgments in a naturalistic setting

Organizational Behavior and Human Decision Processes

Volumn 85, Issue 2, 2001, Pages 265-290

  Johnson, Johnnie E V ^a   Bruce, Alistair C ^b  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^b UNIVERSITY OF NOTTINGHAM   (United Kingdom)

Author keywords

Betting; Calibration; Decision making; Judgment; Probability

Indexed keywords

EID: 0345981657     PISSN: 07495978     EISSN: None     Source Type: Journal    
DOI: 10.1006/obhd.2000.2949     Document Type: Article

References (106)

1. Ali M. M. Some evidence of the efficiency of a speculative market. Econometrica. 47:1979;387-392.
2. Anderson G., Brown R. I. F. Real and laboratory gambling, sensation-seeking, and arousal. British Journal of Psychology. 75:1984;401-410.
3. Asch P., Malkiel B. G., Quandt R. E. Market efficiency in racetrack betting. Journal of Business. 57:1984;165-175.
4. Ashton R. H. Effects of justification and a mechanical aid on judgment performance. Organizational Behavior and Human Decision Processes. 52:1992;292-306.
5. Ayton P., Wright G. Subjective probability: what should we believe? Wright G., Ayton P. Subjective probability. 1994;163-183 Wiley, Chichester.
6. Bacon-Shone, J. H, Lo, V. S. Y, &, Busche, K. (1992), Modelling winning probability. (Research report), Department of Statistics, University of Hong Kong, 10.
7. Balthasar H. U., Boschi R. A. A., Menke M. M. Calling the shots in R & D. Harvard Business Review. 1978;151-160.
8. Baranski J. V., Petrusic W. M. The calibration and resolution of confidence in perceptual judgments. Perception and Psychophysics. 55:1994;412-428.
9. Baranski J. V., Petrusic W. M. On the calibration of knowledge and perception. Canadian Journal of Experimental Psychology. 49:1995;397-407.
10. Bazerman M. H. Judgment in managerial decision-making. 1994;Wiley, Chichester.
11. Beach L. R., Barnes V. E., Christen-Szalanski J. J. J. Beyond heuristics and biases: A contingency model of judgmental forecasting. Journal of Forecasting. 5:1986;143-157.
12. Beach L. R., Christensen-Szalanski J., Barnes V. Assessing human judgment: Has it been done, can it be done, should it be done? Wright G., Ayton P. Judgmental forecasting. 1987;49-62 Wiley, Chichester.
13. Beach L. R., Lipshitz R. Why classical decision theory is an inappropriate standard for evaluating and aiding most human decision making. Klein G. A., Orasanu J., Calderwood R., Zsambok C. E. Decision making in action: models and methods. 1993;21-35 Ablex, Norwood.
14. Bennett M. J. Heuristics and the weighting of base rate information in diagnostic tasks by nurses. 1980;Monash University.
15. Björkman M. Internal use theory: Calibration and resolution of confidence in general knowledge. Organizational Behavior and Human Decision Processes. 58:1994;386-405.
16. Braun P. A., Yaniv I. A case study of expert judgment: Economists’ probabilities versus base-rate model forecasts. Journal of Behavioral Decision Making. 5:1992;217-231.
17. Brenner L. A., Koehler D. J., Liberman V., Tversky A. Overconfidence in probability and frequency judgments. Organizational Behavior and Human Decision Processes. 65:1995;212-220.
18. Bruce A. C., Johnson J. E. V. Toward an explanation of betting as a leisure pursuit. Leisure Studies. 11:1992;201-218.
19. Bruce A. C., Johnson J. E. V. Costing excitement in leisure betting. Leisure Studies. 14:1995;48-63.
20. Cambridge R. M., Shreckengost R. C. Are you sure? The subjective probability assessment test. 1980;Office of Training, Central Intelligence Agency, Langley.
21. Christensen-Szalanski J. J. J., Beck D. E., Christensen-Szalanski C. M., Koepsell T. D. The effect of journal coverage on physicians’ perception of risk. Journal of Applied Psychology. 68:1983;278-284.
22. Cohen M. S. The naturalistic bases of decision biases. Klein G. A., Orasanu J., Calderwood R., Zsambok C. E. Decision making in action: Model and methods. 1993a;51-99 Ablex, Norwood.
23. Cohen M. S. Three paradigms for viewing decision biases. Klein G. A., Orasanu J., Calderwood R., Zsambok C. E. Decision making in action: Models and methods. 1993b;36-50 Ablex, Norwood.
24. Curley S. P., Yates J. F., Abrams R. A. Psychological sources of ambiguity avoidance. Organizational Behavior and Human Decision Processes. 38:1986;230-256.
25. Cyert R. M., March J. G. A behavioural theory of the firm. 1963;Prentice-Hall, Englewood Cliffs.
26. Detmer D. E., Fryback D. G., Gassner K. Heuristics and biases in medical decision-making. Journal of Medical Education. 53:1978;682-683.
27. Ebbesen E. G., Konecni V. J. On the external validity of decision-making research: What do we know about decisions in the real world? Wallsten T. Cognitive processes in choice and decision behavior. 1980;78-94 Erlbaum, Hillsdale.
28. Einhorn H. J. Expert measurement and mechanical combination. Organizational Behavior and Human Performance. 7:1972;86-106.
29. Eiser J. R., van der Pligt J. Attitudes and decisions. 1988;Routledge, London.
30. Elstein A. S., Shulman L. S., Sprafka S. A. Medicine problem solving: an analysis of clinical reasoning. 1978;Harvard University Press, Cambridge.
31. Erev I., Wallsten T. S., Budescu D. V. Simultaneous over- and under-confidence: the role of error in judgment processes. Psychological Review. 101:1994;519-527.
32. Ferrell W. R. Discrete subjective probabilities and decision analysis: elicitation, calibration and combination. Wright G., Ayton P. Subjective probability. 1994;410-451 Wiley, Chichester.
33. Ferrell W. R., McGoey P. J. A model of calibration for subjective probabilities. Organizational Behavior and Human Performance. 26:1980;32-53.
34. Figlewski S. Subjective information and market efficiency in a betting market. Journal of Political Economy. 87:1979;75-88.
35. Fischhoff B., Bruine de Bruin W. Fifty-fifty = 50%? Journal of Behavioral Decision Making. 12:1999;149-163.
36. Ford J. K., Schmitt N., Schectman S. L., Hills B. M., Doherty M. L. Process tracing methods: contributions, problems and neglected research questions. Organizational Behavior and Human Decision Processes. 43:1989;75-117.
37. Gabriel P. E., Marsden J. R. An examination of market efficiency in British racetrack betting. Journal of Political Economy. 98:1990;874-885.
38. Gigerenzer G., Hell W., Blank H. Presentation and content: the use of base rates as a continuous variable. Journal of Experimental Psychology: Human Perception and Performance. 14:1988;513-525.
39. Gigerenzer G., Hoffrage U., Kleinbolting H. Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review. 98:1991;506-528.
40. Goodwin P., Wright G. Decision analysis for management judgement. 1992;Wiley, Chichester.
41. Griffin D., Tversky A. The weighting of evidence and the determinants of confidence. Cognitive Psychology. 24:1992;411-435.
42. Hammond K. R. Naturalistic decision making from a Brunswikian viewpoint: Its past, present, future. Klein G. A., Orasanu J., Calderwood R., Zsambok C. E. Decision making in action: Models and methods. 1993;205-227 Ablex, Norwood.
43. Hazard, T. H, &, Peterson, C. R. (1973), Odds versus probabilities for categorical events. (Tech. Rep. No. 73-2), McLean, VA, Decisions and Designs, Inc.
44. Hey J. D. Experiments in economics. 1991;Blackwell, Oxford.
45. Hey J. D. Experiments in economics - and psychology. Lee S. E. G., Webley P., Young B. M. New directions in economic psychology - Theory, experiment and application. 1992;85-98 Edward Elgar, Aldershot.
46. Hoerl A. E., Falein H. K. Reliability of subjective evaluation in a high incentive situation. Journal of the Royal Statistical Society. 137:1974;227-230.
47. Hogarth R. M. Decision time as a function of task complexity. Wendt D., Vlek C. Utility, probability and human decision making. 1975;125-142 Reidel, Dordrecht.
48. Hogarth R. M. Beyond static biases: Functional and dysfunctional aspects of judgmental heuristics. 1980;University of Chicago, Graduate School of BusinessCenter for Decision Research, Chicago.
49. Janis J. L., Mann L. Victims of Groupthink. 1972;Houghton-Mifflin, Boston.
50. Janis J. L., Mann L. Decision making: A psychological analysis of conflict, choice and commitment. 1977;Free Press, New York.
51. Johnson J. E. V., Bruce A. C. Risk strategy under task complexity: A multivariate analysis of behavior in a naturalistic setting. Journal of Behavioral Decision Making. 11:1998;1-17.
52. Joyce E. J., Biddle G. C. Anchoring and adjustment in probabilistic inference in auditing. Journal of Accounting Research. 19:1981;120-145.
53. Juslin P. An explanation of the hard-easy effect in studies of realism of confidence in one’s general knowledge. European Journal of Cognitive Psychology. 5:1993;55-71.
54. Juslin P. The overconfidence phenomenon as a consequence of informal experimenter-guided selection of almanac items. Organizational Behavior and Human Decision Processes. 57:1994;226-246.
55. Kabus I. You can bank on uncertainty. Harvard Business Review. 1976;95-105.
56. Kahneman D., Solvic P., Tversky A. Judgment under uncertainty: Heuristics and biases. 1982;Cambridge University Press, Cambridge.
57. Kahneman D., Tversky A. Subjective probability: a judgement of representativeness. Cognitive Psychology. 3:1972;430-454.
58. Kahneman D., Tversky A. Prospect theory: an analysis of decisions under risk. Econometrica. 47:1979;263-291.
59. Keren G. Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica. 77:1987;217-273.
60. Keren G. Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica. 77:1991;217-273.
61. Keren G., Wagenaar W. A. Violation of utility theory in unique and repeated gambles. Journal of Experimental Psychology: Learning, Memory, and Cognition. 13:1987;387-391.
62. Keren G. On the calibration of probability judgments: some critical comments and alternative perspectives. Journal of Behavioral Decision Making. 10:1997;269-278.
63. Klein G. A. Recognition primed decisions. Rouse W. B. Advances in man-machine system research. 5:1989;47-92 JAI Press, Greenwich.
64. Koriat A., Lichtenstein S., Fischhoff B. Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory. 6:1980;107-118.
65. Lichtenstein S., Fischhoff B. Do those who know also know more about how much they know? Organizational Behavior and Human Performance. 20:1977;159-183.
66. Lichtenstein S., Fischhoff B., Phillips L. D. Calibration of probabilities: The state of the art. Jungermann H., De Zeeuw G. Decision making and change in human affairs. 1977;79-88 Reidel, Amsterdam.
67. Lichtensein S., Fischhoff B., Phillips L. D. Calibration of probabilities: The state of the art to 1980. Kahneman P., Slovic P., Tversky A. Judgement under uncertainty: Heuristics and biases. 1982;306-334 Cambridge University Press, Cambridge.
68. Lichtenstein S., Slovic P. Reversals of preference between bids and choices in gambling decisions. Journal of Experimental Psychology. 89:1971;46-55.
69. Lindsay P. H., Norman D. A. Human information processing. 1972;Academic Press, New York.
70. Lipshitz R. Converging themes in the study of decision making in realistic settings. Klein G. A., Orasanu J., Calderwood R., Zsambok C. E. Decision making in action: Models and methods. 1993;103-137 Ablex, Norwood.
71. Lock A. Integrating group judgments in subjective forecasts. Wright G., Ayton P. Judgmental forecasting. 1987;109-127 Wiley, Chichester.
72. Lopes L. L. Decision making in the short run. Journal of Experimental Psychology: Human Perception and Performance. 10:1981;465-485.
73. Lopes L. L. When time is of the essence: averaging, aspiration and the short run. Organizational Behavior and Human Decision Processes. 65:1996;179-189.
74. Lusted L. B. A study of the efficiency of diagnostic radiologic procedures: Final report on diagnostic efficiency. 1977;Efficiency Study Committee of the American College of Radiology, Chicago.
75. Malhotra N. K. Information load and consumer decision making. Journal of Consumer Research. 8:1982;419-430.
76. McClelland A. G. R., Bolger F. The calibration of subjective probabilities: Theories and models 1980-94. Wright G., Ayton P. Subjective probability. 1994;453-482 Wiley, Chichester.
77. McFadden D. Conditional logit analysis of qualitative choice behaviour. Zarembka P. Frontiers in econometrics: Economic theory and mathematical economics. 1974;105-142 Academic Press, New York.
78. Murphy A. H., Brown B. G. A Comparative evaluation of objective and subjective weather forecasts in the United States. Wright G. Behavioral decision making. 1985;178-193 Plenum, New York.
79. Murphy A. H., Winkler R. L. Can weather forecasters formulate reliable forecasts of precipitation and temperature? National Weather Digest. 2:1977;2-9.
80. Murphy A. H., Winkler R. L. Probability forecasting in meteorology. Journal of the American Statistical Association. 79:1984;489-500.
81. Neal M. ’You lucky punters!’ A Study of gambling in betting shops. Sociology. 32:1998;581-600.
82. Northcraft G. B., Neale M. A. Experts, amateurs and real estate: an anchoring-and-adjustment perspective on property pricing decisions. Organizational Behavior and Human Decision Processes. 39:1987;84-97.
83. Omodei M. M., Wearing A. J. Decision-making in complex dynamic settings - a theoretical model incorporating motivation, intention, affect and cognitive performance. Sprache & Kognition. 14:1995;75-90.
84. Orasanu J., Connolly T. The reinvention of decision making. Klein G. A., Orasanu J., Calderwood R., Zsambok C. E. Decision making in action: Models and methods. 1993;302 Ablex, Norwood.
85. Phillips L. D. On the adequacy of judgmental forecasts. Wright G., Ayton P. Judgmental forecasting. 1987;11-30 Wiley, Chichester.
86. Pitz G. F. Decision making and cognition. Jungerman H., de Zeeuw G. Decision making in human affairs. 1977;136-145 Reidel, Amsterdam.
87. Rosecrance J. Professional horse racing gambling: Working without a safety net. Work and Occupations. 15:1988;220-236.
88. Russo J. E., Shoemaker P. J. H. Decision traps. 1989;Doubleday, New York.
89. Saunders D. M., Turner D. E. Gambling and leisure: The case of racing. Leisure Studies. 6:1987;281-289.
90. Shanteau J. Competence in experts: the role of task characteristics. Organizational Behavior and Human Decision Processes. 53:1992;252-266.
91. Smith J. F., Kida T. Heuristics and biases: Expertise and task realism in auditing. Psychological Bulletin. 109:1991;472-485.
92. Smith V. D. Theory, experiment and economics. Journal of Economic Perspectives. 3:1989;151-169.
93. Sniezek J. A., Paese P. W., Switzer F. S. I. The effect of choosing on confidence in choice. Organizational Behavior and Human Decision Processes. 46:1990;264-282.
94. Snyder W. Horse-racing: the efficient markets model. Journal of Finance. 78:1978;1109-1118.
95. Stratigakos S. Sensory calibration. 1994;University of ArizonaSystems and Industrial Engineering Dept, Tucson.
96. Suantek L., Bolger F., Ferrell W. R. The hard-easy effect in subjective probability calibration. Organizational Behavior and Human Decision Processes. 67:1996;201-221.
97. Timmermans D. The impact of task complexity on information use in multi-attribute decision making. Journal of Behavioral Decision Making. 6:1993;95-111.
98. Tomassini L., Solomon I., Romney M., Krogstad J. Calibration of auditors’ probabilistic judgments: some empirical evidence. Organizational Behavior and Human Performance. 30:1982;391-406.
99. Tversky A., Kahneman D. Judgement under uncertainty: Heuristics and biases. Science. 185:1974;1124-1131.
100. von Winterfeldt D., Edwards W. Decision analysis and behavioral research. 1986;Cambridge University Press, Cambridge.
101. Waller W. S., Shapiro B., Sevcik G. Do cost-based pricing biases persist in laboratory markets. Accounting, Organizations and Society. 24:1999;717-739.
102. Weddell, D. H., & Bockenholt, U.Moderation of preference reversals in the long run. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, 429-438.
103. Wright G. Changes in the realism and distribution of probability assessment as a function of question type. Acta Psychologica. 52:1982;165-174.
104. Wright G., Ayton P. Immediate and short-term judgmental forecasting: Personoligism, Situationism or Interactionism? Personality and Individual Differences. 9:1988;109-120.
105. Wright P. Consumer choice strategies: Simplifying vs optimising. Journal of Marketing Research. 11:1975;60-67.
106. Zakay D. The relationship between the probability assessor and the outcomes of an event as a determiner of subjective probability. Acta Psychologica. 53:1983;271-280.