Forecast aggregation via recalibration

Machine Learning

Volumn 95, Issue 3, 2014, Pages 261-289

  Turner, Brandon M ^a   Steyvers, Mark ^b   Merkle, Edgar C ^c   Budescu, David V ^d   Wallsten, Thomas S ^e  

^a STANFORD UNIVERSITY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF MISSOURI   (United States)

^d FORDHAM UNIVERSITY   (United States)

^e UNIVERSITY OF MARYLAND   (United States)

Author keywords

Aggregation; Calibration; Forecasting; Hierarchical Bayesian models; Individual differences; Systematic distortions; Wisdom of the crowd

Indexed keywords

AGGLOMERATION; BAYESIAN NETWORKS; CALIBRATION; HIERARCHICAL SYSTEMS; SOCIAL SCIENCES COMPUTING;

BETTER PERFORMANCE; CALIBRATION FUNCTIONS; FORECAST PERFORMANCE; HIERARCHICAL BAYESIAN MODELS; HIERARCHICAL MODEL; INDIVIDUAL DIFFERENCES; SYSTEMATIC BIAS; WISDOM OF THE CROWDS;

FORECASTING;

EID: 84901498395     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-013-5401-4     Document Type: Article

References (74)

1. Ariely, D., Au, W. T., Bender, R. H., Budescu, D. V., Dietz, C. B., Gu, H., Wallsten, T. S., & Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology. Applied, 6, 130-147.
2. Arkes, H. R., Dawson, N. V., Speroff, T., Harrell, F. E. Jr., Alzola, C., Phillips, R., Desbiens, N., Oye, R. K., Knaus, W., & Connors, A. F. Jr. (1995). The covariance decomposition of the probability score and its use in evaluating prognostic estimates. Medical Decision Making, 15, 120-131.
3. Armstrong, J. S. (2001). Principles of forecasting. Norwell: Kluwer Academic.
4. Birnbaum, M. H., & McIntosh, W. R. (1996). Violations of branch independence in choices between gambles. Organizational Behavior and Human Decision Processes, 67, 91-110.
5. Breiman, L. (1996). Bagging predictors. Machine Learning, 24, 123-140.
6. Brenner, L. A., Koehler, D. J., Liberman, V., & Tversky, A. (1996). Overconfidence in probability and frequency judgments: a critical examination. Organizational Behavioral and Human Decision Processes, 65, 212-219.
7. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1-3.
8. Broomell, S. B., Budescu, D. V., & Por, H. H. (2011). Pair-wise comparison of multiple models. Judgement and Decision Making, 6, 821-831.
9. Brown, S., & Steyvers, M. (2009). Detecting and predicting changes. Cognitive Psychology, 58, 49-67.
10. Budescu, D. V., & Johnson, T. R. (2011). A model-based approach for the analysis of the calibration of probability judgments. Judgment and Decision Making, 6, 857-869.
11. Buntine, W. L. (1994). Operations for learning with graphical models. Journal of Artificial Intelligence Research, 2, 159-225.
12. Camerer, C. F., & Ho, T. H. (1994). Violations of the betweenness axiom and nonlinearity in probability. Journal of Risk and Uncertainty, 8, 167-196.
13. Christensen, R., Johnson, W., Branscum, A., & Hanson, T. E. (2011). Bayesian ideas and data analysis: an introduction for scientists and statisticians. Boca Raton: CRC Press.
14. Christensen-Szalanski, J. J. J., & Bushyhead, J. B. (1981). Physicians' use of probabilistic information in a real clinical setting. Journal of Experimental Psychology. Human Perception and Performance, 7, 928-935.
15. Clemen, R. T. (1986). Calibration and the aggregation of probabilities. Management Science, 32, 312-314.
16. Clemen, R. T. (1989). Combining forecasts: a review and annotated bibliography (with discussion). International Journal of Forecasting, 5, 559-583.
17. Clemen, R. T., & Winkler, R. L. (1986). Combining economic forecasts. Journal of Business & Economic Statistics, 4, 39-46.
18. Cooke, R. M. (1991). Experts in uncertainty: opinion and subjective probability in science. New York: Oxford University Press.
19. Erev, I., Wallsten, T. S., & Budescu, D. V. (1994). Simultaneous over- and underconfidence: The role of error in judgment processes. Psychological Review, 101, 519-527.
20. Ferrell, W. R., & McGoey, P. J. (1980). A model of calibration for subjective probabilities. Organizational Behavior and Human Decision Processes, 26, 32-53.
21. Freund, Y., & Schapire, R. E. (1996). Experiments with a new boosting algorithm. In Machine learning: Proceedings of the thirteenth international conference (pp. 325-332). San Francisco: Morgan Kauffman.
22. Fryback, D. G., & Erdman, H. (1979). Prospects for calibrating physicians' probabilistic judgments: design of a feedback system. In Proceedings of the IEEE international conference on cybernetics and society.
23. Galton, F. (1907). Vox populi. Nature, 75, 450-451.
24. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2004). Bayesian data analysis. New York: Chapman and Hall.
25. Gonzalez, R., & Wu, G. (1999). On the shape of the probability weighting function. Cognitive Psychology, 38, 129-166.
26. Griffin, D., & Tversky, A. (1992). The weighing of evidence and the determinants of confidence. Cognitive Psychology, 24, 411-435.
27. Härdle, W. (1992). Applied nonparametric regression. Cambridge: Cambridge University Press.
28. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning (2nd ed.). New York: Springer.
29. Hemmer, P., Steyvers, M., & Miller, B. J. (2011). The wisdom of crowds with informative priors. In Proceedings of the 32rd annual conference of the cognitive science society. Lawrence Erlbaum.
30. Hora, S. C. (2004). Probability judgments for continuous quantities: Linear combinations and calibration. Management Science, 50, 597-604.
31. Jang, Y., Wallsten, T. S., & Huber, D. E. (2012). A stochastic detection and retrieval model for the study of metacognition. Psychological Review, 119, 186-200.
32. Juslin, P., & Olsson, H. (1997). Thurstonian and Brunswikian origins of uncertainty in judgment: a sampling model of confidence in sensory discrimination. Psychological Review, 104, 344-366.
33. Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263-291.
34. Karmarker, U. S. (1978). Subjectively weighted utility: a descriptive extension of the expected utility model. Organizational Behavior and Human Performance, 21, 61-72.
35. Keren, G. (1991). Calibration and Probability Judgments: Conceptual and Methodological Issues. Acta Psychologica.
36. Lee, M. D. (2008). Three case studies in the Bayesian analysis of cognitive models. Psychonomic Bulletin & Review, 15, 1-15.
37. Lee, M. D., Steyvers, M., de Young, M., & Miller, B. J. (2011). A model-based approach to measuring expertise in ranking tasks. In Proceedings of the 33rd annual conference of the cognitive science society. Cognitive Science Society.
38. Lee, M. D., & Wagenmakers, E.-J. (2012). A course in Bayesian graphical modeling for cognitive science. Available from http://www. ejwagenmakers.com/BayesCourse/BayesBookWeb.pdf; last downloaded January 1, 2012.
39. Lichtenstein, S., Fischoff, B., & Phillips, L. D. (1982). Calibration of probabilities: the state of the art to 1980. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: heuristics and biases (pp. 306-334). Cambridge: Cambridge University Press.
40. Merkle, E. C. (2010). Calibrating subjective probabilities using hierarchical Bayesian models. In S.-K. Chai, J. J. Salerno, & P. L. Mabry (Eds.), Lecture notes in computer science: Vol. 6007. Social computing, behavioral modeling, and prediction (SBP) 2010 (pp. 13-22).
41. Merkle, E. C., Smithson, M., & Verkuilen, J. (2011). Hierarchical models of simple mechanisms underlying confidence in decision making. Journal of Mathematical Psychology, 55, 57-67.
42. Merkle, E. C., & Van Zandt, T. (2006). An application of the Poisson race model to confidence calibration. Journal of Experimental Psychology. General, 135, 391-408.
43. Miller, B. J., Hemmer, P., Steyvers, M., & Lee, M. D. (2009). The wisdom of crowds in ordering problems. In Proceedings of the ninth international conference on cognitive modeling, Manchester, UK.
44. Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12, 595-600.
45. Murphy, A. H., & Winkler, R. L. (1974). Credible interval temperature forecasting: some experimental results. Monthly Weather Review, 102, 784-794.
46. Murphy, A. H., & Winkler, R. L. (1984). Probability forecasting in meteorology. Journal of the American Statistical Association, 79, 489-500.
47. O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006). Uncertain judgements: eliciting experts' probabilities. Hoboken: Wiley.
48. Page, L., & Clemen, R. T. (2012). Do prediction markets produce well calibrated probability forecasts? Manuscript in preparation.
49. Pleskac, T. J., & Busemeyer, J. R. (2010). Two-stage dynamic signal detection theory: a theory of choice, decision time, and confidence. Psychological Review, 117, 864-901.
50. Plummer, M. (2003). JAGS: a program for analysis of Bayesian graphical models using Gibbs sampling. In Proceedings of the 3rd international workshop on distributed statistical computing.
51. Shiffrin, R. M., Lee, M. D., Kim, W., & Wagenmakers, E.-J. (2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32, 1248-1284.
52. Shlomi, Y., & Wallsten, T. S. (2010). Subjective recalibration of advisors' probability estimates. Psychonomic Bulletin & Review, 17, 492-498.
53. Silverman, B. W. (1986). Density estimation for statistics and data analysis. London: Chapman & Hall.
54. Soll, J. B. (1996). Determinants of overconfidence and miscalibration: the roles of random error and ecological structure. Organizational Behavior and Human Decision Processes, 65, 117-137.
55. Staël von Holstein, C. A. S. (1970). Measurement of subjective probability. Acta Psychologica, 34, 146-159.
56. Steyvers, M., Lee, M. D., & Miller, B. J. (2009). Wisdom of crowds in the recollection of order information. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, & A. Culotta (Eds.), Advances in neural information processing systems. (Vol. 22, pp. 1785-1793). New York: MIT Press.
57. Suantak, L., Bolger, F., & Ferrell, W. R. (1996). The hard-easy effect in subjective probability calibration. Organizational Behavior and Human Decision Processes, 67, 201-221.
58. Surowiecki, J. (2004). The wisdom of crowds. New York: Random House.
59. Thomas, R. P., Dougherty, M. R., Sprenger, A. M., & Harbison, J. I. (2008). Diagnostic hypothesis generation and human judgment. Psychological Review, 115, 155-185.
60. Tversky, A., & Fox, C. R. (1995). Weighing risk and uncertainty. Psychological Review, 102, 269-283.
61. Tversky, A., & Kahneman, D. (1992). Advances in prospect theory: cumulative representations of uncertainty. Journal of Risk and Uncertainty, 5, 297-323.
62. Van Zandt, T. (2000). ROC curves and confidence judgments in recognition memory. Journal of Experimental Psychology. Learning, Memory, and Cognition, 26, 582-600.
63. Wallsten, T. S., & Budescu, D. V. (1983). Encoding subjective probabilities: a psychological and psychometric review. Management Science, 29, 151-173.
64. Wallsten, T. S., Budescu, D. V., & Erev, I. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10, 243-268.
65. Wallsten, T. S., & González-Vallejo, C. (1994). Statement verification: a stochastic model of judgment and response. Psychological Review, 101, 490-504.
66. Warnaar, D., Merkle, E., Steyvers, S., Wallsten, T., Stone, E., Budescu, D., Yates, J., Sieck, W., Arkes, H., Argenta, C., Shin, Y., & Carter, J. (2012). The aggregative contingent estimation system: selecting, rewarding, and training experts in a wisdom of crowds approach to forecasting. In 2012 AAAI symposium. on Wisdom of the Crowd.
67. Wright, G. (1982). Changes in the realism and distribution of probability assessments as a function of question type. Acta Psychologica, 52, 165-174.
68. Wright, G., & Wisudha, A. (1982). Distribution of probability assessments for almanac and future event questions. Scandinavian Journal of Psychology, 23, 219-224.
69. Wu, G., & Gonzalez, R. (1996). Curvature of the probability weight function. Management Science, 42, 1676-1690.
70. Yates, J. F. (1982). External correspondence: decompositions of the mean probability score. Organizational Behavior and Human Performance, 30, 132-156.
71. Yates, J. F. (1990). Judgment and decision making. Englewood Cliffs: Prentice Hall.
72. Yi, S. K. M., Steyvers, M., & Lee, M. D. (2011). The wisdom of crowds in combinatorial problems. Cognitive Science.
73. Yi, S. K. M., Steyvers, M., Lee, M. D., & Dry, M. (2010). Wisdom of crowds in minimum spanning tree problems. In Proceedings of the 32nd annual conference of the cognitive science society. Lawrence Erlbaum.
74. Zhang, H., & Maloney, L. T. (2011). Ubiquitous log odds: a common representation of probability and frequency distortion in perception, action, and cognition. Frontiers in Neuroscience, 6, 1-14.