Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs

Annals of Applied Statistics

Volumn 8, Issue 2, 2014, Pages 1256-1280

  Satopää, Ville A ^a   Jensen, Shane T ^a   Mellers, Barbara A ^b   Tetlock, Philip E ^b   Ungar, Lyle H ^c  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

^c UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Bias estimation; Calibration; Dynamic linear model; Expert forecast; Hierarchical modeling; Probability aggregation; Subjective probability; Time series

Indexed keywords

EID: 84903758799     PISSN: 19326157     EISSN: 19417330     Source Type: Journal    
DOI: 10.1214/14-AOAS739     Document Type: Article

References (55)

1. ALLARD, D., COMUNIAN, A. and RENARD, P. (2012). Probability aggregation methods in geoscience. Math. Geosci. 44 545-581. MR2947804
2. ARIELY, D., AU, W. T., BENDER, R. H., BUDESCU, D. V., DIETZ, C. B., GU, H., WALLSTEN, T. S. and ZAUBERMAN, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied 6 130-147.
3. BAARS, J. A. and MASS, C. F. (2005). Performance of national weather service forecasts compared to operational, consensus, and weighted model output statistics. Weather and Forecasting 20 1034-1047.
4. BARON, J., MELLERS, B. A., TETLOCK, P. E., STONE, E. and UNGAR, L. H. (2014). Two reasons to make aggregated probability forecasts more extreme. Decis. Anal. 11. DOI:10.1287/deca.2014.0293.
5. BATCHELDER, W. H., STRASHNY, A. and ROMNEY, A. K. (2010). Cultural consensus theory: Aggregating continuous responses in a finite interval. In Advances in Social Computing (S.-K. Chaim, J. J. Salerno and P. L. Mabry, eds.) 98-107. Springer, Berlin.
6. BIER, V. (2004). Implications of the research on expert overconfidence and dependence. Reliability Engineering & System Safety 85 321-329.
7. BONFERRONI, C. E. (1936). Teoria Statistica Delle Classi e Calcolo Delle Probabilitá. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze 8 3-62.
8. BRIER, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 1-3.
9. BRÖCKER, J. and SMITH, L. A. (2007). Increasing the reliability of reliability diagrams. Weather and Forecasting 22 651-661.
10. BUJA, A., STUETZLE, W. and SHEN, Y. (2005). Loss functions for binary class probability estimation and classification: Structure and applications. Statistics Department, Univ. Pennsylvania, Philadelphia, PA. Available at http://stat.wharton.upenn.edu/~buja/PAPERS/ paper-proper-scoring.pdf.
11. CARTER, C. K. and KOHN, R. (1994). On Gibbs sampling for state space models. Biometrika 81 541-553. MR1311096
12. CHEN, Y. (2008). Learning classifiers from imbalanced, only positive and unlabeled data sets. Project Report for UC San Diego Data Mining Contest. Dept. Computer Science, Iowa State Univ., Ames, IA. Available at https://www.cs.iastate.edu/~yetianc/cs573/files/CS573_ProjectReport_ YetianChen.pdf.
13. CLEMEN, R. T. andWINKLER, R. L. (2007). Aggregating probability distributions. In Advances in Decision Analysis: From Foundations to Applications (W. Edwards, R. F. Miles and D. von Winterfeldt, eds.) 154-176. Cambridge Univ. Press, Cambridge.
14. COOKE, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. Clarendon Press, New York. MR1136548
15. EREV, I., WALLSTEN, T. S. and BUDESCU, D. V. (1994). Simultaneous over-and underconfidence: The role of error in judgment processes. Psychological Review 66 519-527.
16. GELMAN, A., CARLIN, J. B., STERN, H. S. and RUBIN, D. B. (2003). Bayesian data analysis. CRC press, Boca Raton.
17. GELMAN, A., JAKULIN, A., PITTAU, M. G. and SU, Y.-S. (2008). A weakly informative default prior distribution for logistic and other regression models. Ann. Appl. Stat. 2 1360-1383. MR2655663
18. GEMAN, S. and GEMAN, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. Institute of Electrical and Electronics Engineer (IEEE) Transactions on Pattern Analysis and Machine Intelligence 6 721-741.
19. GENEST, C. and ZIDEK, J. V. (1986). Combining probability distributions: A critique and an annotated bibliography. Statist. Sci. 1 114-148. MR0833278
20. GENT, I. P. and WALSH, T. (1996). Phase transitions and annealed theories: Number partitioning as a case study. In Proceedings of European Conference on Artificial Intelligence (ECAI 1996) 170-174. Wiley, New York.
21. GNEITING, T. and RANJAN, R. (2013). Combining predictive distributions. Electron. J. Stat. 7 1747-1782. MR3080409
22. GNEITING, T., STANBERRY, L. I., GRIMIT, E. P., HELD, L. and JOHNSON, N. A. (2008). Rejoinder on: Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds [MR2434318]. TEST 17 256-264. MR2434326
23. GOOD, I. J. (1952). Rational decisions. J. R. Stat. Soc. Ser. B Stat. Methodol. 14 107-114. MR0077033
24. HASTINGS, C. JR., MOSTELLER, F., TUKEY, J. W. and WINSOR, C. P. (1947). Low moments for small samples: A comparative study of order statistics. Ann. Math. Statistics 18 413-426. MR0022335
25. HAYES, B. (2002). The easiest hard problem. American Scientist 90 113-117.
26. KARMARKAR, N. and KARP, R. M. (1982). The differencing method of set partitioning. Technical Report UCB/CSD 82/113, Computer Science Division, Univ. California, Berkeley, CA.
27. KELLERER, H., PFERSCHY, U. and PISINGER, D. (2004). Knapsack Problems. Springer, Dordrecht. MR2161720
28. LICHTENSTEIN, S., FISCHHOFF, B. and PHILLIPS, L. D. (1977). Calibration of Probabilities: The State of the Art. In Decision Making and Change in Human Affairs (H. Jungermann and G. De Zeeuw, eds.) 275-324. Springer, Berlin.
29. LUBINSKI, D. and HUMPHREYS, L. G. (1996). Seeing the forest from the trees: When predicting the behavior or status of groups, correlate means. Psychology, Public Policy, and Law 2 363.
30. MELLERS, B., UNGAR, L., BARON, J., RAMOS, J., GURCAY, B., FINCHER, K., SCOTT, S. E., MOORE, D., ATANASOV, P. and SWIFT, S. A. ET AL. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science 25. DOI:10.1177/0956797614524255.
31. METROPOLIS, N., ROSENBLUTH, A.W., ROSENBLUTH, M. N., TELLER, A. H. and TELLER, E. (1953). Equation of state calculations by fast computing machines. The Journal of Chemical Physics 21 1087-1092.
32. MIGON, H. S., GAMERMAN, D., LOPES, H. F. and FERREIRA, M. A. R. (2005). Dynamic models. In Bayesian Thinking: Modeling and Computation. Handbook of Statist. 25 553-588. Elsevier/North-Holland, Amsterdam. MR2490539
33. MILLS, T. C. (1991). Time series techniques for economists. Cambridge Univ. Press, Cambridge.
34. MORGAN, M. G. (1992). Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge Univ. Press, Cambridge.
35. MURPHY, A. H. and WINKLER, R. L. (1987). A general framework for forecast verification. Monthly Weather Review 115 1330-1338.
36. NEAL, R. M. (2003). Slice sampling. Ann. Statist. 31 705-767. MR1994729
37. PEPE, M. S. (2003). The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford Statistical Science Series 28. Oxford Univ. Press, Oxford. MR2260483
38. PRIMO, C., FERRO, C. A., JOLLIFFE, I. T. and STEPHENSON, D. B. (2009). Calibration of probabilistic forecasts of binary events. Monthly Weather Review 137 1142-1149.
39. RAFTERY, A. E., GNEITING, T., BALABDAOUI, F. and POLAKOWSKI, M. (2005). Using Bayesian model averaging to calibrate forecast ensembles. Monthly Weather Review 133 1155-1174.
40. RANJAN, R. and GNEITING, T. (2010). Combining probability forecasts. J. R. Stat. Soc. Ser. B Stat. Methodol. 72 71-91. MR2751244
41. SANDERS, F. (1963). On subjective probability forecasting. Journal of Applied Meteorology 2 191-201.
42. SATOPÄÄ, V. A., BARON, J., FOSTER, D. P., MELLERS, B. A., TETLOCK, P. E. and UNGAR, L. H. (2014a). Combining multiple probability predictions using a simple logit model. International Journal of Forecasting 30 344-356.
43. SATOPÄÄ, V. A., JENSEN, S. T., MELLERS, B. A., TETLOCK, P. E. and UNGAR, L. H. (2014b). Supplement to "Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs. " DOI:10.1214/14-AOAS739SUPP.
44. SHLYAKHTER, A. I., KAMMEN, D. M., BROIDO, C. L. andWILSON, R. (1994). Quantifying the credibility of energy projections from trends in past data: The US energy sector. Energy Policy 22 119-130.
45. TANNER, J., WILSON, P. and SWETS, J. A. (1954). A decision-making theory of visual detection. Psychological Review 61 401-409.
46. TETLOCK, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton Univ. Press, Princeton, NJ.
47. UNGAR, L., MELLERS, B., SATOPÄÄ, V., TETLOCK, P. and BARON, J. (2012). The good judgment project: A large scale test of different methods of combining expert predictions. In The Association for the Advancement of Artificial Intelligence 2012 Fall Symposium Series, Univ. Pennsylvania, Philadelphia, PA.
48. VISLOCKY, R. L. and FRITSCH, J. M. (1995). Improved model output statistics forecasts through model consensus. Bulletin of the American Meteorological Society 76 1157-1164.
49. WALLACE, B. C. andDAHABREH, I. J. (2012). Class probability estimates are unreliable for imbalanced data (and how to fix them). In Institute of Electrical and Electronics Engineers (IEEE) 12th International Conference on Data Mining (International Conference on Data Mining) 695-704. IEEE Computer Society, Washington, DC.
50. WALLSTEN, T. S., BUDESCU, D. V. and EREV, I. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making 10 243-268.
51. WILSON, A. G. (1994). Cognitive factors affecting subjective probability assessment. Discussion Paper 94-02, Institute of Statistics and Decision Sciences, Duke Univ., Chapel Hill, NC.
52. WILSON, P.W., D'AGOSTINO, R. B., LEVY, D., BELANGER, A.M., SILBERSHATZ, H. andKANNEL, W. B. (1998). Prediction of coronary heart disease using risk factor categories. Circulation 97 1837-1847.
53. WINKLER, R. L. and JOSE, V. R. R. (2008). Comments on: Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds [MR2434318]. TEST 17 251-255. MR2434325
54. WINKLER, R. L. and MURPHY, A. H. (1968). "Good" probability assessors. Journal of Applied Meteorology 7 751-758.
55. WRIGHT, G., ROWE, G., BOLGER, F. and GAMMACK, J. (1994). Coherence, calibration, and expertise in judgmental probability forecasting. Organizational Behavior and Human Decision Processes 57 1-25.