Aggregating expert judgement

Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales - Serie A: Matematicas

Volumn 105, Issue 1, 2011, Pages 181-206

  French, Simon ^a  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

Aggregation of expert judgement; Cooke's Principles; e Democracy; e Participation; Expert judgement; Expert problem; Group decision problem; Meta analysis; Textbook problem; Web based group decision support systems (wGDSS)

Indexed keywords

ARTIFICIAL INTELLIGENCE; TEXTBOOKS; UNCERTAINTY ANALYSIS; WEBSITES;

AGGREGATION OF EXPERT JUDGMENT; COOKE PRINCIPLE; E-DEMOCRACY; E-PARTICIPATION; EXPERT JUDGMENT; EXPERT PROBLEM; GROUP DECISION PROBLEM; META-ANALYSIS; TEXTBOOK PROBLEM; WEB BASED; WEB-BASED GROUP DECISION SUPPORT SYSTEM;

DECISION SUPPORT SYSTEMS;

EID: 79952577946     PISSN: 15787303     EISSN: 15791505     Source Type: Journal    
DOI: 10.1007/s13398-011-0018-6     Document Type: Article

References (177)

1. Adhikari, S., Bayley, C., Bedford, T., Busby, J. S., Cliffe, A., Devgun, G., Eid, M., French, S., Keshvala, R., Pollard, S., Soane, E., Tracy, D., Wu, S.: Human Reliability Analysis: A Review and Critique. Manchester Business School, Manchester (2008).
2. Armstrong, J. S.: Combining forecasts. In: Armstrong, J. S. (ed.) Principles of Forecasting: a Handbook for Researchers and Practitioners, pp. 417-439. Kluwer, Norwell (2001).
3. Arnell N. W., Tompkins E. L., Adger W. N.: Eliciting information from experts on the likelihood of rapid climate change. Risk Anal. 25(6), 1419-1431 (2005).
4. Arrow K. J.: Social Choice and Individual Values. Wiley, New York (1963).
5. Ashby D., Smith A. F. M.: Evidence-based medicine as Bayesian decision-making. Stat. Med. 19, 3291-3305 (2000).
6. Aspinal W.: A route to more tractiable expert advice. Nature 463, 294-295 (2010).
7. Bacharach M.: Group decisions in the face of differences of opinion. Manag. Sci. 22(2), 182-191 (1975).
8. Bayley, C., French, S.: Public participation: comparing approaches. J. Risk Res. (2010, in press).
9. Bedford T., Cooke R.: Probabilistic Risk Analysis: Foundations and Methods. Cambridge University Press, Cambridge (2001).
10. Bedford T., Quigley J., Walls L.: Expert elicitation for reliable system design. Stat. Sci. 21(4), 428-450 (2006).
11. Beierle, T., Cayford, J.: Democracy in Practice: Public Participation in Environmental Decisions. Resources for the Future (2002).
12. Bennett, P. G., Calman, K. C. (eds): Risk Communication and Public Health: Policy Science and Participation. Oxford University Press, Oxford (1999).
13. Bennett, P. G., Calman, K. C., Curtis, S., Fischbacher-Smith, D. (eds): Risk Communication and Public Health, 2nd edn. Oxford University Press, Oxford (2010).
14. Bier V. M.: Implications of the research on expert overconfidence and dependence. Reliab. Eng. Syst. Saf. 85, 321-329 (2004).
15. Bolger F., Wright G.: Assessing the quality of expert judgment: issues and analysis. Decis. Support Syst. 11(1), 1-24 (1994).
16. Booker J. M., Meyer M. A.: Sources and effects of interexpert correlation: an empirical study. IEEE Trans. Syst. Man Cybern. 18, 135-142 (1988).
17. Boone I., van der Stede Y., Bollaerts K., Messens W., Vose D., Daube G., Aerts M., Mintiens K.: Expert judgement in a risk assessment model for Salmonella spp. in pork: the performance of different weighting schemes. Prev. Vet. Med. 92(3), 224-234 (2009).
18. Bordley R. F.: Combining the opinions of experts who partition events differently. Decis. Anal. 6(1), 38-46 (2009).
19. Borenstein M., Hedges L. V., Higgins J. P. T., Rothstein H. R.: Introduction to Meta-Analysis. Wiley, Chichester (2009).
20. Burns W. J., Clemen R. T.: Covariance structure models and influence diagrams. Manag. Sci. 39(7), 816-834 (1993).
21. Clemen R. T.: Calibration and the aggregation of probabilities. Manag. Sci. 32(3), 312-314 (1986).
22. Clemen R. T.: Combining forecasts. Int. J. Forecast. 5, 559-583 (1989).
23. Clemen R. T., Winkler R. L.: Aggregating point estimates: a flexible modelling approach. Manag. Sci. 39, 501-515 (1993).
24. Clemen R. T., Reilly T.: Correlations and copulas for decision and risk analyses. Manag. Sci. 45(2), 208-224 (1999).
25. Clemen, R. T., Winkler, R. L.: Combining probability distributions from experts in risk analysis. Risk Anal. 19 (1999).
26. Clemen R. T., Fischer G. W., Winkler R. L.: Assessing dependence: some experimental results. Manag. Sci. 46(8), 1100-1115 (2000).
27. Clemen, R. T., Lichtendahl, K. C.: Debiasing expert overconfidence: a Bayesian calibration model. PSAM6, San Juan, Puerto Rico (2002).
28. Clemen, R. T., Winkler, R. L.: Aggregating probability distributions. In: Edwards, W., Miles, R. F., Von Winterfeldt, D. (eds.) Advances in Decision Analysis: From Foundations to Applications, pp. 154-176. Cambridge University Press, Cambridge (2007).
29. Clemen R. T.: Comment on Cooke's classical method. Reliab. Eng. Syst. Saf. 93(5), 760-765 (2008).
30. Cleveland W. S.: The Elements of Graphing Data. Murray Hill, New Jersey (1994).
31. Conitzer, V., Rothe, J. (eds.) COMSOC2010: Proceedings of the Third International Workshop on Computational Social Choice, September 13-16, 2010. Dusseldorf University Press, Dusseldorf (2010).
32. Cooke R. M.: Experts in Uncertainty. Oxford University Press, Oxford (1991).
33. Cooke, R. M., Bedford, T., Kraan, B.: Report on the Benchmark Workshop on Uncertainty/Sensitivity Analysis Codes. European Safety and Reliability Association. Copies from R. Cooke, Mathematics and Informatics. TU Delft, Delft (1997).
34. Cooke R. M., Goossens L. H. J.: Expert judgement elicitation for risk assessments of critical infrastructures. J. Risk Res. 7(6), 643-656 (2004).
35. Cooke, R. M., Probst, K. N.: Highlights of the Expert Judgement Policy Symposium and Technical Workshop. Resources for the Future, Washington, DC (2006).
36. Cooke R. M.: Expert judgement studies. Reliab. Eng. Syst. Saf. 93, 655-777 (2007).
37. Cooke R. M., ElSaadany S., Xinzheng H.: On the performance of social network and likelihood based expert weighting schemes. Reliab. Eng. Syst. Saf. 93(5), 745-756 (2007).
38. Cooke R. M., Goossens L. H. J.: TU delft expert judgement database. Reliab. Eng. Syst. Saf. 93(5), 657-674 (2007).
39. Cooke R. M.: Response to the reliability of aggregated probability judgments obtained through Cooke's classical model. TU Delft, Delft (2008).
40. Dalkey N., Helmer O.: An experimental application of the Delphi method to the use of experts. Manag. Sci. 9(3), 458-467 (1963).
41. Dawid A. P., DeGroot M. H., Mortera J.: Coherent combination of experts' opinions (with discussion). Test 4(2), 263-313 (1995).
42. De Finetti B.: Theory of Probability. Wiley, Chichester (1974).
43. DeGroot M. H., Bayarri M. J.: What Bayesians expect of each other. J. Am. Stat. Assoc. 86(416), 924-932 (1991).
44. Department of Health: Communicating About Risks to Public Health: Pointers to Good Practice. HMSO, London (1998).
45. DeWispelare A. R., Herren L. T., Clemen R. T.: The use of probability elicitation in the high-level nuclear waste regulation program. Int. J. Forecast. 11, 5-24 (1995).
46. Dryzek J. S., List C.: Social choice theory and deliberative democracy: a reconciliation. Br. J. Political Sci. 33(1), 1-28 (2003).
47. Eden, C., Radford, J. (eds): Tackling Strategic Problems: the Role of Group Decision Support. Sage, London (1990).
48. Ehrhardt J., Brown J., French S., Kelly G. N., Mikkelsen T., Muller H: RODOS: decision making support for off-site emergency management after nuclear accidents. Kerntechnik 62, 122-128 (1997).
49. Ehrhardt J., Weiss A.: RODOS: Decision Support for Off-Site Nuclear Emergency Management in Europe. EUR19144EN. European Community, Luxembourg (2000).
50. Faria A., Smith J. Q.: Conditionally externally Bayesian pooling operators in chain graphs. Ann. Stat. 25(4), 1740-1761 (1997).
51. Franco A., Shaw D., Westcombe M.: Problem structuring methods I. J. Oper. Res. Soc. 57, 757-878 (2006).
52. Franco A., Shaw D., Westcombe M.: Problem structuring methods II. J. Oper. Res. Soc. 58, 545-682 (2007).
53. Franke G. R.: Applications of meta-analysis for marketing and public policy: a review. J. Public Policy Market. 20(2), 186-200 (2001).
54. French S.: Updating of belief in the light of someone else's opinion. J. R. Stat. Soc. A 143, 43-48 (1980).
55. French S.: Consensus of opinion. Eur. J. Oper. Res. 7, 332-340 (1981).
56. French, S.: Group consensus probability distributions: a critical survey. In: Bernardo, J. M., DeGroot, M. H., Lindley, D. V., Smith, A. F. M. (eds.) Bayesian Statistics, vol. 2, pp. 183-201. North-Holland, Amsterdam (1985).
57. French S.: Calibration and the expert problem. Manag. Sci. 32, 315-321 (1986).
58. French, S.: Conflict of belief: when advisers disagree. In: Bennett, P. G. (ed.) Analysing Conflict and its Resolution: Some Mathematical Contributions, pp. 93-111. Oxford University Press, Oxford (1987).
59. French, S., Cooke, R. M., Vogt, F.: The use of expert judgement in the context of a physical model. In: Bernardo, J. M., Berger, J. O., Dawid, A. P., Smith, A. F. M. (eds.) Bayesian Statistics, vol. IV, pp. 617-624. Oxford University Press, Oxford (1991).
60. French S., Rios Insua D.: Statistical Decision Theory. Arnold, London (2000).
61. French S., Maule A. J., Mythen G., Wales C.: Trust and Risk Communications. Manchester Business School, Manchester (2002).
62. French S.: Modelling, making inferences and making decisions: the roles of sensitivity analysis. TOP 11(2), 229-252 (2003).
63. French, S., Battson, A.: Review and implications of research on the cognitive understanding of uncertainties relating to spatial and temporal plots. ENSEMBLE Project Report, Manchester Business School, Manchester (2003).
64. French, S., Bayley, C.: An analysis of the ENSEMBLE exercises from the perspective of decision making. ENSEMBLE Project Report, Manchester Business School, Manchester (2003).
65. French, S., Rios Insua, D., Ruggeri, F.: e-Participation and decision analysis. Decis. Anal. (2006, under revision).
66. French S.: Web-enabled strategic GDSS, e-democracy and Arrow's Theorem: a Bayesian perspective. Decis. Support Syst. 43, 1476-1484 (2007).
67. French S., Carter E., Niculae C.: Decision support in nuclear and radiological emergency situations: are we too focused on models and technology?. Int. J. Emerg. Manag. 4(3), 421-441 (2007).
68. French S., Rios Insua D., Ruggeri F.: e-participation and decision analysis. Decis. Anal. 4(4), 1-16 (2007).
69. French S., Maule A. J., Papamichail K. N.: Decision Behaviour, Analysis and Support. Cambridge University Press, Cambridge (2009).
70. French, S., Bedford, T., Pollard, S., Soane, E.: Human Reliability Analysis: a critique and review for the management of risk. Saf. Sci. (2010, in press).
71. Galmarini S., Bianconi R., Klug W., Mikkelsen T., Addis R., Andronopoulos S., Astrup P., Baklanov A., Bartniki J., Bartzis J., Bellasio R., Bompay F., Buckley R., Bouzom M., Champion H., D'Amours R., Davakis E., Eleveld H., Geertsema G. T., Glaab H., Kollax M., Ilvonen M., Manning A., Pechinger U., Persson C., Polreich E., Ptempski S., Prodanova M., Saltbones J., Slaper H., Sofiev M. A., Syrakov D., Sorenson J. H., Van der Auwera L., Valkama I., Zelazny R.: Can the confidence in long range atmospheric transport models be increased? The pan-European experience of ENSEMBLE. Radiat. Prot. Dosim. 109(1-2), 19-24 (2004).
72. Garthwaite P. H., Kadane J. B., O'Hagan A.: Statistical methods for eliciting probability distributions. J. Am. Stat. Assoc. 100(470), 680-701 (2005).
73. Genest C., Zidek J. V.: Combining probability distributions: a critique and annotated bibliography. Stat. Sci. 1, 114-148 (1986).
74. Genest C., Wagner C. G.: Further evidence against independence preservation in expert judgement synthesis. Aequationes Mathematicae 32, 74-86 (1987).
75. Geyskens, I., Krishnan, R., Steenkamp, J.-B. E. M., Cunha, P. V.: A review and evaluation of meta-analysis practices in management research. J. Manag. (2009). doi:10.1177/0149206308328501.
76. Gigerenzer G.: Reckoning with Risk: Learning to live with Uncertainty. Penguin Books, Harmondsworth (2002).
77. Gilovich, T., Griffin, D., Kahneman, D. (eds): Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge University Press, Cambridge (2002).
78. Givens G. H., Tweedie, Smith, D.D.: Publication bias in meta-analysis: a Bayesian data-augmentation approach to account for issues exemplified in the passive smoking debate. Stat. Sci. 12(4), 221-250 (1997).
79. Goldstein M., Rougier J. C.: Reified Bayesian Modelling and inference for physical systems (with discussion). J. Stat. Plan. Inference 139, 1221-1256 (2009).
80. Goldstein, M.: External Bayesian analysis for computer simulators. In: Bernardo, J. M., Bayarri, M. J., Bergeret, J. O., et al. (eds.) Bayesian Statistics, vol. 9. Oxford University Press, Oxford (2010, in press).
81. Goossens L. H. J., Kelly G. N.: Special issue: expert judgement and accident consequence uncertainty analysis. Radiat. Prot. Dosim. 90(3), 293-381 (2000).
82. Gregory R. S., Fischhoff B., McDaniels T.: Acceptable input: using decision analysis to guide public policy deliberations. Decis. Anal. 2(1), 4-16 (2005).
83. Hartung J., Knapp J., Sinha B. K.: Statistical Meta-Analysis with Applications. Wiley, Hoboken (2008).
84. Hastie, R.: Experimental evidence of group accuracy. In: Grofman, B., Owen, G. (eds.) Information Pooling and Group Decision Making, pp. 129-157. JAI Press, Greenwich (1986).
85. Hockey G. R. J., Maule A. J., Clough P. J., Bdzola L.: Effects of negative mood on risk in everyday decision making. Cogn. Emot. 14, 823-856 (2000).
86. Hodge J. K., Klima R. E.: The Mathematics of Voting and Elections: a Hands-On Approach. American Mathematical Society, Rhode Island (2005).
87. Hora, S.: Eliciting probabilities from experts. In: Edwards, W., Miles, R. F., Von Winterfeldt, D. (eds.) Advances in Decision Analysis: From Foundations to Applications, pp. 129-153. Cambridge University Press, Cambridge (2007).
88. Hora S. C.: Probability judgements for continuous quantities: linear combinations and calibration. Manag. Sci. 50(5), 597-604 (2004).
89. Hora, S. C.: Expert judgement. In: Melnick, E. L., Everitt, B. S.: Encylcopedia of Quantitative Risk Analysis and Assessment, pp. 667-676. Wiley, Chichester (2008).
90. Jacobs R. A.: Methods for combining experts' probability assessments. Neural Comput. 7, 867-888 (1995).
91. James A., Low-Choy S., Mengersen K.: Elicitator: an expert elicitation tool for regression in ecology. Environ. Model. Softw. 25(1), 129-145 (2010).
92. Jeffreys H.: Theory of Probability. Oxford University Press, Oxford (1961).
93. Jouini M. N., Clemen R. T.: Copula models for asggregating expert opinions. Oper. Res. 44, 444-457 (1996).
94. Kadane J. B., Schervish M. J., Seidenfeld T.: Rethinking the Foundations of Statistics. Cambridge University Press, Cambridge (1999).
95. Kahneman, D., Slovic, P., Tversky, A. (eds): Judgement under Uncertainty: Heuristics and Biases. Cambridge University Press, Cambridge (1982).
96. Kallen, M. J., Cooke, R. M.: Expert aggregation with dependence. In: Bonano, E. J., Camp, A. J., Majors, M. J., Thompson, R. A. (eds.) Proabilistic Safety Assessment and Management, pp. 1287-1294. Elsevier, Amsterdam (2002).
97. Karacapilidis N. I., Pappis C. P.: A framework for group decision support systems: combining AI tools and OR techniques. Eur. J. Oper. Res. 103, 373-388 (1997).
98. Keeney R. L., Raiffa H.: Decisions with Multiple Objectives: Preferences and Value Trade-offs. Wiley, New York (1976).
99. Keith D. W.: When is it better to combine expert judgements?. Clim. Chang. 33, 111-143 (1996).
100. Kelly F. S.: Arrow Impossibility Theorems. Academic Press, New York (1978).
101. Kennedy M. C., O'Hagan A.: Bayesian calibration of computer models. J. R. Stat. Soc. B 63, 425-464 (2001).
102. Koning, J.-L.: How electronic voting can escape Arrow's Impossibility Theorem. In: Padget, J., Neira, R., Diaz de Leon, J. L. (eds.) e-Government and e-Democracy: Progress and Challenges, pp. 138-146. Zacatenco, Mexico, Instituo Politecnico Nacional, Centro de Investigacion en Computacion, Unidad Profesional "Aldofo Lopez Mateos" (2003).
103. Kurowicka D., Cooke R. M.: Uncertainty Analysis with High Dimensional Modelling. Wiley, Chichester (2006).
104. Langford, I., Marris, C., O'Riordan, T.: Public reactions to risk: social structures, images of science and the role of trust. In: Bennett, P. G., Calman, K. C. (eds.) Risk Communication and Public Health: Policy, Science and Participation, pp. 33-50. Oxford University Press, Oxford (1999).
105. Leach, M., Scoones, I., Wynne, B. (eds): Science and Citizens. Zed Books, London (2005).
106. Leamer E. E.: Specification Searches. Wiley, Chichester (1978).
107. Lehmann H. P., Goodman S. N.: Bayesian communication: a clinically significant paradigm for electronic publication. J. Am. Med. Inf. Assoc. 7(3), 254-266 (2000).
108. Lichtendahl, K. C.: Bayesian Models of Expert Forecasts. PhD Thesis. Department of Business Administration, Duke University, Durham, NC (2005).
109. Lichtendahl K. C., Winkler R. L.: Probability elicitation, scoring rules, and competition among forecasters. Manag. Sci. 53(11), 1745-1755 (2007).
110. Lichtenstein, S., Fischhoff, B., Phillips, L. D.: Calibration of probabilities: the state of the art to 1980. In: Kahneman, D., Slovic, P., Tversky, A. (eds.) Judgement Under Uncertainty, pp. 306-334. Cambridge University Press, Cambridge (1982).
111. Lin S.-W., Bier V. M.: A study of expert overconfidence. Reliab. Eng. Syst. Saf. 93, 711-721 (2008).
112. Lin S.-W., Cheng C.-H.: The reliability of aggregated probability judgments obtained through Cooke's classical model. J. Model. Manag. 4(2), 149-161 (2009).
113. Lindley D. V., Tversky A., Brown R. V.: On the reconciliation of probability judgements (with discussion). J. R. Stat. Soc. A 142, 146-180 (1979).
114. Lindley D. V.: The improvement of probability judgements. J. R. Stat. Soc. A 145, 117-126 (1982).
115. Lindley D. V.: Reconciliation of probability distributions. Oper. Res. 31(5), 866-880 (1983).
116. Lindley, D. V.: Reconciliation of discrete probability distributions. In: Bernardo, J. M., DeGroot, M. H., Lindley, D. V., Smith, A. F. M. (eds.) Bayesian Statistics, vol. 2, pp. 375-390. North Holland, Amsterdam (1985).
117. Linstone H. A., Turoff M.: The Delphi Method: Techniques and Applications. Addison-Wesley, London (1978).
118. Lipscomb J., Parmigiani G., Hasselblad V.: Combining expert judgment by hierarchical modeling: an application to physician staffing. Manag. Sci. 44(2), 149-161 (1998).
119. Low-Choy, S., Rousseau, J., Mengersen, K.: Poster on 'Balancing expert consensus and diversity'. Bayesian Statistics 9. Valencia (2010).
120. Madansky, A.: Externally Bayesian groups. RM-4141-PR. RAND (1964).
121. Maule A. J.: Translating risk management knowledge: the lessons to be learned from research on the perception and communication of risk. Risk Manag. 6(2), 15-27 (2004).
122. Maule, A. J.: Risk communication and organisations. In: Starbuck, W., Hodgkinson, G. (eds.) The Oxford Handbook of Organizational Decision Making. Oxford University Press, Oxford (2008).
123. Merrick J. R. W.: Getting the right mix of experts. Decis. Anal. 5(1), 43-52 (2008).
124. Merrick J. R. W.: Bayesian simulation and decision analysis: an expository survey. Decis. Anal. 6(4), 222-238 (2009).
125. Mikkelsen, T., Galmarini, S., Bianconi, R., French, S.: ENSEMBLE: methods to reconcile disparate national long-range dispersion forecasts. RISO, Roskilde (2003).
126. Morris, C. N., Normand, S. L.: Hierarchical models for combining information and meta-analyses. In: Bernardo, J. M., Berger, J. O., Dawid, A. P., Smith, A. F. M. (eds.) Bayesian Statistics 4, pp. 321-344. Oxford, Oxford University Press (1992).
127. Morton A., Ackermann F., Belton V.: Technology-driven and model-driven approaches to group decision support: focus, research philosophy, and key concepts. Eur. J. Inf. Syst. 12(2), 110-126 (2003).
128. Mumpower J. L., Stewart T. R.: Expert judgement and expert disagreement. Thinking Reason. 2(2-3), 191-211 (1996).
129. Murray J. V., Goldizen A. W., O'Leary R. A., McAlpine C., Possingham H. P., Low-Choy S.: How useful is expert opinion for predicting the distribution of a species within and beyond the region of expertise? A case study using brush-tailed rock-wallabies Petrogale penicillata. J. Appl. Ecol. 46(4), 842-851 (2009).
130. Newbold P., Granger C. W. J.: Experience with forecasting univariate time series and the combination of forecasts. J. R. Stat. Soc. A 137, 131-164 (1974).
131. O'Hagan A.: Eliciting expert beliefs in substantial practical applications. Statistician 47(1), 21-35 (1998).
132. O'Hagan, A., Kennedy, M. C., Oakley, J. E.: Uncertainty analysis and other inference tools for complex computer codes. In: Bernardo, J. M., Berger, J. O., Dawid, A. P., Smith, A. F. M. (eds.) Bayesian Statistics 6, pp. 503-524. Oxford University Press, Oxford (1998).
133. O'Hagan A., Forester J.: Bayesian Statistics. Edward Arnold, London (2004).
134. O'Hagan A., Buck C. E., Daneshkhah A., Eiser R., Garthwaite P. H., Jenkinson D., Oakley J. E., Rakow T.: Uncertain Judgements: Eliciting Experts' Probabilities. Wiley, Chichester (2006).
135. O'Neill S. J., Osborn T. J., Hulme M., Lorenzoni I., Watkinson A. R.: Using expert knowledge to assess uncertainties in future polar bear populations under climate change. J. Appl. Ecol. 45(6), 1649-1659 (2008).
136. Ortiz N. R., Wheeler T. A., Breeding R. J., Hora S., Meyer M. A., Keeney R. L.: Use of expert judgement in NUREG 1150. Nucl. Eng. Des. 126, 313-331 (1991).
137. Phillips, L. D.: Decision conferencing. In: Edwards, W., Miles, R. F., von Winterfeldt, D. (eds.) Advances in Decision Analysis: From Foundations to Applications, pp. 375-399. Cambridge University Press, Cambridge, (2007).
138. Predd J., Osherson S., Kulkarni H.: Aggregating probabilistic forecasts from incoherent and abstaining experts. Decis. Anal. 5(4), 177-189 (2008).
139. Raiffa H.: The Art and Science of Negotiation: How to Resolve Conflicts and Get the Best out of Bargaining. Belknap Press/Harvard University, Cambridge (1982).
140. Raiffa H., Richardson J., Metcalfe D.: Negotiation Analysis: the Science and Art of Collaborative Decision Making. Harvard University Press, Cambridge (2002).
141. Ranjan R., Ranjan R.: Combining probability forecasts. J. R. Stat. Soc. B 72(1), 71-91 (2010).
142. Reagan-Cirincione P.: Combining group facilitation, decision modelling and information technology to improve the accuracy of group judgement Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology. Organ. Behav. Hum. Decis. Process 58, 246-270 (1994).
143. Renn O., Webler T., Rakel H., Dienel P., Johnson B.: Public participation in decision making: a three step-procedure. Policy Sci. 26(3), 189-214 (1993).
144. Renn, O., Webler, T., Wiedermann, P. (eds): Fairness and Competence in Citizen Participation: Evaluating Models and Environmental Discourse. Kluwer, Dordrecht (1995).
145. Renn O.: The role of risk communication and public dialogue for improving risk management. Risk Decis. Policy 3, 3-50 (1998).
146. Renn O.: Risk Governance. Earthscan, London (2008).
147. Renooij S.: Probability elicitation for belief networks: issues to consider. Knowl. Eng. Rev. 16(3), 255-269 (2001).
148. Rios Insua D., Ruggeri F.: Robust Bayesian Analysis. Springer, New York (2000).
149. Rios Insua, D., French, S. (eds): e Democracy: a Group Decision and Negotiation Perspective. Group Decision and Negotiation. Springer, Dordrecht (2010).
150. Rosenthal R., DiMatteo M. R.: Meta-analysis: recent developments in quantitative methods for literature reviews. Annu. Rev. Psychol. 52, 59-82 (2001).
151. Rosqvist T.: Bayesian aggregation of experts' judgements on failure intensity. Reliab. Eng. Syst. Saf. 70(3), 283-289 (2000).
152. Rougier J. C., Guillas S., Maute A., Richmond A. D.: Expert knowledge and multivariate emulation: the thermospher-ionosphere electrodynamics general circulation model (TIE-GCM). Technometrics 51(4), 414-424 (2009).
153. Rowe G., Wright G.: The Delphi technique as a forecasting tool: issues and analysis. Int. J. Forecast. 15, 353-375 (1999).
154. Savage L. J.: The Foundations of Statistics. Dover, New York (1972).
155. Scapolo F., Miles I.: Eliciting experts' knowledge: a comparison of two methods. Technol. Forecast. Soc. Chang. 73(6), 679-704 (2006).
156. Shanteau, J.: Expert Judgment and Financial Decision Making. In: Green, B. (ed.) Risky Business: Risk Behavior and Risk Management. Stockholm University, Stockholm (1995).
157. Skjong, R., Wentworth, B. H.: Expert Judgement and Risk Perception. In: Proceedings of the Eleventh (2001) International Offshore and Polar Engineering Conference, Stavanger, Norway, The International Society of Offshore and Polar Engineers (2001).
158. Slovic, P.: Trust, emotion, sex, politics and science: surveying the risk-assessment battlefield. In: Bazerman, M., Messick, D., Tenbrunsel, A., Wade-Benzoni, K. (eds.) Environment, Ethics and Behaviour. New Lexington Press, San Francisco (1997).
159. Slovic P.: Perceptions of Risk. Earthscan Library, London (2001).
160. Smith J. Q., Faria A.: Bayesian Poisson models for the graphical combination of dependent expert information. J. R. Stat. Soc. B 62(3), 525-544 (2000).
161. Smith J. Q.: Bayesian Decision Analysis: Principles and Practice. Cambridge University Press, Cambridge (2010).
162. Smith L.: Chaos: A Very Short Introduction. Oxford University Press, Oxford (2007).
163. Stanley T. D.: Wheat from chaff: meta-analysis as quantitative literature review. J. Econ. Perspect. 15(3), 131-151 (2001).
164. Suantak L., Bolger F., Ferrell W. R.: The hard-easy effect in subjective probability calibration. Organ. Behav. Human Decis. Process. 67, 201-221 (1996).
165. Surowiecki J.: The Wisdom of Crowds. Random House, New York (2004).
166. Sutton A. J., Abrams K. R.: Bayesian methods in meta-analysis and evidence synthesis. Stat. Methods Med. Res. 10(4), 277-303 (2001).
167. Szwed, P. S., van Dorp, J. R.: A Bayesian Model for Rare Event Risk Assessment Using Expert Judgement About Paired Scenario Comparisons. ASEM National Conference (2002).
168. Szwed P. S., van Dorp J. R., Merrick J. R. W., Mazzuchi T. A., Singh A.: A Bayesian paired comparison approach for relative accident probability assessment with covariate information. Eur. J. Oper. Res. 169(1), 157-177 (2006).
169. Taylor A. D.: Social Choice and the Mathematics of Manipulation. Cambridge University Press, Cambridge (2005).
170. Turner R. M., Speigelhalter D. J., Smith G. C. S., Thompson S. G.: Bias modelling in evidence synthesis. J. R. Stat. Soc. A 172(1), 21-47 (2009).
171. Winkler R. L.: Combining probability distributions from dependent information sources. Manag. Sci. 27, 479-488 (1981).
172. Wiper, M. W.: Calibration and use of expert probability judgements, PhD. School of Computer Studies, University of Leeds, Leeds (1990).
173. Wiper M. W., French S., Cooke R. M.: Hypothesis test based calibration scores. Statistician 43, 231-236 (1994).
174. Wiper M. W., French S.: Combining experts' opinions using a normal-Wishart model. J. Forecast. 14, 25-34 (1995).
175. Wisse B., Bedford T., Quigley J.: Expert judgement combination using moment methods. Reliab. Eng. Syst. Saf. 93, 675-686 (2007).
176. Wright, G., Ayton, P. (eds): Subjective Probability. Wiley, Chichester (1994).
177. Zacharakis A. L., Shepherd D. A.: The nature of information and overconfidence on venture capitalist's decision making. J. Bus. Ventur. 16, 311-332 (2001).