Eliciting Knowledge for Analytical Representation

IEEE Transactions on Systems, Man and Cybernetics

Volumn 19, Issue 3, 1989, Pages 448-461

  Fischhoff, Baruch ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DECISION THEORY AND ANALYSIS; RISK STUDIES--ANALYSIS;

ANALYTICAL LANGUAGES; COST-BENEFIT ANALYSIS; FORECASTING; KNOWLEDGE ELICITATION; KNOWLEDGE REPRESENTATION; PROBABILISTIC RISK ANALYSIS;

ARTIFICIAL INTELLIGENCE;

EID: 0024668403     PISSN: 00189472     EISSN: 21682909     Source Type: Journal    
DOI: 10.1109/21.31052     Document Type: Article

References (108)

1. T.B. Sheridan, B. Fischhoff, M. Posner, and R.W. Pew, “Supervisory control systems,” in Research Needs for Human Factors. Washington, DC: Committee on Human Factors, National Research Council, 1983.
2. B.G. Buchanan and E.H. Shortliffe, Eds., Rule-Based Expert Systems: The MYCIN Experiments. Reading, MA: Addison-Wesley, 1984.
3. J. Mumpower, Ed., Expert Systems and Expert Judgment. Berlin: Springer-Verlag, 1987.
4. M. Henrion, M.G. Morgan, I. Nair, and C. Wiecha, “Evaluating an information system for policy modeling and uncertainty analysis,” J. Amer. Society Inform. Sci., vol. 37, no. 5, pp. 319–330, 1986.
5. R.R. Tenney and N.R. Sandell, “Structures for distributed decisionmaking,” IEEE Trans. Syst. Man Cybern., vol. SMC-11, no. 8, pp. 517–527, 1981.
6. D.M. Johnson, Systematic Introduction to the Psychology of Thinking. New York: Harper & Row, 1972.
7. E.C. Poulton, “The new psychophysics: Six models for magnitude estimation,” Psych. Bull., vol. 69, pp. 1–19, 1968.
8. N.J. McCormick, Reliability and Risk Analysis. New York: Academic Press, 1981.
9. U.S. Nuclear Regulatory Commission, PRA Procedures Guide. Washington, DC: The Commission, 1983, NUREG/CR-2300.
10. National Research Council, Reliability Analysis in NASA. Washington, DC: The Council, 1988.
11. National Research Council, Research Needs for Human Factors. Washington, DC: The Council, 1983.
12. National Research Council, Survey Measure of Subjective Phenomena. Washington, DC: The Council, 1982.
13. D. von Winterfeldt and W. Edwards, Decision Analysis and Behavioral Research. New York: Cambridge University Press, 1986.
14. J. Bentkover, V. Covello, and J. Mumpower, Eds., Benefits Assessment: The State of the Art. Amsterdam: Reidel, 1985.
15. B. Fischhoff, “Judgmental aspects of forecasting: Needs and possible trends,” Int. J. Forecasting, vol. 4, 1988.
16. R. Rosenthal and R.L. Rosnow, Artifact in Behavioral Research. New York: Academic Press, 1969.
17. National Research Council. Risk Assessment in the Federal Government. Washington, DC: The Council, 1983.
18. Office of Science and Technology Policy, “Chemical carcinogens: Review of the science and its associated principles,” Fed. Register, vol. 49, no. 100, pp. 21594–21661, 1984.
19. B. Fischhoff, S. Lichtenstein, P. Slovic, S. Derby, and R. Keeney, Acceptable Risk. New York: Cambridge University Press, 1981.
20. R. Budnitz, “Applications of probabilistic risk assessment to nuclear power plants,” in V. Covello and M. Merkhofer, Eds., Risk Assessment. New York: Plenum, 1988.
21. G. Ostberg, H. Hoffstedt, G. Holm, B. Klingernstierna, B. Ryd-nert, V. Samsonowitz, and L. Sjöberg, Inconceivable Events in Handling Material in Heavy Mechanical Engineering Industry. Stockholm: National Defence Research Institute, 1977, FTL-A16, pp. 71E.
22. R. Kadlec, “Field and laboratory event investigation for hazard characterization,” in Risk Assessment, V. Covello and M. Merkhofer, Eds. New York: Plenum, 1988.
23. R. Pew, D.C. Miller, and C.E. Feeher, Evaluation of Proposed Control Room Improvements Through A nalysis of Critical Operation Decisions. Palo Alto, CA: Electric Power Research Institute, 1982, EPRI NP-1982.
24. J.V. Rodricks and R.G. Tardiff, “Animal research methods and dose-response assessment,” in Risk Assessment, V. Covello and M. Merkhofer, Eds. New York: Plenum, 1988.
25. M.S. Tockman and A.M. Lilienfeld, “Epidemiologic methods in risk assessment,” in Risk Assessment, V. Covello and M. Merkhofer, Eds. New York: Plenum, 1988.
26. E.J. O'Flaherty and M.E. Andersen, “Pharmacokinetics methods in risk assessment,” in Risk Assessment, V. Covello and M. Merkhofer, Eds. New York: Plenum, 1988.
27. J.D. Hackney and W.S. Linn, “Controlled exposures of humans to air pollutants,” in Risk Assessment, V. Covello and M. Merkhofer, Eds. New York: Plenum, 1988.
28. C.C. Travis, “Modeling methods for toxic chemical exposure assessment,” in Risk Assessment, V. Covello and M. Merkhofer, Eds. New York: Plenum, 1988.
29. F.P. Sioshansi, “Subjective evaluation using expert judgment: An application,” IEEE Trans. Syst. Man Cybern., vol. SMC-13, no. 3, pp. 391–397, 1983.
30. J.H. Davis, Group Performance. Reading, MA: Addison-Wesley, 1982.
31. D.G. Myers and H. Lamm, “The group polarization phenomenon,” Psych. Bull., vol. 83, no. 4, pp. 602–627, 1976.
32. G.G. Pinder, “Groundwater contaminant transport modeling,” Environ. Sci. Technol., vol. 18, no. 4, pp. 108A-114A, 1984.
33. E. Tulving, “Episodic and semantic memory,” in Organization of Memory, E. Tulving and W. Donaldson, Eds. New York: Academic Press, 1972.
34. R. Lachman, J.T. Lachman, and E.C. Butterfield, Cognitive Psychology and Information Processing. Hillsdale, NJ: Erlbaum, 1979.
35. T. Wallsten and D.F. Budescu, “Encoding subjective probabilities: A psychological and psychometric review,” Manag. Sci., vol. 29, pp. 151–173, 1983.
36. B. Fischhoff and C. Whipple, “Risk assessment: Evaluating air quality with subjective estimates,” Environ. Professional, vol. 3, no. 3, pp. 277–291, 1981.
37. M.G. Morgan, M. Henrion, and S.C. Morris, Expert Judgments for Policy Analysis. Brookhaven, NY: Brookhaven National Laboratory, 1981.
38. B. Fischhoff, “Debiasing,” in Judgment Under Uncertainty: Heuristics and Biases, D. Kahneman, P. Slovic, and A. Tversky, Eds. New York: Cambridge University Press, 1982.
39. B. Brehmcr, “In a word: Not from experience,” Acta Psychol., vol. 45, pp. 223–241, 1980.
40. M. Henrion and B. Fischhoff, “Assessing uncertainty in physical constants.” Amer. J. Phys., vol. 54, no. 9, pp. 791–798, 1986.
41. S. Lichtenstein and B. Fischhoff, “Training for calibration,” in Organiz. Behav. Human Perform., vol. 26, pp. 149–171, 1980.
42. R.E. Nisbctt, D.H. Krantz, C. Jcpson, and Z. Kunda, “The use of statistical heuristics in everyday inductive reasoning,” Psych. Rev., vol. 90, pp. 339–363, 1983.
43. A.H. Murphy and R.L. Winkler, “Probability of precipitation forecasts,” J. Amer. Statistical Assn., vol. 79, pp. 489–500, 1984.
44. R. Beyth-Marom, S. Dekel, R. Gombo, and M. Shaked. Thinking under Uncertainty. Hillsdale, NJ: Erlbaum, 1985.
45. D. Kahneman and A. Tversky, “Intuitive predictions: Biases and corrective procedures,” TIMS Studies Manag. Sci., vol. 12, pp. 313–327, 1980.
46. C.S. Spetzler and C.A. Stael von Holstein, “Probability encoding in decision analysis,” Manag. Sci., vol. 22. pp. 340–358, 1975.
47. C.A. Stael von Holstein and J.E. Matheson. A Manual for Encoding Probability Distributions. Menlo Park, CA: SRI International, 1979.
48. D.W. Boyd and S.G. Regulinski, Characterizing Uncertainty in Technology Cost and Performance. Palo Alto, CA: Decision Focus Inc., 1979.
49. G. Mancini, D.D. Woods, and E. Hallnagel, Eds., Cognitive Engineering in Dynamic Worlds. London: Academic Press, in press.
50. B. Fischhoff. S. Watson, and C. Hope, “Defining risk,” Policy Sci., vol. 17, pp. 123–139, 1984.
51. J. Rasmussen and W.B. Rouse, Eds., Human Detection and Diagnosis of System Failure. New York: Plenum, 1981.
52. T.B. Sheridan, “Human error in nuclear power plants,” Technol. Rev., vol. 82, no. 4, pp. 23–33, 1980.
53. R.W. Bailev, Human Performance in Engineering. Englewood Cliffs. NJ: Prentice Hall, 1982.
54. W. Hammer, Product Safety and Management Engineering. Englewood Cliffs, NJ: Prentice Hall, 1980.
55. B. Fischhoff, D. MacGregor, and L. Blackshaw. “Creating categories for databases,” Int. J. Man-Machine Syst., vol. 27. pp. 33–63, 1987.
56. G.W. Furnas, T.K. Landauer, L.M. Gomez, and S.T. Dumais, “Statistical semantics: Analysis of the potential performance of key-word information systems,” Bell Syst. Tech. J., vol. 62, pp. 1753–1806, 1983.
57. G.L. Murphy and D.L. Medin, “The role of theories in conceptual coherence,” Psych. Rev., vol. 92, pp. 289–316, 1985.
58. B.A. Bailar and N.D. Rothwell, “Measuring employment and unemployment,” in Surveying Subjective Phenomena, C.F. Turner and E. Martin, Eds. New York: Russell Sage, 1984.
59. R.M. Bernero, “Probabilistic risk assessment—Problems and uncertainties,” remarks at NRC Workshop on Safety Goals. Harpers Ferry, WV, July 23, 1981.
60. P.M. Fitts and M.L. Posner, Human Performance. Belmont. CA: Wadsworth, 1979.
61. J. St. B. Evans. The Psychology of Deductive Reasoning. Routledge & Kegan Paul, 1982.
62. J.B. Brooke, J.F. Cook, and K.D. Duncan, “Effects of computer aiding and pretraining on fault location,” Ergonomics, vol. 26, no. 7, pp. 669–686, 1983.
63. R.M. Dawes, “The robust beauty of improper linear models in decision making,” Amer. Psychologist, vol. 34, pp. 571–582, 1979.
64. R.E. Nisbett and L. Ross, Human Inference: Strategies and Shortcomings of Social Judgment. Englewood Cliffs. NJ: Prentice-Hall, 1980.
65. K.A. Ericsson and H.A. Simon, “Verbal reports as data,” Psych. Rev., vol. 87, pp. 215–251, 1980.
66. R.E. Nisbett and T.D. Wilson, “Telling more than we know: Verbal reports on mental processes,” Psych. Rev., vol. 84, no. 3, pp. 231–259, 1977.
67. D.G. Myers and H. Lamm, “The polarizing effect of group discussion,” Amer. Scientist, vol. 63, pp. 297–303, 1975.
68. I. Janis, Victims of Groupthink. Boston: Houghton Mifflin, 1972.
69. C. Hohenemser, R.W. Kates, and P. Slovic, “The nature of technological hazard,” Sci., vol. 220, pp. 378–384, 1983.
70. A.M. Collins and E.F. Loftus, “A spreading-activation theory of semantic processing,” Psych. Rev., vol. 82. pp. 407–428, 1975.
71. E.C. Poulton, “Quantitative subjective assessments are almost always biased, sometimes completely misleading,” British J. Psych., vol. 68, pp. 409–425, 1977.
72. E.C. Poulton, “Biases in quantitative judgments,” Appl. Ergonomics, vol. 13, pp. 31–41, 1982.
73. B. Fischhoff, P. Slovic, and S. Lichtenstein, “Knowing what you want: Measuring labile values,” in Cognitive Processes in Choice and Decision Behavior, T. Wallsten, Ed. Hillsdale, NJ: Erlbaum, 1980.
74. B. Fischhoff and D. MacGregor. “Judged lethality.” Risk Anal., vol. 2, pp. 559–575, 1983.
75. J.C. Nunnally, Tests and Measurements. New York: McGraw-Hill, 1959.
76. P. Slovic, B. Fischhoff, and S. Lichtenstein, “Characterizing perceived risk,” in Perilous Progress: Managing the Hazards of Technology, R.W. Kates, C. Hohenemser, and J.X. Kasperson, Eds. Boulder, CO: Westview, 1985.
77. S. Lichtenstein, P. Slovic, B. Fischhoff, M. Layman, and B. Coombs, “Judged frequency of lethal events,” J. Experimental Psych.: Human Learn. Memory, vol. 4, pp. 551–578, 1978.
78. J.F. Kidd, “The utilization of subjective probabilities in production planning,” Acta Psych., vol. 34, pp. 338–347, 1970.
79. N.M. Bradburn and S. Sudman, Improving Interview Method and Questionnaire Design. San Francisco, CA: Jossey-Bass, 1979.
80. S. Payne, The Art of Asking Questions, 2nd Edition. Princeton, NJ: Princeton University Press, 1980.
81. H. Schuman and S. Presser, Questions and Answers in Attitude Surveys. New York: Academic Press, 1981.
82. B.G. Andreas. Experimental Psychology. New York: Wiley, 1972.
83. R.C. Atkinson, R.J. Hermstein, G. Lindzcy, and R.D. Luce, Eds., Stevens' Handbook of Experimental Psychology. New York: Wiley, 1988.
84. E.C. Carterrette and M.P. Friedman, Handbook of Perception, Volume 2. New York: Academic Press, 1974.
85. L.E. Marks. Sensory Processes: The New Psychophysics. New York: Academic Press, 1974.
86. B. Fischhoff, P. Slovic, and S. Lichtenstein, “Fault trees: Sensitiv-itv of estimated failure probabilities to problem representation,” J. Experimental Psych.: Human Perception Perform., vol. 4, pp. 330–344, 1978.
87. C.F. Gettys. C.A. Manning, and J.T. Casey, An Evaluation of Human Act Generation Performance. Norman, OK: Decision Processes Laboratory, University of Oklahoma, 1981, TR 15-8–81.
88. G.F. Pitz, N.J. Sachs, and J. Heerboth, “Procedures of eliciting choices in the analysis of individual decision,” Organiz. Behav. Human Perform., vol. 26, pp. 396–408, 1980.
89. S. Fiske and S.E. Taylor, Social Cognition. Reading, MA: Addison-Wesley, 1984.
90. A.H. Hastorf and A.M. Isen, Eds., Cognitive Social Psychology. Amsterdam: North Holland, 1982.
91. B. Fischhoff and R. Beyth-Marom, “Hypothesis evaluation from a Bayesian perspective.” Psych. Rev., vol. 90, pp. 239–260, 1983.
92. J.S. Armstrong, Long Range Forecasting. New York: Wiley, 1985.
93. B. Fischhoff, O. Svenson, and P. Slovic, “Active responses to environmental hazards,” in Handbook of Environmental Psychology, D. Stokols and I. Altman Eds. New York: Wiley, 1987.
94. D. Kahneman, P. Slovic, and A. Tverky, Eds., Judgment under Uncertainty: Heuristics and Biases. New York: Cambridge University Press, 1982.
95. R. Krzysztofowicz, “Why should a forecaster and a decision maker use Bayes theorem,” Water Resources J., vol. 19, pp. 327–336, 1983.
96. S. Lichtenstein, B. Fischhoff, and L.D. Phillips, “Calibration of probabilities: State of the art to 1980,” in Judgment Under Uncertainty: Heuristics and Biases, D. Kahneman, P. Slovic, and A. Tversky. Eds. New York: Cambridge University Press, 1982.
97. R. Beyth-Marom, “How probable is probable? Numerical translation of verbal probabilitv expressions,” J. Forecasting, vol. 1, pp. 257–269, 1982.
98. A.H. Murphy and B.G. Brown, “Forecast terminology: Composition and interpretation of public weather forecasts,” Bull. Amer. Meteorolog. Soc, vol. 64, pp. 13–22, 1983.
99. G.L. Wells and E.F. Loftus, Eds., Eyewitness Testimony: Psychological Perspectives. New York: Cambridge University Press, 1983.
100. M. Henrion and M.G. Morgan, “A computer aid for risk and other policy analysis,” Risk Anal., vol. 5, no. 3. 195–208, 1985.
101. A. Koriat, S. Lichtenstein. and B. Fischhoff, “Reasons for Confidence,” J. Experimental Psych.: Human Learn. Memory, vol. 6, pp. 107–118. 1980.
102. D.T. Campbell and J.C. Stanley, “Experimental and quasi-experimental designs for research on teaching.” in Handbook for Research on Teaching, N.L. Gage, Ed. Skokie, IL: Rand McNally, 1963.
103. B. Fischhoff, “For those condemned to study the past: Reflections on historical judgment,” in New Directions for Methodology of Behavior Science: Fallible Judgment in Behavioral Research, R.A. Shweder and D.W. Fiske, Eds. San Francisco, CA: Jossey-Bass. 1980.
104. B. Fischhoff, “Hindsight # foresight: The effect of outcome knowledge on judgment under uncertainty,” J. Experimental Psych.: Human Perception Perform., vol. 1. pp. 288–299, 1975.
105. T.R. Stewart and M.H. Glantz, “Expert judgment and climate forecasting: A methodological critique of 'Climate Change to the Year 2000′,” Climate Change, vol. 7, pp. 159–183, 1986.
106. B. Fischhoff, “Clinical decision analysis,” Operations Res., vol. 28, pp. 28–43, 1980.
107. National Research Council, Priority Mechanisms for Toxic Chemicals. Washington, DC: The Council, 1983.
108. J.E. Hunter, F.L. Schmidt, and G.B. Jackson, Meta-Analysis. Beverly Hills, CA: Sage, 1982.