Use of the critical decision method to elicit expert knowledge: A case study in the methodology of cognitive task analysis

Human Factors

Volumn 40, Issue 2, 1998, Pages 254-276

  Hoffman, Robert R ^a,d   Crandall, Beth ^b,e   Shadbolt, Nigel ^c,f  

^a ADELPHI UNIVERSITY   (United States)

^b Klein Associates   (United States)

^c UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^d UNIVERSITY OF CINCINNATI   (United States)

^e WRIGHT STATE UNIVERSITY   (United States)

^f UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

CRITICAL PATH ANALYSIS; ERGONOMICS; EXPERT SYSTEMS; KNOWLEDGE REPRESENTATION; RELIABILITY;

COGNITIVE TASK ANALYSIS; CRITICAL DECISION METHODS; INSTRUCTIONAL DESIGN; MULTIPLE PASS EVENT RETROSPECTION GUIDE; SYSTEM DEVELOPMENT;

DECISION SUPPORT SYSTEMS;

ANALYTIC METHOD; ARTICLE; COGNITION; DECISION MAKING; HUMAN; RELIABILITY; TASK PERFORMANCE;

EID: 0032104351     PISSN: 00187208     EISSN: None     Source Type: Journal    
DOI: 10.1518/001872098779480442     Document Type: Article

References (119)

1. Adelman, L. (1989). Management issues in knowledge elicitation. IEEE Transactions on Systems, Man and Cybernetics, 19, 483-488.
2. Bainbridge, L. (1979). Verbal reports as evidence of the process operator's knowledge. International Journal of Man-Machine Studies, 11, 411-436.
3. Beach, L. R., & Lipshitz, R. (1993). Why classical decision theory is an inappropriate standard for evaluating and aiding most human decision making. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 21-35). Norwood, NJ: Ablex.
4. Benjafield, J. (1969). Evidence that "thinking aloud" constitutes an externalization of inner speech. Psychonomic Science, 15, 83-84.
5. Berry, D. C. (1987). The problem of implicit knowledge. Expert Systems, 4, 144-151.
6. Brezovic, C. P., Klein, G., & Thordsen, M. (1987). Decision making in armored platoon command (Tech. Report MDA903-85-C-0327, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
7. Briggs, R. W., & Goldberg, J. H. (1995). Battlefield recognition of armored vehicles. Human Factors, 37, 596-610.
8. Brown, B. (1989). The taming of an expert: An anecdotal report. SIGART Newsletter, 108, 133-135.
9. Buckner, V. G. (1997, May). Use of stories in training new employees improves recall. Poster presented at the Annual Convention of the American Psychological Society, Washington, DC.
10. Calderwood, R., Crandall, B. W., & Klein, G. A. (1987). Expert and novice fireground command decisions (Tech. Report MDA903-85-C-0327, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
11. Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representations of physics problems by experts and novices. Cognitive Science, 5, 121-152.
12. Chi, M. T. H., Glaser, R., & Farr, M. J. (Eds.). (1988). The nature of expertise. Mahwah, NJ: Erlbaum.
13. Chignell, M. H., & Peterson, J. G. (1988). Strategic issues in knowledge engineering. Human Factors, 30, 381-394.
14. Christensen-Szalanski, J. J. (1993). A comment on applying experimental findings of cognitive biases to naturalistic environments. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 252-264). Norwood, NJ: Ablex.
15. Clancey, W. J. (1993). The epistemology of a rule-based expert system: A framework for explanation. Artificial Intelligence, 20, 215-251.
16. Cleaves, D. A. (1987). Cognitive biases and corrective techniques: Proposals for improving elicitation procedures for knowledge-based systems. International Journal of Man-Machine Studies, 27, 155-166.
17. Cook, R. I., Woods, D. D., & McDonald, J. S. (1991). Human performance in anesthesia (Report CSEL91.003). Columbus: Ohio State University, Department of Industrial and Systems Engineering.
18. Cooke, N. J. (1992). Eliciting semantic relations for empirically derived networks. International Journal of Man-Machine Studies, 37, 721-750.
19. Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies, 41, 801-849.
20. Cooke, N. J., & Rowe, A. L. (1994). Evaluating mental model elicitation methods. In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting (pp. 261-265). Santa Monica, CA: Human Factors and Ergonomics Society.
21. Crandall, B. (1989). A comparative study of think-aloud and critical decision knowledge elicitation methods. SIGART Newsletter, 108, 144-146.
22. Crandall, B., & Calderwood, R. (1989). Clinical assessment skills of experienced neonatal intensive care nurses (Report Contract 1-R43-NR01911-01, National Center for Nursing, National Institutes of Health, Bethesda, MD). Fairborn, OH: Klein Associates, Inc.
23. Crandall, B., & Gamblian, V. (1991). Guide to early sepsis assessment in the NICU. Fairborn, OH: Klein Associates, Inc.
24. Crandall, B., & Getchell-Reiter, K. (1993). Critical decision method: A technique for eliciting concrete assessment indicators from the "intuition" of NICU nurses. Advances in Nursing Sciences, 16, 42-51.
25. Crandall, B., & Klein, G. A. (1988a). Critical cues for MI and cardiogenic shock symptoms. Unpublished manuscript.
26. Crandall, B., & Klein, G. A. (1988b). Key components of MIS performance (Report, Standard Register Company, Inc.). Fairborn, OH: Klein Associates, Inc.
27. Cullen, J., & Bryman, A. (1988). The knowledge acquisition bottleneck: Time for reassessment? Expert Systems, 5, 216-225.
28. Ericsson, K. A., & Simon, H. (1993). Protocol analysis: Verbal reports as data (2nd ed.). Cambridge: MIT Press.
29. Evans, J. St. B. T. (1988). The knowledge elicitation problem: A psychological perspective. Behavior and Information Technology, 7, 111-130.
30. Evans, J. St. B. T. (1989). Bias in human reasoning: Causes and consequences. Mahwah, NJ: Erlbaum.
31. Fernberger, S. W. (1937). A psychological cycle. American Journal of Psychology, 50, 207-217.
32. Fischhoff, B. (1989). Eliciting knowledge for analytical representation. IEEE Transactions on Systems, Man and Cybernetics, 19, 448-461.
33. Fisher, R. P., & Geiselman, R. E. (1992). Memory-enhancing techniques for investigative interviewing: The cognitive interview. Springfield, IL: Charles C Thomas.
34. Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51, 327-358.
35. Fleishman, E. A. (1975). Toward a taxonomy of human performance. American Psychologist, 30, 1127-1149.
36. Ford, K. M., & Adams-Webber, J. R. (1992). Knowledge acquisition and constructive epistemology. In R. Hoffman (Ed.), The cognition of experts: Psychological research and empirical AI (pp. 121-136). Mahwah, NJ: Erlbaum.
37. Ford, K. M., & Bradshaw, J. M. (Eds.). (1993). Knowledge acquisition as modeling. New York: Wiley.
38. Fraser, J. M., Smith, P. J., & Smith, J. W. (1992). A catalog of errors. International Journal of Man-Machine Studies, 37, 265-307.
39. Gammack, J. G., & Anderson, A. (1990, February). Constructive interaction in knowledge engineering. Expert Systems, 7, 19-26.
40. Gordon, S. E., & Gill, R. T. (1997). Cognitive task analysis. In C. Zsambok & G. Klein (Eds.), Naturalistic decision making (pp. 131-149). Mahwah, NJ: Erlbaum.
41. Gordon, S. E., Gill, R. T., & Dingus, T. A. (1987). Designing for the user: The role of human factors in expert system development. AI Applications in Environmental Science, 1, 35-46.
42. Gordon, S. E., Schmierer, K. A., & Gill, R. T. (1993). Conceptual graph analysis: Knowledge acquisition for instructional system design. Human Factors, 35, 459-481.
43. Hayes, P. I., Ford, K. M., & Agnew, N. (1994, Fall). On babies and bath water: A cautionary tale. AI Magazine, 15-26.
44. Hoffman, R. R. (1987, Summer). The problem of extracting the knowledge of experts from the perspective of experimental psychology. AI Magazine, 8, 53-67.
45. Hoffman, R. R. (1994). Constructivism versus realism or constructivism and realism? Journal of Experimental and Theoretical Artificial Intelligence, 6, 431-435.
46. Hoffman, R. R. (1997). A proposed paradigm for the experimental comparison of KA software tools. EVALKA [on-line], Computer Science and Engineering Department, University of New South Wales, Australia. http://www.cse.unsw.edu.au/∼timm/pub/eval.
47. Hoffman, R. R., Feltovich, P. J., & Ford, K. M. (1997). A general framework for conceiving of expertise and expert systems in context. In P. J. Feltovich, K. M. Ford, & R. R. Hoffman (Eds.), Expertise in context (pp. 543-580). Cambridge: MIT Press/AAAI Books.
48. Hoffman, R. R., Shadbolt, N. R., Burton, A. M., & Klein, G. (1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes, 62, 129-158.
49. Holsapple, C. W., & Raj, V. S. (1994). An exploratory study of two KA methods. Expert Systems, 11, 77-87.
50. Howard, G. S. (1994). Why do people say nasty things about self-reports? Journal of Organizational Behavior, 15, 399-404.
51. Hutton, R. J. B., Thordsen, M., & Mogford, R. (1997). Recognition primed decision model in air traffic controller error analysis. In R. S. Jensen & L. A. Rakovan (Eds.), Proceedings of the Ninth International Symposium on Aviation Psychology (pp. 721-726). Columbus: Ohio State University.
52. Jacob, V. S., Gaultney, L. D., & Salvendy, G. (1986). Strategies and biases in human decision making and their implications for expert systems. Behavior and Information Technology, 5, 119-140.
53. Jones, S. K., Frisch, D., & Yurak, T. J. (1996). The effect of outcome information on the recall and evaluation of individuals' own decisions. Unpublished manuscript, Ohio State University, Columbus, OH.
54. Kaempf, G. L., Thordsen, M. L., & Klein, G. (1991). Application of an expertise-centered taxonomy to training decisions (Tech. Report Contract MDA903-91-C-0050, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
55. Kaempf, G. L., Wolf, S., Thordsen, M. L., & Klein, G. (1992). Decision making in the AEGIS combat information center (Tech. Report Contract N66001-90-C-6023, Naval Command, Control, and Ocean Surveillance Center). Fairborn, OH: Klein Associates, Inc.
56. Kidd, A. L. (Ed.). (1987). Knowledge acquisition for expert systems: A practical handbook. New York: Plenum.
57. Kim, J., & Courtney, J. F. (1988). A survey of knowledge acquisition techniques and their relevance to managerial problem domains. Decision Support Systems, 4, 269-284.
58. Kinney, W. R., & Uecker, W. C. (1982). Mitigating the consequences of anchoring in auditor judgment. Accounting Review, 57, 55-69.
59. Klein, G. (1987). Applications of analogical reasoning. Metaphor and Symbolic Activity, 2, 201-218.
60. Klein, G. (1989). Utility of the critical decision method for eliciting knowledge from expert C debuggers (Report Purchase Order 339404, AT&T Bell Laboratories). Fairborn, OH: Klein Associates, Inc.
61. Klein, G. (1992). Using knowledge engineering to preserve corporate memory. In R. R. Hoffman (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 170-190). Mahwah, NJ: Erlbaum.
62. Klein, G. (1993a). Naturalistic decision making - Implications for design (Tech. Report Contract SOAR 93-01, Crew System Ergonomics Information Analysis Center). Fairborn, OH: Klein Associates, Inc.
63. Klein, G. (1993b). The recognition-primed decision model: Looking back, looking forward. In C. Zsambok & G. Klein (Eds.), Naturalistic decision making (pp. 285-292). Mahwah, NJ: Erlbaum.
64. Klein, G., & Brezovic, C. P. (1986). Design engineers and the design process: Decision strategies and human factors literature. In Proceedings of the Human Factors Society 30th Annual Meeting (pp. 771-775). Santa Monica, CA: Human Factors and Ergonomics Society.
65. Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid decision making on the fire ground. In Proceedings of the Human Factors Society 30th Annual Meeting (pp. 576-580). Santa Monica, CA: Human Factors and Ergonomics Society.
66. Klein, G., Calderwood, R., & MacGregor, D. (1989). Critical decision method of eliciting knowledge. IEEE Transactions on Systems, Man and Cybernetics, 19, 462-472.
67. Klein, G., & Hoffman, R. R. (1993). Seeing the invisible: Perceptual-cognitive aspects of expertise. In M. Rabinowitz (Ed.), Cognitive science foundations of instruction (pp. 203-226). Mahwah, NJ: Erlbaum.
68. Klein, G., Orasanu, J., Calderwood, R., & Zsambok, C. E. (Eds.). (1993). Decision making in action: Models and methods. Norwood, NJ: Ablex.
69. Klein, G. A., & Thordsen, M. L. (1991). Representing cockpit crew decision making. In R. S. Jensen (Ed.), Proceedings of the Sixth International Symposium on Aviation Psychology (pp. 1026-1031). Columbus: Ohio State University.
70. Klein, G., & Weitzenfeld, J. (1982). The use of analogues in comparability analysis. Applied Ergonomics, 13, 99-104.
71. Klinger, D. W., Andriole, S. J., Militello, L. G., Adelman, L., Klein, G., & Gomes, M. E. (1993). Designing for performance: A cognitive systems engineering approach to modifying an AWACS human-computer interface (Tech. Report Contract AL/CF-TR 1993-0093, Air Force Materiel Command). Fairborn, OH: Klein Associates, Inc.
72. Klinger, D. W., & Gomes, M. G. (1993). A cognitive systems engineering application for interface design. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (pp. 16-20). Santa Monica, CA: Human Factors and Ergonomics Society.
73. Kuipers, B., & Kassirer, J. P. (1987). Representing expert knowledge. In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 240-285). Cambridge, England: Cambridge University Press.
74. Kuipers, B., Moskowitz, A. J., & Kassirer, J. P. (1988). Critical decisions under uncertainty. Cognitive Science, 12, 177-210.
75. LaFrance, M. (1992). Excavation, capture, collection, and creation: Computer scientists' metaphors for eliciting human expertise. Metaphor and Symbolic Activity, 7, 135-156.
76. Lesgold, A. M., Rubinson, H., Feltovich, P., Glaser, R., Klopfer, D., & Wang, Y. (1988). Expertise in a complex skill: Diagnosing X-ray pictures. In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. 311-342). Mahwah, NJ: Erlbaum.
77. McGraw, K. L., & Scale, M. R. (1988). Knowledge elicitation with multiple experts: Considerations and techniques. Artificial Intelligence Review, 2, 31-44.
78. Meyer, M. A., & Payton, R. C. (1992). Towards an analysis and classification of approaches to knowledge acquisition from examination of textual metaphor. Knowledge Acquisition, 4, 347-369.
79. Miller, T. E., Kyne, M., & Lim, L. (1994). Blood banking: The need for training and evaluation (Report Grant 1R43-Hl49723-01, National Institutes of Health). Fairborn, OH: Klein Associates, Inc.
80. Miller, T. E., & Lim, L. S. (1993). Using knowledge engineering in the development of an expert system to assist targeteers in assessing battle damage and making weapons decisions for hardened structure targets (Tech. Report Contract DACA39-92-C-0050, U.S. Army Corps of Engineers). Fairborn, OH: Klein Associates, Inc.
81. Miller, T. E., Wolf, S. P., Thordsen, M. L., & Klein, G. (1992). A decision-centered approach to storyboarding anti-air warfare interfaces (Tech. Report Contract N66001-90-C-6023, Naval Command, Control, and Ocean Surveillance Center). Fairborn, OH: Klein Associates, Inc.
82. Monk, A. (1985). How and when to collect behavioral data. In A. Monk (Ed.), Fundamentals of human-computer interaction (pp. 69-79). San Diego, CA: Academic.
83. Mullin, T. M. (1989). Experts' estimation of uncertain quantities and its implications for knowledge acquisition. IEEE Transactions on Systems, Man and Cybernetics, 19, 616-625.
84. Neale, I. M. (1988). First generation expert systems: A review of knowledge acquisition methodologies. Knowledge Engineering Review, 3, 105-146.
85. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.
86. Noy, N. F., & Hafner, C. D. (1997, Fall). The state of the art in ontology design. AI Magazine, 53-74.
87. Olson, J. R., & Reuter, H. (1987). Extracting expertise from experts: Methods for knowledge acquisition. Expert Systems, 4, 152-168.
88. Orr, J. (1985). Social aspects of expertise (Report). Palo Alto, CA: Xerox PARC.
89. Patrick, J. (1992). Training: Research and practice. San Diego, CA: Academic.
90. Pew, R. W., Miller, D. C., & Feehrer, C. E. (1981). Evaluation of proposed control room improvements through analysis of critical operator decisions (Research Report NP-1982). Cambridge, MA: Bolt, Beranek, & Newman, Inc.
91. Prerau, D. (1989). Developing and managing expert systems: Proven techniques for business and industry. Reading, MA: Addison-Wesley.
92. Regoczei, S. B., & Hirst, G. (1992). Knowledge and knowledge acquisition in the computational context. In R. R. Hoffman (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 12-28). Mahwah, NJ: Erlbaum.
93. Rolandi, W. G. (1986). Knowledge engineering in practice. AI Expert, 1, 58-62.
94. Rowe, A. L., & Cooke, N. J. (1995). Measuring mental models: Choosing the right tools for the job. Human Resource Development Quarterly, 6, 243-255.
95. Rowe, A. L., Cooke, N. J., Hall, E. P., & Halgren, T. L. (1996). Toward an on-line knowledge assessment methodology: Building on the relationship between knowing and doing. Journal of Experimental Psychology: Applied, 2, 31-47.
96. Sanderson, P. M., Verhapge, A. G., & Fuld, R. B. (1989). Statespace and verbal protocol methods for studying the human operator in process control. Ergonomics, 32, 1343-1372.
97. Shadbolt, N., & Burton, A. M. (1989). The empirical study of knowledge elicitation techniques. SIGART Newsletter, 108, 21-33.
98. Shanteau, J. (1984). Some unasked questions about the psychology of expert decision makers. In M. E. El-Hawary (Ed.), Proceedings of the 1984 IEEE Conference on Systems, Man, & Cybernetics (pp. 23-45). New York: Institute of Electrical and Electronics Engineers.
99. Shanteau, J. (1992). Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes, 53, 252-266.
100. Shattuck, L., & Woods, D. D. (1994). The critical incident technique 40 years later. In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting (pp. 1080-1084). Santa Monica, CA: Human Factors and Ergonomics Society.
101. Shields, M. D., Solomon, I., & Waller, W. S. (1987). Effects of alternative sample space representations on the accuracy of auditors' uncertainty judgments. Accounting, Organizations, and Society, 12, 375-385.
102. Sleeman, D., & Shadbolt, N. R. (1996). Report of the ESPRC Workshop on Software Assisted Knowledge Acquisition. Aberdeen, Scotland: University of Aberdeen, Computing Science Department.
103. Spector, P. E. (1994). Using self-report questionnaires in OB research: A comment on the use of a controversial method. Journal of Organizational Behavior, 15, 385-392.
104. Taynor, J., Crandall, B., & Wiggins, S. (1987). The reliability of the critical decision method (Tech. Report Contract MDA903-86-C-0170, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
105. Thordsen, M. L., Militello, L. G., & Klein, G. A. (1992). Cognitive task analysis of critical team decision making during multiship engagements (Tech. Report Contract F33615-90-C-0005, U.S. Air Force Armstrong Laboratory). Fairborn, OH: Klein Associates, Inc.
106. Thordsen, M. L., Wolf, S. P., & Crandall, B. (1990). User defined requirements for the ISTARS self defense suite (Report Subcontract A62497L, UNISYS Corporation). Fairborn, OH: Klein Associates, Inc.
107. Tolcott, M. A., Marvin, F. F., & Lehner, P. E. (1989). Expert decision making in evolving situations. IEEE Transactions on Systems, Man and Cybernetics, 19, 606-615.
108. Tweney, R. D., Doherty, M. E., & Mynatt, C. R. (Eds.). (1981). On scientific thinking. New York: Columbia University Press.
109. Vicente, K. J., Christoffersen, K., & Pereklita, A. (1995). Supporting operator problem solving through ecological interface design. IEEE Transactions on Systems, Man and Cybernetics, 25, 529-545.
110. Waterman, D. A. (1986). A guide to expert systems. Reading, MA: Addison-Wesley.
111. Weitzenfeld, J. S., & Klein, G. (1979). Analogical reasoning as a discovery logic (Technical Report Contract F49620-79-C-0179, Air Force Office of Scientific Research). Fairborn, OH: Klein Associates, Inc.
112. Weitzenfeld, J. S., Riedl, T. R., Freeman, J. T., Klein, G., & Musa, J. (1991). Knowledge elicitation for software engineering expertise. In J. E. Tomayko (Ed.), Proceedings of the 1991 Software Engineering Education (SEI) Conference (pp. 283-296). New York: Springer-Verlag.
113. Wilson, T. D., & Schooler, J. W. (1991). Thinking too much: Introspection can reduce the quality of preferences and decisions. Journal of Personality and Social Psychology, 60, 181-192.
114. Wood, L. E., Davis, T. C., Clay, S. L., Ford, J. M., & Lammersen, S. (1995). Evaluation of interviewing methods and mediating representations for knowledge acquisition. International Journal of Expert Systems, 8, 1-23.
115. Wood, L. E., & Ford, K. M. (1993). Structuring interviews with experts during knowledge elicitation. In K. M. Ford & J. M. Bradshaw (Eds.), Knowledge acquisition as modeling (pp. 71-90). New York: Wiley.
116. Woods, D. D. (1993). Process-tracing methods for the study of cognition outside of the experimental psychology laboratory. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 228-251). Norwood, NJ: Ablex.
117. Zsambok, C. E., Beach, L. R., & Klein, G. (1992). A literature review of analytical and naturalistic decision making. Fairborn, OH: Klein Associates, Inc.
118. Zsambok, C. E., Kaempf, G. L., Crandall, B., & Kyne, M. (1996). A cognitive model of a prototype training program for OJT providers (Tech. Report Contract MDA903-93-C-0092, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
119. Zsambok, C. E., & Klein, G. (Eds.). (1997). Naturalistic decision making. Mahwah, NJ: Erlbaum.