Measuring attributes of rigor in information analysis

Macrocognition Metrics and Scenarios: Design and Evaluation for Real-World Teams

Volumn , Issue , 2010, Pages 65-83

  Zelik, Daniel J ^a   Patterson, Emily S ^b   Woods, David D ^c  

^a OHIO STATE UNIVERSITY   (United States)

^b THE OHIO STATE UNIVERSITY WEXNER MEDICAL CENTER   (United States)

^c The Ohio State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84868516410     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (62)

1. Bennett, K. B., & Flach, J. M. (1992). Graphical Displays: Implications for divided attention, focused attention, and problem solving. Human Factors, 34(5), 513-533.
2. Berkowitz, B., & Goodman, A. (1991). Strategic intelligence for American national security. Princeton, NJ: Princeton University Press.
3. Betts, R. K. (1978). Analysis, war, and decision: Why intelligence failures are inevitable. World Politics, 31(2), 61-89.
4. Betts, R. K. (2002). Fixing intelligence. Foreign Affairs, 81(1), 43-59.
5. Butts, C. T. (2008). Social network analysis: A methodological introduction. Asian Journal of Social Psychology, 11(1), 13-41.
6. Clark, R. M. (2007). Intelligence analysis: A target-centric approach. Washington, DC: CQ Press.
7. Cooper, J. R. (2005, December). Curing analytic pathologies: Pathways to improved intelligence analysis (Center for the Study of Intelligence). Washington, DC: U.S. Government Printing Office.
8. Crippen, D. L., Daniel, C. C., Donahue, A. K., Helms, S. J., Livingstone, S. M., O'Leary, R., & Wegner, W. (2005). Annex A.2: Observations by Dr. Dan L. Crippen, Dr. Charles C. Daniel, Dr. Amy K. Donahue, Col. Susan J. Helms, Ms. Susan Morrisey Livingstone, Dr. Rosemary O'Leary, and Mr. William Wegner. In Return to Flight Task Group (Eds.), Return to Flight Task Group final report (pp. 188-207). Washington, DC: U.S. Government Printing Office.
9. Davies, D., & Dodd, J. (2002). Qualitative research and the question of rigor. Qualitative Health Research, 12, 279-289.
10. Davis, J. (2002). Improving CIA analytic performance: Strategic warning. The Sherman Kent Center for Intelligence Analysis Occasional Papers, 1(1). Retrieved October 27, 2008, from https://www.cia.gov/library/kent-center-occasional-papers/vol1no1.htm
11. De Keyser, V., & Woods, D. D. (1990). Fixation errors: Failures to revise situation assessment in dynamic and risky systems. In A. G. Colombo & A. Saiz de Bustamante (Eds.), Systems reliability assessment (pp. 231-251). Dordrecht, The Netherlands: Kluwer Academic.
12. Defense Science Board Task Force. (2003). Final report of the Defense Science Board Task Force on the role and status of DoD red teaming activities. Washington, DC: U.S. Government Printing Office.
13. Dekker, S. (2005). Ten questions about human error: A new view of human factors and system safety. Mahwah, NJ: Lawrence Erlbaum.
14. Dekker, S., & Hollnagel, E. (2004). Human factors and folk models. Cognition, Technology & Work, 6(2), 79-86.
15. Elm, W., Potter, S., Tittle, J., Woods, D., Grossman, J., & Patterson, E. (2005). Finding decision support requirements for effective intelligence analysis tools. In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting (pp. 297-301). Santa Monica, CA: Human Factors and Ergonomics Society.
16. Feder, S. (1987). FACTIONS and Politicon: New ways to analyze politics. Studies in Intelligence, 31(1), 41-57.
17. Folker, R. D. (2000). Intelligence analysis in theater joint intelligence centers: An experiment in applying structured methods (Joint Military Intelligence College Occasional Paper Number Seven). Washington, DC: U.S. Government Printing Office.
18. Gardiner, L. K. (1989). Dealing with intelligence-policy disconnects. Studies in Intelligence, 33(2), 1-9.
19. Guerlain, S. A., Smith, P.J., Obradovich, J. H., Rudmann, S., Strohm, P., Smith, J. W., Svirbely, J., & Sachs, L. (1999). Interactive critiquing as a form of decision support: An empirical evaluation. Human Factors, 41(1), 72-89.
20. Grossman, J., Woods, D. D., & Patterson, E. S. (2007). Supporting the cognitive work of information analysis and synthesis: A study of the military intelligence domain. In Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting (pp. 348-352). Santa Monica, CA: Human Factors and Ergonomics Society.
21. Heuer, R. J. (1987). Nosenko: Five paths to judgment. Studies in Intelligence, 31(3), 71-101.
22. Heuer, R. J. (1999). Psychology of intelligence analysis. Washington, DC: U.S. Government Printing Office.
23. Hong, L., & Page, S. E. (2004). Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences, 101, 16385-16389.
24. Johnson, P. E., Grazioli, S., Jamal, K., & Berryman, R. G. (2001). Detecting deception: Adversarial problem solving in a low base-rate world. Cognitive Science, 25(3), 355-392.
25. Johnston, R. (2005). Analytic culture in the US Intelligence Community: An ethnographic study. Washington, DC: U.S. Government Printing Office.
26. Kent, S. (1966). Strategic intelligence for American world policy. Princeton, NJ: Princeton University Press.
27. Klein G. A. (2010). Macrocognitive measures for evaluating cognitive work. In E. S. Patterson & J. Miller (Eds.), Macrocognition metrics and scenarios: Design and evaluation for real-world teams. Aldershot, UK: Ashgate Publishing.
28. Klein, G., Moon, B., & Hoffman, R. R. (2006a). Making sense of sensemaking 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 70-73.
29. Klein, G., Moon, B., & Hoffman, R. R. (2006b). Making sense of sensemaking 2: A macrocognitive model. IEEE Intelligent Systems, 21(5), 88-92.
30. th International Conference on Naturalistic Decision Making. Mahwah, NJ: Erlbaum.
31. Klein, G., Ross, K. G., Moon, B., Klein, D. E., Hoffman, R. R., & Hollnagel, E. (2003). Macrocognition. IEEE Intelligent Systems, 18(3), 81-85.
32. Krebs, V. (2002, April 1). Uncloaking terrorist networks. First Monday, 7(4). Retrieved September 30, 2009, from http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/941/863
33. Krizan, L. (1999). Intelligence essentials for everyone (Joint Military Intelligence College Occasional Paper Number Six). Washington, DC: U.S. Government Printing Office.
34. Mangio, C. A., & Wilkinson, B. J. (2008, March). Intelligence analysis: Once again. Paper presented at the International Studies Association 2008 Annual Convention, San Francisco, CA.
35. Marrin, S. (2007). Intelligence analysis: Structured methods or intuition? American Intelligence Journal, 25(1), 7-16.
36. McLaughlin, J. (2008). Serving the national policymaker. In R. Z. George & J. B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles and innovations (pp. 71-81). Washington, DC: Georgetown University Press.
37. Medina, C. A. (2002). The coming revolution in intelligence analysis: What to do when traditional models fail. Studies in Intelligence, 46(3). Retrieved February 10, 2007, from https://www.cia.gov/csi/studies/vol46no3/article03.html
38. Military Operations Research Society. (2006, October 3). MORS special meeting terms of reference. Paper presented at Bringing Analytical Rigor to Joint Warfighting Experimentation, Norfolk, VA. Retrieved December 11, 2008, from http://www.mors.org/meetings/bar/tor.htm
39. Miller, J. E., & Patterson, E. S. (2008). Playback technique using a temporally sequenced cognitive artifact for knowledge elicitation. In Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting (pp. 523-527). Santa Monica, CA: Human Factors and Ergonomics Society.
40. Miller, J. E., Patterson, E. S., & Woods, D. D. (2006). Elicitation by critiquing as a cognitive task analysis methodology. Cognition, Technology & Work, 8(2), 90-102.
41. Moon, I., & Carley, K. (2007). Modeling and simulating terrorist networks in social and geospatial dimensions. IEEE Intelligent Systems, 22(5), 40-49.
42. Moore, D. T. (2007). Critical thinking and intelligence analysis (Joint Military Intelligence College Occasional Paper Number Fourteen). Washington, DC: U.S. Government Printing Office.
43. Moore, D. T., & Reynolds, W. N. (2006). So many ways to lie: The complexity of denial and deception. Defense Intelligence Journal, 15(2), 95-116.
44. Morse, J. M. (2004). Preparing and evaluating qualitative research proposals. In C. Seale, G. Gobo, J. F. Giubrium, & D. Silverman (Eds.), Qualitative research practice (pp. 493-503). London: SAGE.
45. Patterson, E. S., Roth, E. M., & Woods, D. D. (2001). Predicting vulnerabilities in computer-supported inferential analysis under data overload. Cognition, Technology & Work, 3, 224-237.
46. Patterson, E. S., Zelik, D., McNee, S., & Woods, D. D. (2008). Insights from applying rigor metric to healthcare incident investigations. In Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting (pp. 1766-1770). Santa Monica, CA: Human Factors and Ergonomics Society.
47. Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd edn). Thousand Oaks, CA: Sage.
48. Pherson, R. (2005, March). Overcoming analytic mindsets: Five simple techniques. Paper presented at the annual meeting on Emerging Issues in National and International Security, Washington, DC.
49. Saltelli, A., Chan, K., & Scott, E. M. (Eds.) (2000). Sensitivity analysis. New York: John Wiley & Sons.
50. Sandelowski, M. (1986). The problem of rigor in qualitative research. Advances in Nursing Science, 8(3), 27-37.
51. Sandelowski, M. (1993). Rigor or rigor mortis: The problem of rigor in qualitative research revisited. Advances in Nursing Science, 16(2), 1-8.
52. Schraagen, J. M., Klein, G., & Hoffman, R. R. (2008). The macrocognition framework of naturalistic decision making. In J. M. Schraagen, L. G. Militello, T. Ormerod, & R. Lipshitz (Eds.), Naturalistic decision making and macrocognition (pp. 3-26). Aldershot, UK: Ashgate.
53. Schum, D. A. (1987). Evidence and inference for the intelligence analyst (Vol. 1). Lanham, MD: University Press of America.
54. Serra, J. (2008). Proactive intelligence. Futures, 40(7), 664-673.
55. Smith, M. W., Branlat, M., Stephens, R. J., & Woods, D. D. (2008, April). Collaboration support via analysis of factions. Papers presented at the NATO Research and Technology Organisation Human Factors and Medicine Panel Symposium, Copenhagen, Denmark.
56. Tenet, G. (1998). Unclassified recommendations of the Jeremiah report. Retrieved December 26, 2008, from http://ftp.fas.org/irp/cia/product/jeremiah-decl.pdf
57. Trent, S. A., Patterson, E. S., & Woods, D. D. (2007). Challenges for cognition in intelligence analysis. Journal of Cognitive Engineering and Decision Making, 1(1), 75-97.
58. Treverton, G. F., & Gabbard, C. B., (2008). Assessing the tradecraft of intelligence analysis. (RAND National Security Research Division). Retrieved October 21, 2008 from http://www.rand.org/pubs/technical_reports/2008/RAND_TR293.pdf
59. Woods, D. D., & Hollnagel, E. (2006). Joint cognitive systems: Patterns in cognitive systems engineering. New York, NY: Taylor & Francis.
60. Woods, D. D., Patterson, E. S., & Roth, E. M. (2002). Can we ever escape from data overload? A cognitive systems diagnosis. Cognition, Technology & Work, 4(1), 22-36.
61. Zelik, D. J., Patterson, E. S., & Woods, D. D. (2007). Judging sufficiency: How professional intelligence analysts assess analytical rigor. In Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting (pp. 318-322). Santa Monica, CA: Human Factors and Ergonomics Society.
62. Zelik, D., Woods, D. D., & Patterson, E. S. (2009). The supervisor's dilemma: Judging when analysis is sufficiently rigorous. Paper presented at the CHI 2009 Sensemaking Workshop, Boston, MA.