The CACHE study: Group effects in computer-supported collaborative analysis

Computer Supported Cooperative Work

Volumn 17, Issue 4, 2008, Pages 353-393

  Convertino, Gregorio ^a   Billman, Dorrit ^b,d   Pirolli, Peter ^b   Massar, J P ^c   Shrager, Jeff ^d  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

^b PALO ALTO RESEARCH CENTER   (United States)

^c Stanford University ^*

^d STANFORD UNIVERSITY   (United States)

Author keywords

CACHE; Collaboration; CSCW; Group bias; Group decision making; Intelligence analysis

Indexed keywords

CACHE; COLLABORATION; COLLABORATIVE ANALYSIS; CSCW; CSCW SYSTEMS; DISTRIBUTED TEAMING; EXPERIMENTAL INVESTIGATIONS; GROUP BIAS; GROUP COMPOSITION; GROUP DECISION-MAKING; HOMOGENEOUS GROUPS; HUMAN COGNITION; INTELLIGENCE ANALYSIS; INTELLIGENCE ANALYSTS; RESEARCH AREA; WEB-BASED SOFTWARES;

SYSTEMATIC ERRORS; WORLD WIDE WEB;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 50849128190     PISSN: 09259724     EISSN: 15737551     Source Type: Journal    
DOI: 10.1007/s10606-008-9080-9     Document Type: Article

References (60)

1. Adelman, L., Tolcott, M.A. and T.A. Bresnick (1993): Examining the effect of information order on expert judgment, Organizational Behavior and Human Decision Processes, vol. 56, pp. 348-369.
2. D. Arnott 2006 Cognitive Biases and Decision Support Systems Development: A Design Science Approach Information Systems Journal 16 1 55 78 Arnott, D. (2006): Cognitive Biases and Decision Support Systems Development: A Design Science Approach. Information Systems Journal, vol. 16(1), pp. 55-78 doi: 10.1111/j.1365-2575.2006.00208.x.
3. I. Benbasat J. Lim 2000 Information Technology Support For Debiasing Group Judgments: An Empirical Evaluation Organizational Behavior and Human Decision Processes 83 167 183 Benbasat, I. and J. Lim (2000): Information Technology Support For Debiasing Group Judgments: An Empirical Evaluation. Organizational Behavior and Human Decision Processes, vol. 83, pp. 167-183 doi: 10.1006/obhd.2000.2905.
4. K. Benjamin 1990 Why We Still Use Our Heads Instead of Formulas: Toward an Integrative Approach Psychological Bulletin 107 3 296 310 Benjamin, K. (1990): Why We Still Use Our Heads Instead of Formulas: Toward an Integrative Approach. Psychological Bulletin, vol. 107(3), pp. 296-310 doi: 10.1037/0033-2909.107.3.296.
5. B.H. Bornstein A.C. Emler 2001 Rationality in Medical Decision Making: A Review of the Literature On Doctors' Decision-making Biases Journal of Evaluation in Clinical Practice 7 2 97 107 Bornstein, B.H. and A.C. Emler (2001): Rationality in Medical Decision Making: A Review of the Literature On Doctors' Decision-making Biases. Journal of Evaluation in Clinical Practice, vol. 7(2), pp. 97-107 doi: 10.1046/j.1365-2753.2001.00284.x.
6. Camerer, C.F. and E.J. Johnson (1991): The Process-Performance Paradox in Expert Judgment: How Can Experts Know so Much and Predict so Badly? In K.A. Ericsson and J. Smith (eds): Towards a General Theory of Expertise: Prospects and LimitsNew York: Cambridge University Press, pp. 195-217.
7. Card, S.K. (2005): The Science of Analytical Reasoning. In J.J. Thomas and K.A. Cook (eds): Illuminating the Path: the Research and Development Agenda for Visual AnalyticsIEEE CS: Los Alamitos, CA.
8. G.B. Chapman G.R. Bergus A.S. Elstein 1996 Order of Information Affects Clinical Judgment Journal of Behavioral Decision Making 9 3 201 211 Chapman, G.B., G.R. Bergus and A.S. Elstein (1996): Order of Information Affects Clinical Judgment. Journal of Behavioral Decision Making, vol. 9(3), pp. 201-211 doi: 10.1002/(SICI)1099-0771(199609)9:3<201::AID-BDM229>3.0.CO;2-J.
9. Cheikes, B.A., M.J. Brown, P.E. Lehner and L. Alderman (2004): Confirmation Bias in Complex Analyses. Technical Report No. MTR 04B0000017. Bedford, MA: MITRE.
10. Cho, H.-K. (2004): The effect of Delphi Structure on Small and Medium-sized Asynchronous Groups. Ph. D. Dissertation Thesis, New Jersey Institute of Technology, Information Systems Department.
11. Cho, H.-K. and M. Turoff (2001): Debiasing Group Judgments through Computerized Delphi Systems. In Proceedings of AMCIS 2001, August, Boston, MA.
12. Convertino, G., H.M. Mentis, P. Bhambare, C. Ferro, J.M. Carroll and M.B. Rosson (2008): Comparing Media in Emergency Planning. In Proceedings of the 5th International ISCRAM. Conference. Washington, DC, USA, May 4-7, 2008.
13. Cook, M.B. and H.S. Smallman (2007): Collaborative intelligence analysis: Debiasing through graphical evidence layout. In: Proceedings of the 51st Annual Meeting of the Human Factors and Ergonomics Society, Baltimore, MD, pp. 16-20.
14. Cummings, J.N. (2004): Work Groups, Structural Diversity, and Knowledge Sharing in a Global Organization. Management Science, vol. 50(3), pp. 352-364.
15. E.B. Davis R.H. Ashton 2002 Threshold Adjustment in Response to Asymmetric Loss Functions: The Case of Auditors' "Substantial Doubt" Thresholds Organizational Behavior and Human Decision Processes 89 1082 1099 Davis, E.B. and R.H. Ashton (2002): Threshold Adjustment in Response to Asymmetric Loss Functions: The Case of Auditors' "Substantial Doubt" Thresholds. Organizational Behavior and Human Decision Processes, vol. 89, pp. 1082-1099 doi: 10.1016/S0749-5978(02)00009-2.
16. A.R. Dennis J.S. Valacich 1993 Computer Brainstorms: More Heads are Better Than One The Journal of Applied Psychology 78 4 531 537 Dennis, A.R. and J.S. Valacich (1993): Computer Brainstorms: More Heads are Better Than One. The Journal of Applied Psychology, vol. 78(4), pp. 531-537 doi: 10.1037/0021-9010. 78.4.531.
17. Dennis, A.R., K.M. Hilmer and N.J. Taylor (1997): Information Exchange and Use in GSS and Verbal Group Decision Making: Effects of Minority Influence. Journal of Management Information Systems Archive, vol. 14(3), pp. 61-88.
18. Fjermestad, J. and S.R. Hiltz (2001): An Assessment of Group Support Systems Research: Methodology. Journal of Management Information Systems, vol. 15(3), pp. 7-149.
19. J.A. Fugelsang C.B. Stein A.E. Green K.N. Dunbar 2004 Theory and Data Interactions of the Scientific Mind: Evidence From the Molecular and the Cognitive Laboratory Canadian Journal of Experimental Psychology 58 2 86 95 Fugelsang, J.A., C.B. Stein, A.E. Green and K.N. Dunbar (2004): Theory and Data Interactions of the Scientific Mind: Evidence From the Molecular and the Cognitive Laboratory. Canadian Journal of Experimental Psychology, vol. 58(2), pp. 86-95 doi: 10.1037/h0085799.
20. B.R. Gallupe L.M. Bastianutti W.H. Cooper 1991 Unblocking Brainstorms The Journal of Applied Psychology 76 137 142 Gallupe, B.R., L.M. Bastianutti and W.H. Cooper (1991): Unblocking Brainstorms. The Journal of Applied Psychology, vol. 76, pp. 137-142 doi: 10.1037/0021-9010.76.1.137.
21. J.F. George K. Duffy M. Ahuja 2000 Countering the Anchoring and Adjustment Bias with Decision Support Systems Decision Support Systems 29 2 195 206 George, J.F., K. Duffy and M. Ahuja (2000): Countering the Anchoring and Adjustment Bias with Decision Support Systems. Decision Support Systems, vol. 29(2), pp. 195-206 doi: 10.1016/S0167-9236(00)00074-9.
22. Gettys, C.F., C. Kelly III and C.R. Peterson (1982): The Best-guess Hypothesis in Multistage Inference. In D. Kahneman, P. Slovic and A. Tversky (eds): Judgment Under Undertainty: Heuristics and BiasesNew York: Cambridge University Press, pp. 370-377.
23. D. Gigone R. Hastie 1996 The Impact of Information on Small Group Choice Journal of Personality and Social Psychology 72 132 140 Gigone, D. and R. Hastie (1996): The Impact of Information on Small Group Choice. Journal of Personality and Social Psychology, vol. 72, pp. 132-140 doi: 10.1037/0022-3514.72.1.132.
24. Heuer, Richards J. Jr. (1999): The Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency.
25. R. Hightower L. Sayeed 1995 The Impact of Computer-mediated Communication Systems on Biased Group Discussion Computers in Human Behavior 11 1 33 44 Hightower, R. and L. Sayeed (1995): The Impact of Computer-mediated Communication Systems on Biased Group Discussion. Computers in Human Behavior, vol. 11(1), pp. 33-44 doi: 10.1016/0747-5632(94)00019-E.
26. R.M. Hogarth Hillel J. Einhorn 1992 Order effects in belief updating: The belief-adjustment model Cognitive Psychology 24 1 1 55 Hogarth, R.M. and Hillel J. Einhorn (1992): Order effects in belief updating: The belief-adjustment model. Cognitive Psychology, vol. 24(1), pp. 1-55 doi: 10.1016/0010-0285(92) 90002-J.
27. Johnston, R. (2003): Reducing Analytic Error: Integrating Methodologists into Teams of Substantive Experts. Studies in Intelligence, vol. 47(1), pp. 57-65.
28. Johnston, R. (2005): Analytic Culture in the U.S. Intelligence Community: An Ethnographic Study. Washingon, DC: Central Intelligence Agency, Center for the Study of Intelligence.
29. Johnson, E.M. and S.M. Halpin (1974): Multistage Inference Models For Intelligence Analysis. Report AD 785-639. Arlington, VA: Army Research Institute for the Behavioral and Social Sciences(Distributed by US Department of Commerce, Springfield, VA).
30. Kahneman, D., P. Slovic and A. Tversky (eds) (1982): Judgment under Uncertainty: Heuristics and Biases. New York: Cambridge University Press.
31. Donald L. Keefer Craig W. Kirkwood James L. Corner 2004 Perspective on Decision Analysis Applications, 1990-2001 Decision Analysis 1 1 4 22 Keefer, Donald L., Craig W. Kirkwood and James L. Corner (2004): Perspective on Decision Analysis Applications, 1990-2001. Decision Analysis, vol. 1(1), pp. 4-22 doi: 10.1287/deca.1030.0004.
32. N.L. Kerr S.R. Tindale 2004 R.S. Group Performance and Decision Making Annual Review of Psychology 55 623 655 Kerr, N.L. and S.R. Tindale (2004): R.S. Group Performance and Decision Making. Annual Review of Psychology, vol. 55, pp. 623-655 doi: 10.1146/annurev.psych.55.090902.142009.
33. Norbert L. Kerr R.J. MacCoun G.P. Kramer 1996 Bias in Judgment: Comparing Individuals and Groups Psychological Review 103 4 687 719 Kerr, Norbert L., R.J. MacCoun and G.P. Kramer (1996): Bias in Judgment: Comparing Individuals and Groups. Psychological Review, vol. 103(4), pp. 687-719 doi: 10.1037/0033-295X.103.4.687.
34. Kerstholt, J.H. and J.L. Jackson (1998): Judicial Decision Making: Order of Evidence Presentation and Availability of Background Information. Applied Cognitive Psychology, vol. 12, pp. 445-454.
35. S. Kiesler L. Sproull 1992 Group Decision Making and Communication Technology Organizational Behavior and Human Decision Processes 52 96 123 Kiesler, S. and L. Sproull (1992): Group Decision Making and Communication Technology. Organizational Behavior and Human Decision Processes, vol. 52, pp. 96-123 doi: 10.1016/0749-5978(92)90047-B.
36. J. Klayman Y.-W. Ha 1987 Confirmation, Disconfirmation, and Information in Hypothesis Testing Psychological Review 94 2 211 228 Klayman, J. and Y.-W. Ha (1987): Confirmation, Disconfirmation, and Information in Hypothesis Testing. Psychological Review, vol. 94(2), pp. 211-228 doi: 10.1037/0033-295X.94.2.211.
37. Kraut, R.E. (2003): Applying Social Psychological Theory to the Problems of Group Work. In John M. Carroll (ed): HCI Models, Theories, and Frameworks: Toward a Multidisciplinary ScienceNew York: Morgan Kaufmann, pp. 325-356.
38. J.R. Larson Jr. P.G. Foster-Fishman C.B. Keys 1994 Discussion of Shared and Unshared Information in Decision-making Groups Journal of Personality and Social Psychology 67 446 461 Larson, J.R. Jr., P.G. Foster-Fishman and C.B. Keys (1994): Discussion of Shared and Unshared Information in Decision-making Groups. Journal of Personality and Social Psychology, vol. 67, pp. 446-461 doi: 10.1037/0022-3514.67.3.446.
39. L.-H. Lim I. Benbasat 1997 The Debiasing Role of Group Support Systems: An Experimental Investigation of the Representativeness Bias Int. J. Human-Computer Studies 47 453 471 Lim, L.-H. and I. Benbasat (1997): The Debiasing Role of Group Support Systems: An Experimental Investigation of the Representativeness Bias. Int. J. Human-Computer Studies, vol. 47, pp. 453-471 doi: 10.1006/ijhc.1997.0137.
40. R. Lipshitz G. Klein J. Orasanu E. Salas 2001 Taking stock of naturalistic decision making Journal of Behavioral Decision Making 14 5 331 352 Lipshitz, R., G. Klein, J. Orasanu and E. Salas (2001): Taking stock of naturalistic decision making. Journal of Behavioral Decision Making, vol. 14(5), pp. 331-352 doi: 10.1002/bdm.381.
41. Neale, D.C., J.M. Carroll and R.M. Beth (2004): Evaluating Computer-supported Cooperative WOrk: Models And Frameworks. In Proceedings of the ACM conference on Computer Supported Cooperative Work, Chicago, IL.
42. Nunamaker, J.F. Jr., A.R. Dennis, J.S. Valacich, D.R. Vogel and J.F. George (1991): Electronic Meeting Systems to Support Group Work. Communications of the ACM, vol. 34(7), pp. 40-61.
43. B.M. Perrin B.J. Barnett L. Walrath J.D. Grossman 2001 Information Order and Outcome Framing: An Assessment of Judgment Bias, In a Naturalistic Decision-Making Context Human Factors 43 2 227 238 Perrin, B.M., B.J. Barnett, L. Walrath and J.D. Grossman (2001): Information Order and Outcome Framing: An Assessment of Judgment Bias, In a Naturalistic Decision-Making Context. Human Factors, vol. 43(2), pp. 227-238 doi: 10.1518/001872001775900968.
44. Pirolli, P., T. Lee and Stuart K. Card (2004): Leverage Points for Analyst Technology Identified through Cognitive Task Analysis Technical Report. Palo Alto, CA: PARC.
45. Pirolli, P., L. Good, J. Heiser, J. Shrager and S. Hutchins (2005): UIR Technical Report. Palo Alto, CA: PARC.
46. P. Reagan-Cirincione 1994 Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology Organizational Behavior and Human Decision Processes 58 246 270 Reagan-Cirincione, P. (1994): Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology. Organizational Behavior and Human Decision Processes, vol. 58, pp. 246-270 doi: 10.1006/obhd.1994.1036.
47. Russell, D.M., M.J. Stefik, P. Pirolli, and S.K. Card (1993): The Cost Structure of Sensemaking. In: Proceedings of the SIGCHI conference on Human factors in computing systems, April 24-29, 1993, Amsterdam, The Netherlands, pp. 269-276.
48. J. Scholtz E. Morse P.S. Michelle 2006 Evaluation metrics and methodologies for user-centered evaluation of intelligent systems Interacting with Computers 18 1186 1214 Scholtz, J., E. Morse and P.S. Michelle (2006): Evaluation metrics and methodologies for user-centered evaluation of intelligent systems. Interacting with Computers, vol. 18, pp. 1186-1214 doi: 10.1016/j.intcom.2006.08.014.
49. S. Schultz-Hart D. Frey C. Lüthgens S. Moscovici 2000 Biased Information Search in Group Decision Making Journal of Personality and Social Psychology 78 4 655 669 Schultz-Hart, S., D. Frey, C. Lüthgens and S. Moscovici (2000): Biased Information Search in Group Decision Making. Journal of Personality and Social Psychology, vol. 78(4), pp. 655-669 doi: 10.1037/0022-3514.78.4.655.
50. J. Shanteau 1992 Competence in Experts: The Role of Task Characteristics Organizational Behavior and Human Decision Processes 53 252 266 Shanteau, J. (1992): Competence in Experts: The Role of Task Characteristics. Organizational Behavior and Human Decision Processes, vol. 53, pp. 252-266 doi: 10.1016/0749-5978(92)90064-E.
51. Shrager, J. (2005): CACHE; The Collaborative Analysis of Competing Hypotheses Environment [computer software]. Palo Alto, CA: Xerox Palo Alto Research Center.
52. Shrager, J., D. Billman, G. Convertino, J.P. Massar and P. Pirolli (in review): CACHE: Web-based support for distributed multi-stage inference.
53. Smallman, H.S. (2008): JIGSAW - Joint Intelligence Graphical Situation Awareness Web for Collaborative Intelligence Analysis. In M.P. Letsky, N.W. Warner, S.M. Fiore and C.A.P. Smith (eds.): Macrocognition in Teams: Theories and Methodologies. Ashgate Publishing Limited, Hampshire, England, pp. 321-337 (in press).
54. Stasser, G. and W. Titus (2003): Hidden Profiles: A Brief History. Psychological Inquiry, vol. 14(3-4), pp. 302-311.
55. S.G. Straus J.E. McGrath 1994 Does the Medium Matter? The Interaction of Task Type and Technology on Group Performance and Member Reactions The Journal of Applied Psychology 79 1 87 89 Straus, S.G. and J.E. McGrath (1994): Does the Medium Matter? The Interaction of Task Type and Technology on Group Performance and Member Reactions. The Journal of Applied Psychology, vol. 79(1), pp. 87-89 doi: 10.1037/0021-9010.79.1.87.
56. M.A. Tolcott F.F. Marvin P.E. Lehner 1989 Expert Decisionmaking in Evolving Situations IEEE Transact. on Systems. Man, and Cybernetics 19 3 606 615 Tolcott, M.A., F.F. Marvin and P.E. Lehner (1989): Expert Decisionmaking in Evolving Situations. IEEE Transact. on Systems. Man, and Cybernetics, vol. 19(3), pp. 606-615 doi: 10.1109/21.31066.
57. Trent, S.A., E.S. Patterson and D.D. Woods (2007): Challenges for Cognition. J. of Cognitive Engineering and Decision Making, vol. 1(1), pp. 75-97.
58. A. Tversky D. Kahneman 1974 Judgments under Uncertainty. Heuristics and Biases Science 185 1124 1131 Tversky, A. and D. Kahneman (1974): Judgments under Uncertainty. Heuristics and Biases. Science, vol. 185, pp. 1124-1131 doi: 10.1126/science.185.4157.1124.
59. D. van Knippenberg M.C. Schippers 2007 Work Group Diversity Annual Review of Psychology 58 515 541 van Knippenberg, D. and M.C. Schippers (2007): Work Group Diversity. Annual Review of Psychology, vol. 58, pp. 515-541 doi: 10.1146/annurev.psych.58.110405.085546.
60. P.C. Wason 1960 On the Failure to Eliminate Hypotheses in a Conceptual Task The Quarterly Journal of Experimental Psychology 12 129 140 Wason, P.C. (1960): On the Failure to Eliminate Hypotheses in a Conceptual Task. The Quarterly Journal of Experimental Psychology, vol. 12, pp. 129-140 doi: 10.1080/17470216008416717.