Rate or trade? Identifying winning ideas in open idea sourcing

Information Systems Research

Volumn 27, Issue 1, 2016, Pages 27-48

  Blohm, Ivo ^a   Riedl, Christoph ^b,c   Füller, Johann ^d   Leimeister, Jan Marco ^a,e  

^a UNIVERSITY OF ST GALLEN   (Switzerland)

^b NORTHEASTERN UNIVERSITY   (United States)

^c HARVARD UNIVERSITY   (United States)

^d INNSBRUCK UNIVERSITY SCHOOL OF MANAGEMENT   (Austria)

^e UNIVERSITY OF KASSEL   (Germany)

Author keywords

Computer mediated communication and collaboration; Crowdsourcing; Decision support systems; Idea evaluation; Preference markets; Rating scales

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMMERCE; CROWDSOURCING; DECISION MAKING; DECISION SUPPORT SYSTEMS; RATING;

BEHAVIORAL DECISION MAKINGS; COMPUTER MEDIATED COMMUNICATION; DECISION MAKING SUPPORT; IDEA EVALUATIONS; PERCEPTUAL DIFFERENCE; PREFERENCE MARKETS; RANDOMIZED EXPERIMENTS; RATING SCALE;

QUALITY CONTROL;

EID: 84962861117     PISSN: 10477047     EISSN: 15265536     Source Type: Journal    
DOI: 10.1287/isre.2015.0605     Document Type: Article

References (98)

1. Afuah A, Tucci C (2012) Crowdsourcing as a solution to distant search. Acad. Management Rev. 37(3):355-375.
2. Amabile TM (1996) Creativity in Context. Update to Social Psychology of Creativity (Westview Press, Boulder, CO).
3. Arrow KJ, Forsythe R, Gorham M, Hahn R, Ledyard JO, Levmore S, Litan R, et al. (2008) The promise of prediction markets. Sci. 320(5878):877-878.
4. Bagozzi RP, Yi Y (1988) On the evaluation of structural equation models. J. Acad. Marketing Sci. 16(1):74-94.
5. Baron RM, Kenny DA (1986) The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. J. Personality Soc. Psych. 51(6): 1173-1182.
6. Berg H, Proebsting TA (2009) Hanson's automated market maker. J. Prediction Markets 3(1):45-59.
7. Blohm I, Leimeister JM, Krcmar H (2013) Crowdsourcing: How to benefit from (too) many great ideas. MIS Quart. Executive 12(4):199-211.
8. Blohm I, Bretschneider U, Leimeister JM, Krcmar H (2011a) Does collaboration among participants lead to better ideas in IT-based idea competitions? An empirical investigation. Internat. J. Networking Virtual Organ. 9(2):106-122.
9. Blohm I, Riedl C, Leimeister JM, Krcmar H (2011b) Idea evaluation mechanisms for collective intelligence in open innovation communities: Do traders outperform raters? Proc. Internat. Conf. Inform. Systems (ICIS'11), Shanghai.
10. Boudreau K, Guinan E, Lakhani K, Riedl C (2016) Looking across and looking beyond the knowledge frontier: Intellectual distance and resource allocation in science. Management Sci. ePub ahead of print January 8, http://dx.doi.org/10.1287/mnsc.2015.2285.
11. Boudreau K, Gaule P, Lakhani KR, Riedl C, Woolley AW (2014) From crowds to collaborators: Initiating effort and catalyzing interactions among online creative workers. Working Paper 14-060, Harvard Business School, Cambridge, MA.
12. Campbell DT, Fiske DW (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psych. Bull. 56(2): 81-105.
13. Chalmers PA (2003) The role of cognitive theory in human-computer interface. Comput. Human Behav. 19(5):593-607.
14. Chen L, Goes P, Marsden JR, Zhang Z (2009) Design and use of preference markets for evaluation of early stage technologies. J. Management Inform. Systems 26(3):45-70.
15. Cohen J, Cohen P, West SG, Aiken LS (2003) Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (Lawrence Erlbaum Associates, Mahwah, NJ).
16. Compeau D, Marcolin BL, Kelley H, Higgins C (2012) Generalizability of information systems research using students-A reflection on our practices and recommendations for future research. Inform. Systems Res. 23(4):1093-1109.
17. Cowgill B, Zitzewitz E (2015) Corporate prediction markets: Evidence from Google, Ford, and Firm X. Rev. Econom. Stud. 86(4): 1309-1341.
18. Daft RL, Macintosh NB (1981) A tentative exploration into the amount and equivocality of information processing in organizational work units. Admin. Sci. Quart. 26(2):207-224.
19. Dahan E, Soukhoroukova A, Spann M (2010) New product development 2.0: Preference markets how scalable securities markets identify winning product concepts and attributes. J. Product Innovation Management 27(2):937-954.
20. Dahan E, Kim AJ, Lo AW, Poggio T, Chan N (2011) Securities trading of concepts (STOC). J. Marketing Res. 48(3):497-517.
21. Dani V, Madani O, Pennock D, Sanghai S, Galebach B (2006) An empirical comparison of algorithms for aggregating expert predictions. Dechter R, Richardson T, eds. Proc. 22nd Conf. Uncertainty Artificial Intelligence (UAI) (AUAI Press, Arlington, VA), 106-113.
22. Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quart. 13(3):319-340.
23. Dean DL, Hender JM, Rodgers TL, Santanen EL (2006) Identifying quality, novel, and creative ideas: Constructs and scales for idea evaluation. J. Assoc. Inform. Systems 7(10):646-698.
24. Dennis AR, Wixom BH (2001) Investigating the moderators of the group support systems use with meta-analysis. J. Management Inform. Systems 18(3):235-257.
25. Di Gangi PM, Wasko MM (2009) Steal my idea! Organizational adoption of user innovations from a user innovation community: A case study of Dell IdeaStorm. Decision Support Systems 48(1):303-312.
26. Dimoka A, Banker RD, Benbasat I, Davis FD, Dennis AR, Gefen D, Gupta A, et al. (2012) On the use of neurophysiological tools in IS research: Developing a research agenda for NeuroIS. MIS Quart. 36(3):679-702.
27. DuBay WH (2004) The Principles of Readability (Impact Information, Costa Mesa, CA).
28. Easley D, Kleinberg J (2010) Networks, Crowds, and Markets: Reasoning About a Highly Connected World (Cambridge University Press, Cambridge, MA).
29. Einhorn HJ, Hogarth RM (1981) Behavioral decision theory: Processes of judgment and choice. J. Accounting Res. 19(1):1-31.
30. Estellés-Arolas E, González-Ladrón-de-Guevara F (2012) Towards an integrated crowdsourcing definition. J. Inform. Sci. 38(2):189-200.
31. Forman C, Zeebroeck NV (2012) From wires to partners: How the Internet has fostered R&D collaborations within firms. Management Sci. 58(8):1549-1568.
32. Fornell C, Larcker DF (1981) Evaluating structural equation models with unobservable variables and measurement error. J. Marketing Res. 18(2):39-50.
33. Franke N, Shah S (2003) How communities support innovative activities: An exploration of assistance and sharing among end-users. Res. Policy 32(1):157-178.
34. Füller J, Mühlbacher H, Matzler K, Jawecki G (2009) Consumer empowerment through Internet-based co-creation. J. Management Inform. Systems 26(3):71-102.
35. Gefen D, Straub D (2000) The relative importance of perceived ease of use in IS adoption: A study of e-commerce adoption. J. Assoc. Inform. Systems 1(8):1-28.
36. Ghose A, Ipeirotis PG (2011) Estimating the helpfulness and economic impact of product reviews: Mining text and reviewer characteristics. IEEE Trans. Knowl. Data Engrg. 23(10):1498-1512.
37. Girotra K, Terwiesch C, Ulrich KT (2010) Idea generation and the quality of the best idea. Management Sci. 56(4):591-605.
38. Goodwin P, Lawton R (1999) On the asymmetry of the symmetric MAPE. Internat. J. Forecasting 15(4):405-408.
39. Haerem T, Rau D (2007) The influence of degree of expertise and objective task complexity on perceived task complexity and performance. J. Appl. Psych. 92(5):1320-1331.
40. Hair JF, Black WC, Babin BJ, Anderson RE (2010) Multivariate Data Analysis (Pearson, Boston).
41. Hanson R (2003) Combinatorial information market design. Inform. Systems Frontiers 5(1):107-119.
42. Hammedi W, van Riel ACR, Sasovova Z (2011) Antecedents and consequences of reflexivity in new product idea screening. J. Product Innovation Management 28(5):662-679.
43. Hayes AF (2013) Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach (Guilford Press, New York).
44. Hildebrand C, Häubl G, Herrmann A, Landwehr JR (2013) When social media can be bad for you: Community feedback stifles consumer creativity and reduces satisfaction with self-designed products. Inform. Systems Res. 24(1):14-29.
45. Jones Q, Ravid G, Rafaeli S (2004) Information overload and the message dynamics of online interaction spaces: A theoretical model and empirical exploration. Inform. Systems Res. 15(2): 194-210.
46. Kalyuga S (2011) Cognitive load theory: How many types of load does it really need? Ed. Psych. Rev. 23(1):1-19.
47. Kamp G, Koen PA (2009) Improving the idea screening process within organizations using prediction markets: A theoretical perspective. J. Prediction Markets 3(2):39-64.
48. Karahanna E, Agarwal R, Angst CM (2006) Reconceptualizing compatibility beliefs in technology acceptance research. MIS Quart. 30(4):781-804.
49. King R, Smith V, Williams A, van Boening M (1993) The robustness of bubbles and crashes in experimental stock markets. Day RH, Chen P, eds. Nonlinear Dynamics and Evolutionary Economics (Oxford, New York), 183-200.
50. Klare GR (1963) Measurement of Readability (Iowa State University Press, Ames, IA).
51. Kornish LJ, Ulrich KT (2014) The importance of the raw idea in innovation: Testing the sow's ear hypothesis. J. Marketing Res. 51(1):14-26.
52. Kühberger A, Gradl P (2013) Choice, rating, and ranking: Framing effects with different response modes. J. Behav. Decision Making 26(2):109-117.
53. LaComb AC, Barnett A, Pan Q (2007) The imagination market. Inform. Systems Frontiers 9(2-3):245-256.
54. Lakhani KR, Boudreau KJ, Loh P-R, Backstrom L, Baldwin C, Lonstein E, Lydon M, MacCormack A, Arnaout RA, Guinan EC (2013) Prize-based contests can provide solutions to computational biology problems. Nature Biotechnology 31(7):108-111.
55. Leimeister JM, Huber M, Bretschneider U, Krcmar H (2009) Leveraging crowdsourcing-Activation-supporting components for IT-based idea competitions. J. Management Inform. Systems 26(1):197-224.
56. Limayem M, DeSanctis G (2000) Providing decisional guidance for multicriteria decision making in groups. Inform. Systems Res. 11(4):386-401.
57. Lorenz J, Rauhut H, Schweitzer F, Helbing D (2011) How social influence can undermine the wisdom of crowd effect. Proc. Nat. Acad. Sci. 108(22):9020-9025.
58. Luckner S, Weinhardt C (2007) How to pay traders in information markets: Results from a field experiment. J. Prediction Markets 1(2):147-156.
59. Lyytinen K, King JL (2004) Nothing at the center?: Academic legitimacy in the information systems field. J. Assoc. Inform. Systems 5(6):220-247.
60. Mayer RE, Moreno R (2003) Nine ways to reduce cognitive load in multimedia learning. Ed. Psych. 38(1):43-52.
61. Michalke ME (2015) koRpus: An R package for text analysis. Accessed February 17, 2016, https://cran.r-project.org/web/packages/koRpus/index.html.
62. Moore WL (2004) A cross-validity comparison of rating-based and choice-based conjoint analysis models. Internat. J. Res. Marketing 21(3):299-312.
63. Morgan DL (1996) Focus Groups as Qualitative Research (Sage, Thousands Oaks, CA).
64. Muchnik L, Aral S, Taylor SJ (2013) Social influence bias: A randomized experiment. Sci. 341(6146):647-651.
65. Muller D, Judd CM, Yzerbyt VY (2005) When moderation is mediated and mediation is moderated. J. Personality Soc. Psych. 89(6): 852-863.
66. Nowlis SM, Simonson I (1997) Attribute-task compatibility as a determinant of consumer preference reversals. J. Marketing Res. 34(2):205-218.
67. O'Leary DE (2013) Internal corporate prediction markets: "From each according to his bet." Internat. J. Accounting Inform. Systems 14(1):89-103.
68. Olshavsky RW, Spreng RA (1996) An exploratory study of the innovation evaluation process. J. Product Innovation Management 13(6):512-529.
69. Ozer M (2005) Factors which influence decision making in new product evaluation. Eur. J. Oper. Res. 163(3):784-801.
70. Paas F, Tuovinen J, Tabers H, van Gerven P (2003) Cognitive load measurement as a means to advance cognitive load theory. Ed. Psych. 38(1):63-71.
71. Paas FG, Van Merriënboer JJ (1994) Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. J. Ed. Psych. 86(1):122-133.
72. Payne JW, Bettman JR, Coupey E, Johnson EJ (1992) A constructive process view of decision making: Multiple strategies in judgment and choice. Acta Psych. 80(1-3):107-141.
73. Preacher KJ, Rucker DD, Hayes AF (2007) Addressing moderated mediation hypotheses: Theory, methods, and prescriptions. Multivariate Behav. Res. 42(1):185-227.
74. Riedl C, Blohm I, Leimeister JM, Krcmar H (2010) Rating scales for collective intelligence in innovation communities: Why quick and easy decision making does not get it right. Proc. Internat. Conf. Inform. Systems (ICIS'10), St. Louis, 1-21.
75. Riedl C, Blohm I, Leimeister JM, Krcmar H (2013) The effect of rating scales on decision quality and user attitudes in online innovation communities. Internat. J. Electronic Commerce 17(3):7-37.
76. Runco MA, Smith WR (1992) Interpersonal and intrapersonal evaluations of creative ideas. Personality Individual Differences 13(3): 295-302.
77. Saadé GR, Otrakji CA (2007) First impressions last a lifetime: Effect of interface type on disorientation and cognitive load. Comput. Human Behav. 23(1):525-535.
78. Servan-Schreiber E, Wolfers J, Pennock DM, Galebach B (2004) Prediction markets: Does money matter? Electronic Markets 14(3): 243-251.
79. Shaft TM, Vessey I (2006) The role of cognitive fit in the relationship between software comprehension and modification. MIS Quart. 30(1):29-55.
80. Shrout PE, Bolger N (2002) Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psych. Methods 7(4):422-445.
81. Slamka C, Jank W, Skiera B (2012) Second-generation prediction markets for information aggregation: A comparison of payoff mechanisms. J. Forecasting 31(1):469-489.
82. Soukhoroukova A, Spann M, Skiera B (2012) Sourcing, filtering, and evaluating new product ideas: An empirical exploration of the performance of idea markets. J. Product Innovation Management 29(1):100-112.
83. Spann M, Skiera B (2003) Internet-based virtual stock markets for business forecasting. Management Sci. 49(10):1310-1326.
84. Spears B, LaComb C, Interrante J, Barnett J, Senturk-Dogonaksoy D (2009) Examining trader behavior in idea markets: An implementation of GE's imagination markets. J. Prediction Markets 3(1):17-39.
85. Speier C (2006) The influence of information presentation formats on complex task decision-making performance. Internat. J. Human-Comput. Stud. 64(11):1115-1131.
86. Sujan M (1985) Consumer knowledge: Effects on evaluation strategies mediating consumer judgments. J. Consumer Res. 12(1):31-46.
87. Sweller J (1988) Cognitive load during problem solving: Effects on learning. Cognitive Sci. 12(2):257-285.
88. Sweller J, van Merrienboer J, Paas F (1998) Cognitive architecture and instructional design. Ed. Psych. Rev. 10(3):251-296.
89. Tan H-T, Ying Wang E, Zhou BO (2014) When the use of positive language backfires: The joint effect of tone, readability, and investor sophistication on earnings judgments. J. Accounting Res. 52(1):273-302.
90. Tilson D, Lyytinen K, Sörensen C (2010) Digital infrastructures: The missing IS research agenda. Inform. Systems Res. 21(4):748-759.
91. Todd P, Benbasat I (1999) Evaluating the impact of DSS, cognitive effort, and incentives on strategy selection. Inform. Systems Res. 10(4):356-374.
92. Van Merrienboer JJG, Kirschner PA, Kester L (2003) Taking the load off a learner's mind: Instructional design for complex learning. Ed. Psych. 38(1):5-13.
93. Vessey I, Dennis G (1991) Cognitive fit: An empirical study of information acquisition. Inform. Systems Res. 2(1):63-84.
94. Von Hippel E (2005) Democratizing Innovation (MIT Press, Cambridge, MA).
95. Witten IH, Frank E, Hall MA (2011) Data Mining: Practical Machine Learning Tools and Techniques (Morgan Kaufmann, Burlington, MA).
96. Wolfers J, Zitzewitz E (2004) Prediction markets. J. Econom. Perspective 18(2):107-126.
97. Woolley A, Chabris CF, Pentland A, Hashmi N, Malone TW (2010) Evidence for a collective intelligence factor in the performance of human groups. Sci. 330(6004):686-688.
98. Zhao Y, Zhu Q (2014) Evaluation on crowdsourcing research: Current status and future direction. Inform. Systems Frontiers 16(3): 417-434.