An internet-scale idea generation system

ACM Transactions on Interactive Intelligent Systems

Volumn 3, Issue 1, 2013, Pages

  Yu, Lixiu ^a   Nickerson, Jeffrey V ^b  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b STEVENS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Codesign; Creativity; Crowdsourcing; Evolutionary algorithms

Indexed keywords

EVOLUTIONARY ALGORITHMS; INTERNET;

AUTOMATED ALGORITHMS; CO-DESIGNS; COMBINED DESIGN; CREATIVITY; CROWDSOURCING; DESIGN PROBLEMS; PARTICIPATORY DESIGN; SEQUENTIAL COMBINATION;

DESIGN;

EID: 84983574295     PISSN: 21606455     EISSN: 21606463     Source Type: Journal    
DOI: 10.1145/2448116.2448118     Document Type: Article

References (108)

1. ALBINSSON, L., LIND, M., AND FORSGREN, O. 2007. Co-Design: An approach to border crossing, network innovation. In Expanding the Knowledge Economy: Issues, Applications, Case Studies, P. Cunningham and M. Cunningham, Eds., IOS Press, Amsterdam, 977-983.
2. AMABILE, T. 1996. Creativity in Context. Westview Press, Boulder, CO.
3. AMABILE, T. 1998. How to kill creativity. Harvard Bus. Rev. 76, 5, 77-87.
4. AMAZON MECHANICAL TURK. 2010. https://www.mturk.com/mturk/welcome.
5. AMBURGEY, T. L. AND RAO, H. 1996. Organizational ecology: Past, present, and future directions. Acad. Manage. J. 39, 5, 1265-1286.
6. rd International Conference on Design Computing and Cognition (DCC'08). 397-416.
7. BEDERSON, B. B., HU, C., AND RESNIK, P. 2010. Translation by iterative collaboration between monolingual users. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP'10). 54-55.
8. BENKLER, Y. 2006. The Wealth of Networks: How Social Production Transforms Markets and Freedom. Yale University Press, New Haven, CT.
9. BENTLEY, P. J. 1999. Evolutionary Design by Computers. Morgan Kaufmann, San Francisco, CA.
10. rdAnnual ACM Symposium on User Interface Software and Technology (UIST'10). ACM Press, New York, 313-322.
11. BEWS, J. W. 1935. Human Ecology. Oxford University Press, London.
12. BIEWALD, L. AND ALLICK, M. 2011. Massive multiplayer human computation for fun, money, and survival. In Workshop on Crowdsourcing and Human Computation, Human Factors in Computing Systems (CHI'11).
13. BLACKMORE, S. 1999. The Meme Machine. Oxford Press.
14. BODEN, M. A. 1996. Dimensions of Creativity. The MIT Press, Cambridge, MA.
15. BROOKS, F. P. 2010. The Design of Design. Addison-Wesley, Boston, MA.
16. BUCKLEY, P. AND CARTER, M. 2004. A formal analysis of knowledge combination in multinational enterprises. J. Int. Bus. Stud. 35, 5, 371-384. (Pubitemid 39443154)
17. CHENG, C. D. AND KOSORUKOFF, A. 2004. Interactive one-max problem allows to compare the performance of interactive and human-based genetic algorithms. In Proceedings of Genetic and Evolutionary Computation (GECCO'04). 983-993. (Pubitemid 39747290)
18. CHESBROUGH, H., VANHAVERBEKE, W., AND WEST, J. 2006. Open Innovation: Researching a New Paradigm. Oxford University Press, London.
19. COOPER, S., KHATIB, F., TREUILLE, A., BARBERO, J., LEE, J., BEENEN, M., LEAVER-FAY, A., BAKER, D., AND POPOVIC, Z. 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307, 756-760.
20. DAVIS, L. 1987. Genetic Algorithms and Simulated Annealing. Morgan Kaufman, San Fransisco, CA.
21. DAWKINS, R. AND PYLE, L. 1996. The Blind Watchmaker. W. W. Norton and Company, New York.
22. DEB, K. 2001. Multi-Objective Optimization Using Evolutionary Algorithms. Wiley, Chichester, UK.
23. DENNIS, A. R., VALACICH, J. S., CONNOLLY, T., AND WYNNE, B. E. 1996. Process structuring in electronic brain-storming. Inf. Syst. Res. 7, 2, 268-277.
24. DIEHL, M. AND STROEBE, W. 1987. Productivity loss in brainstorming groups: Toward the solution of a riddle. J. Pers. Soc. Psychol. 53, 3, 497-509.
25. DOW, S., KULKARNI, A., BUNGE, B., NGUYEN, T., KLEMMER, S., AND HARTMANN, B. 2011. Shepherding the crowd: managing and providing feedback to crowd workers. In Proceedings of the CHI Extended Abstracts on Human Factors in Computing Systems. ACM Press, New York, 1669-1674.
26. EHN, P. 1988. Work-Oriented Design of Computer Artifacts. Lawrence Erlbaum, Mahwah, NJ.
27. ETERNA. 2011. http://eterna.cmu.edu/content/EteRNA.
28. FINKE, R. 1990. Creative Imagery: Discoveries and Inventions in Visualization. Lawrence Erlbaum, New York.
29. FINKE, R. A., WARD, T. B., AND SMITH, S. M. 1996. Creative Cognition: Theory, Research, and Applications. MIT Press, Cambridge, MA.
30. FISCHER, G., GIACCARDI, E., EDEN, H., SUGIMOTO, M., AND YE, Y. 2005. Beyond binary choices: Integrating individual and social creativity. Int. J. Hum.-Comput. Stud. 63, 4-5, 482-512. (Pubitemid 41261284)
31. FLEMING, L. 2001. Recombinant uncertainty in technological search. Manag. Sci. 47, 1, 117-132. (Pubitemid 34191966)
32. FOGEL, D. B., BACK, T., AND MICHALEWICZ, Z. 2000. Evolutionary Computation: Basic Algorithms and Operators. Taylor and Francis, New York.
33. GERO, J. S. 1994. Computational models of creative design processes. In Artificial Intelligence and Creativity, T. Dartnall, Ed., Kluwer, Dordrecht, 269-281.
34. th Conference on Creativity and Cognition. 3-10.
35. GERO, J. S. 2000. Computational models of innovative and creative design processes. Technol. Forecast. Soc. Change 64, 2-3, 183-196.
36. thInternational Conference on Engineering Design (ICED'09). 15-26.
37. GOLDBERG, D. E. 1989. Genetic Algorithms in Search, Optimization and Machine Learning. Kluwer Academic Publishers, Norwell, MA.
38. GOODING, D. C. 2010. Visualizing scientific inference. Topics Cogn. Sci. 2, 1, 15-35.
39. GOOGLE DOCS DRAWING APPLICATION. 2010. http://docs0.google.com/demo/edit? id=scACRQaIm3t83kVISWPh-Wfrqx#-drawing.
40. GRIER, D. A. 2007. When Computers Were Human. Princeton University Press, Princeton, NJ.
41. GULLEY, N. 2001. Patterns of innovation: A web-based matlab programming contest. In Proceedings of Extended Abstracts on Human Factors in Computing Systems (CHI'01). 337-338.
42. HARGADON, A. AND BECHKY, B. 2006. When collections of creatives become creative collectives: A field study of problem solving at work. Organ. Sci. 17, 4, 484-500. (Pubitemid 44167191)
43. HAYEK, F. A. 1945. The use of knowledge in society. Amer. Econ. Rev. 35, 4, 519-530.
44. HENDERSON, R. M. AND CLARK, K. B. 1990. Architectural innovation: The reconfiguration of existing product technologies and the failure of established firms. Admin. Sci. Quart. 35, 1, 9-30.
45. HOLLAND, J. H. 1975. Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor, MI.
46. th Graphics Interface Conference (GI). 39-46.
47. JOHNSON, C. G. 1999. Exploring the sound-space of synthesis algorithms using interactive genetic algorithms. In Proceedings of the AISB Workshop on Artificial Intelligence and Musical Creativity.
48. KENSING, F. AND BLOMBERG, J. 1998. Participatory design: Issues and concerns. In Proceedings of the Conference on Computer Supported Cooperative Work (CSCW'98). 167-185. (Pubitemid 128510645)
49. KIM, H. S. AND CHO, S. B. 2000. Application of interactive genetic algorithm to fashion design. Engin. Appl. Artif. Intell. 13, 6, 635-644.
50. KING, S., CONLEY, M., LATIMER, B., AND FERRARI, D. 1989. Co-Design: A Process of Design Participation. Van Nostrand Reinhold, New York.
51. KOSORUKOFF, A. 2001. Human based genetic algorithm. In Proceedings of the IEEE Conference on Systems, Man, and Cybernetics. 3464-3469. (Pubitemid 34199490)
52. KITTUR, A., CHI, E. H., AND SUH, B. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'08). 453-456.
53. KOHN, N. W., PAULUS, P. B., AND CHOI, Y. 2011. Building on the ideas of others: An examination of the idea combination process. J. Exp. Soc. Psychol. 47, 554-561.
54. LEWIS, S., DONTCHEVA, M., AND GERBER, E. 2011. Affective computational priming and creativity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'11). 735-744.
55. LITCHFIELD, R. C. 2008. Brainstorming reconsidered: A goal-based view. Acad. Manag. Rev. Arch. 33, 3, 649-668.
56. MAHER, M. L. AND POON, J. 1996. Modelling design exploration as co-evolution. Microcomput. Civil Engin. 11, 3, 195-210.
57. MALONE, T. W., LAUBACHER, R., AND DELLAROCAS, C. N. 2009. Harnessing crowds: Mapping the genome of collective intelligence. MIT Sloan Research, paper no. 4732-09. http://ssrn.com/abstract=1381502.
58. MARK, G., LYYTINEN, K., AND BERGMAN, M. 2007. Boundary objects in design: An ecological view of design artifacts. J. Assoc. Inf. Syst. 8, 11.
59. MASON, W. ANDWATTS, D. J. 2009. Financial incentives and the "performance of crowds". In Proceedings of the Human Computation Workshop (HCOMP): Knowledge Discovery and Data Mining Conference.
60. MATSUKA, T. AND NICKERSON, J. V. 2006. Modeling human hypothesis testing behaviors with simulated evolutionary processes. In Proceedings of the Annual IEEE Congress on Evolutionary Computation. 399-405. (Pubitemid 47130525)
61. th International Conference on Interaction Design and Children. 167-168. (Pubitemid 351438790)
62. MEDNICK, S. A. 1962. The associative basis of the creative process. Psychol. Rev. 69, 220-232.
63. MULLEN, B., JOHNSON C., AND SALAS E. 1991. Productivity loss in brainstorming groups: A metaanalytic integration. Basic Appl. Soc. Psychol. 72, 1, 3-23.
64. nd ed., J. Jacko and A. Sears, Eds., Erlbaum, 1061-1082.
65. MUMFORD, M. D., MOBLEY, M. I., UHLMAN, C. E., REITER-PALMON, R., AND DOARES, L. 1991. Process analytic models of creative thought. Creativity Res. J. 4, 2, 91-122.
66. NADIN, M. AND NOVAK, M. 1989. Interactive diagrams for design genetics: Implementing intelligent processors. In Intelligent CAD Systems, V. Akman, P. J. W. Ten Hagen, and P. J. Veerkamp, Eds., Springer, 247-261.
67. NELSON, R. R. AND WINTER S. G. 1982. An Evolutionary Theory of Economic Change. Belknap Press/Harvard University Press, Cambridge, MA.
68. NICKERSON, J. V. AND SAKAMOTO, Y. 2010. Crowdsourcing creativity: Combining ideas in networks. In Proceedings of the Workshop on Information in Networks.
69. NYE, B. D. 2011. Modeling memes: A memetic view of affordance learning. Doctoral dissertation. University of Pennsylvania, PA.
70. OLSSON, O. AND FREY, B. 2002. Entrepreneurship as recombinant growth. Small Bus. Econ. 19, 2, 69-80. (Pubitemid 37131585)
71. rd ed. Charles Scribners and Son, New York.
72. PARMEE, I. C. 2002. Improving problem definition through interactive evolutionary computation. Artif. Intell. Engin. Des. Anal. Manuf. 16, 03, 185-202. (Pubitemid 35476513)
73. PARMEE, I. C. AND BONHAM, C. R. 2000. Towards the support of innovative conceptual design through interactive designer/evolutionary computing strategies. Artif. Intell. Engin. Des. Anal. Manuf. 14, 01, 3-16.
74. PRASSE, M. AND RITTGEN, P. 1998. Why church's thesis still holds. Some notes on peter wegner's tracts on interaction and computability. Comput. J. 41, 6, 357.
75. PUTMAN, V. L. AND PAULUS, P. B. 2009. Brainstorming, brainstorming rules and decision making. J. Creative Behav. 43, 1, 23-39.
76. QUINN, A. J. AND BEDERSON, B. B. Human computation: A survey and taxonomy of a growing field. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'11). 1403-1412.
77. RAYKAR, V. C., YU, S., ZHAO, L. H., VALADEZ, G. H., FLORIN, C., BOGONI, L., AND MOY, L. 2010. Learning from crowds. J. Mach. Learn. Res. 11, 7, 1297-1322.
78. RESNICK, M., MALONEY, J., MONROY-HERNÁNDEZ, A., RUSK, N., EASTMOND, E., BRENNAN, K., MILLNER, A., ROSENBAUM, E., SILVER, J., AND SILVERMAN, B. 2009. Scratch: Programming for all. Comm. ACM 52, 11, 60-67.
79. RIETZSCHEL, E. F., NIJSTAD, B. A., AND STROEBE, W. 2006. Productivity is not enough: A comparison of interactive and nominal brainstorming groups on idea generation and selection. J. Exp. Soc. Psychol. 42, 2, 244-251. (Pubitemid 43192753)
80. ROSS, J., IRANI, L., SILBERMAN, M. S., ZALDIVAR, A., AND TOMLINSON, B. 2010. Who are the crowdworkers? Shifting demographics in mechanical turk. In altCHI Session of CHI'10 Extended Abstracts on Human Factors in Computing Systems. 2863-2872.
81. SAJNANI, H., JAVANMARDI, S., MCDONALD, D. W., AND LOPES, C. V. 2011. Multi-Label classification of short text: A study on wikipedia barnstars. In Proceedings of the Workshop on Analyzing Microtext (AAAI'11). 56-61.
82. SANDERS, E. AND STAPPERS, P. J. 2008. Co-Creation and the new landscapes of design. CoDes. 4, 1, 5-18.
83. SECRETAN, J., BEATO, N., DAMBROSIO, D. B., RODRIGUEZ, A., CAMPBELL, A., AND STANLEY, K. O. 2008. Picbreeder: Evolving pictures collaboratively online. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'08). 1759-1768.
84. SIMONTON, D. K. 2003. Scientific creativity as constrained stochastic behavior: The integration of product, person, and process perspectives. Psychol. Bull. 129, 4, 475-494. (Pubitemid 37189319)
85. SIMONTON, D. K. 2004. Creativity in Science: Chance, Logic, Genius and Zeitgeist. Cambridge University Press, Cambridge, UK.
86. SMITH, S. M., WARD, T. B., AND SCHUMACHER, J. S. 1993. Constraining effects of examples in a creative generation task. Mem. Cogn. 21, 6, 837-845. (Pubitemid 23331303)
87. TAKAGI, H. 1998. Application of interactive evolutionary computation to optimal tuning of digital hearing aids. In Proceedings of International Conference on Soft Computing (IIZUKA'98). 849-852.
88. TAKAGI, H. 2001. Interactive evolutionary computation: Fusion of the capabilities of EC optimization and human evaluation. Proc. IEEE 89, 9, 1275-1296. (Pubitemid 33766623)
89. TAYLOR, A. AND GREVE, H. R. 2006. Superman or the fantastic four? Knowledge combination and experience in innovative teams. Acad. Manag. J. 49, 4, 693-706.
90. THAGARD, P. 1992. Conceptual Revolutions. Princeton University Press, Princeton, NJ.
91. rd International Conference on Generative Art. 215-226.
92. TOLSTOY, D. 2009. Knowledge combination and knowledge creation in a foreign - Market network. J. S mall Bus. Manag. 47, 2, 202-220.
93. TOOMIM, M., KRIPLEAN, T., PÖRTNER, C., AND LANDAY, J. A. 2011. Utility of human-computer interactions: Toward a science of preference measurement. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'11). ACM Press, New York, 2275-2284.
94. st International Conference on Design Creativity. 209-214.
95. ULRICH, P. V., ANDERSON-CONNELL, L. J., AND WU, W. 2003. Consumer co-design of apparel for mass customization. J. Fashion Market. Manag. 7, 4, 398-412. (Pubitemid 38990727)
96. VON AHN, L. AND DABBISH, L. 2004. Labeling images with a computer game. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI'04). ACM Press, New York, 319-326.
97. VON AHN, L., MAURER, B., MCMILLEN, C., ABRAHAM, D., AND BLUM, M. 2008. RECAPTCHA: Human-Based character recognition via web security measures. Sci. 321, 5895, 1465-1468.
98. VON HIPPEL, E. 1986. Lead users: A source of novel product concepts. Manag. Sci. 32, 7, 791-805.
99. WEGNER, P. 1997. Why interaction is more powerful than algorithms. Comm. ACM 40, 5, 80-91. (Pubitemid 127442927)
100. WEGNER, P. AND EBERBACH, E. 2004. New models of computation. The Comput J. 47, 1, 4-9.
101. WOODBURY, R. F. 1993. A genetic approach to creative design. In Modeling Creativity and Knowledge-Based Creative Design, J. S. Gero and M. L. Maher Eds., Erlbaum, Hillside, NJ, 211-232.
102. YU, L. AND NICKERSON, J. V. 2011a. Cooks or cobblers? Crowd creativity through combination. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI'11). ACM Press, New York, 1393-1402.
103. YU, L. AND NICKERSON, J. V. 2011b. Generating creative ideas through crowds: An experimental study of combination. In Proceedings of the International Conference on Information Systems (ICIS'11).
104. YU, L. AND SAKAMOTO, Y. 2011. Feature selection in crowd creativity. In Proceedings of HCI International Conference. 383-392.
105. YU, L., SAKAMOTO, Y., AND NICKERSON, J. V. 2011. Feature propagation in idea networks. In Proceedings of the Workshop on Information in Networks.
106. YU, L. 2011. Crowd idea generation. Doctoral dissertation, Stevens Institute of Technology, NJ.
107. ZIMMERMAN, J., TOMASIC, A., GARROD C., YOO, D., HIRUNCHAROENVATE, C., AZIZ, R., THIRUVENGADAM, N. R., HUANG, Y., AND STEINFELD, A. 2011. Field trial of tiramisu: Crowd-Sourcing bus arrival times to spur co-design. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI'11). ACM Press, New York, 1677-1686.
108. ZWICKY, F. 1969. Discovery, Invention, Research through the Morphological Approach. The Macmillian Company, Toronto.