The knowledge accelerator: Big picture thinking in small pieces

Conference on Human Factors in Computing Systems - Proceedings

Volumn , Issue , 2016, Pages 2258-2270

  Hahn, Nathan ^a   Chang, Joseph ^a   Kim, Ji Eun ^b   Kittur, Aniket ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b ROBERT BOSCH GMBH   (Germany)

Author keywords

Complex workflow; Crowd work; Crowdsourcing; Design patterns; Information synthesis

Indexed keywords

CROWDSOURCING; HUMAN ENGINEERING; SCAFFOLDS;

COMPLEX WORKFLOWS; COMPUTATIONAL SYSTEM; CROWD WORK; DESIGN PATTERNS; DISTRIBUTED INFORMATION; INFORMATION SYNTHESIS; PROTOTYPE SYSTEM; REAL-WORLD TASK;

HUMAN COMPUTER INTERACTION;

EID: 84974806942     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2858036.2858364     Document Type: Conference Paper

References (67)

1. 2015. Etherpad Lite. https://github.com/ether/etherpad-lite. (2015).
2. Salman Ahmad, Alexis Battle, Zahan Malkani, and Sepander Kamvar. 2011. The jabberwocky programming environment for structured social computing. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 53-64.
3. Paul André, Aniket Kittur, and Steven P Dow. 2014a. Crowd synthesis: Extracting categories and clusters from complex data. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 989-998.
4. Paul André, Robert E Kraut, and Aniket Kittur. 2014b. Effects of simultaneous and sequential work structures on distributed collaborative interdependent tasks. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 139-148.
5. Anonymised. 2016. Alloy: Clustering with Crowds and Computation. In submission to CHI (2016).
6. Michelle Q Wang Baldonado and Terry Winograd. 1997. SenseMaker: an information-exploration interface supporting the contextual evolution of a user's interests. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. ACM, 11-18.
7. Regina Barzilay, Kathleen R McKeown, and Michael Elhadad. 1999. Information fusion in the context of multi-document summarization. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics. Association for Computational Linguistics, 550-557.
8. Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger. 2011. Crowds in Two Seconds: Enabling Realtime Crowd-powered Interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST'11). ACM, New York, NY, USA, 33-42. DOI: http://dx.doi.org/10.1145/2047196.2047201
9. Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. 2010a. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 313-322.
10. Michael S Bernstein, Desney Tan, Greg Smith, Mary Czerwinski, and Eric Horvitz. 2010b. Personalization via friendsourcing. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 2(2010), 6.
11. Michael S Bernstein, Jaime Teevan, Susan Dumais, Daniel Liebling, and Eric Horvitz. 2012. Direct answers for search queries in the long tail. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 237-246.
12. Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and others. 2010. VizWiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 333-342.
13. Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 335-336.
14. Pew Research Center. 2015. Generational differences in online activities. Report. (25 July 2015). http://www.pewinternet.org/2009/01/28/generational-differences-in-online-activities/.
15. Dana Chandler and Adam Kapelner. 2013. Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior & Organization 90(2013), 123-133.
16. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2, 3(2011), 27.
17. Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013. Cascade: Crowdsourcing Taxonomy Creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'13). ACM, New York, NY, USA, 1999-2008. DOI: http://dx.doi.org/10.1145/2470654.2466265
18. Richard L Daft and Karl E Weick. 1984. Toward a model of organizations as interpretation systems. Academy of management review 9, 2(1984), 284-295.
19. Hoa Trang Dang, Diane Kelly, and Jimmy J Lin. 2007. Overview of the TREC 2007 Question Answering Track.. In TREC, Vol. 7. 63.
20. Brenda Dervin. 1983. An overview of sense-making research: Concepts, methods, and results to date. The Author.
21. Brenda Dervin. 1992. From the minds eye of the user: The sense-making qualitative-quantitative methodology. Qualitative research in information management 9(1992), 61-84.
22. Brenda Dervin. 1998. Sense-making theory and practice: an overview of user interests in knowledge seeking and use. Journal of knowledge management 2, 2(1998), 36-46.
23. Günes Erkan and Dragomir R Radev. 2004. LexRank: graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research (2004), 457-479.
24. Dennis A Gioia and Kumar Chittipeddi. 1991. Sensemaking and sensegiving in strategic change initiation. Strategic management journal 12, 6(1991), 433-448.
25. Jade Goldstein, Vibhu Mittal, Jaime Carbonell, and Mark Kantrowitz. 2000. Multi-document summarization by sentence extraction. In Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization-Volume 4. Association for Computational Linguistics, 40-48.
26. Vishal Gupta and Gurpreet Singh Lehal. 2010. A survey of text summarization extractive techniques. Journal of Emerging Technologies in Web Intelligence 2, 3(2010), 258-268.
27. Udo Hahn and Ulrich Reimer. 1999. Knowledge-based text summarization: Salience and generalization operators for knowledge base abstraction. Advances in Automatic Text Summarization (1999), 215-232.
28. Anil K Jain, M. Narasimha Murty, and Patrick J Flynn. 1999. Data clustering: a review. ACM computing surveys (CSUR) 31, 3(1999), 264-323.
29. Jim Jones. 2013. Turkee Ruby Gem. https://github.com/aantix/turkee. (2013).
30. Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining Human and Machine Intelligence in Large-scale Crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (AAMAS'12). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 467-474. http://dl.acm.org/citation. cfm?id=2343576.2343643
31. Ece Kamar and Eric Horvitz. 2013. Light at the End of the Tunnel: A Monte Carlo Approach to Computing Value of Information. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS'13). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 571-578. http://dl.acm.org/citation. cfm?id=2484920.2485011
32. Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J Guo, Robert C Miller, and Krzysztof Z Gajos. 2014. Crowdsourcing step-by-step information extraction to enhance existing how- to videos. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 4017-4026.
33. Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 453-456.
34. Aniket Kittur, Susheel Khamkar, Paul André, and Robert Kraut. 2012. CrowdWeaver: Visually Managing Complex Crowd Work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW'12). ACM, New York, NY, USA, 1033-1036. DOI: http://dx.doi.org/10.1145/2145204.2145357
35. Aniket Kittur and Robert E Kraut. 2008. Harnessing the wisdom of crowds in wikipedia: quality through coordination. In Proceedings of the 2008 ACM conference on Computer supported cooperative work. ACM, 37-46.
36. Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013a. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1301-1318.
37. Aniket Kittur, Andrew M Peters, Abdigani Diriye, Trupti Telang, and Michael R Bove. 2013b. Costs and benefits of structured information foraging. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2989-2998.
38. Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43-52.
39. Aniket Kittur, Bongwon Suh, Bryan A Pendleton, and Ed H Chi. 2007. He says, she says: conflict and coordination in Wikipedia. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 453-462.
40. Gary Klein, Brian Moon, and Robert R Hoffman. 2006. Making sense of sensemaking 2: A macrocognitive model. Intelligent Systems, IEEE 21, 5(2006), 88-92.
41. Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1003-1012.
42. Anand P Kulkarni, Matthew Can, and Bjoern Hartmann. 2011. Turkomatic: automatic recursive task and workflow design for mechanical turk. In CHI'11 Extended Abstracts on Human Factors in Computing Systems. ACM, 2053-2058.
43. Walter Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato Borrello, Raja Kushalnagar, and Jeffrey Bigham. 2012. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 23-34.
44. Walter S Lasecki, Christopher D Miller, and Jeffrey P Bigham. 2013a. Warping time for more effective real-time crowdsourcing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2033-2036.
45. Walter S Lasecki, Christopher D Miller, Raja Kushalnagar, and Jeffrey P Bigham. 2013b. Legion scribe: real-time captioning by the non-experts. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility. ACM, 22.
46. Edith Law and Haoqi Zhang. 2011. Towards Large-Scale Collaborative Planning: Answering High-Level Search Queries Using Human Computation.. In AAAI.
47. Greg Little, Lydia B Chilton, Max Goldman, and Robert C Miller. 2010. Turkit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 57-66.
48. Kurt Luther, Casey Fiesler, and Amy Bruckman. 2013. Redistributing Leadership in Online Creative Collaboration. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW'13). ACM, New York, NY, USA, 1007-1022. DOI: http://dx.doi.org/10.1145/2441776.2441891
49. Inderjeet Mani and Eric Bloedorn. 1997. Multi-document summarization by graph search and matching. arXiv preprint cmp-lg/9712004 (1997).
50. Gary Marchionini. 2006. Exploratory Search: From Finding to Understanding. Commun. ACM 49, 4(April 2006), 41-46. DOI: http://dx.doi.org/10.1145/1121949.1121979
51. Kathleen McKeown, Judith Klavans, Vasileios Hatzivassiloglou, Regina Barzilay, and Eleazar Eskin. 1999. Towards multidocument summarization by reformulation: Progress and prospects. In AAAI/IAAI. 453-460.
52. James McKinney. 2015. TfIdfSimilarity Ruby Gem. https://github.com/jpmckinney/tf-idf-similarity. (2015).
53. Frances J Milliken. 1990. Perceiving and interpreting environmental change: An examination of college administrators' interpretation of changing demographics. Academy of management Journal 33, 1(1990), 42-63.
54. Meredith Ringel Morris, Jarrod Lombardo, and Daniel Wigdor. 2010. WeSearch: supporting collaborative search and sensemaking on a tabletop display. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. ACM, 401-410.
55. Aditya Parameswaran, Ming Han Teh, Hector Garcia-Molina, and Jennifer Widom. 2013. Datasift: An expressive and accurate crowd-powered search toolkit. In First AAAI Conference on Human Computation and Crowdsourcing.
56. Sharoda A Paul and Meredith Ringel Morris. 2009. CoSense: enhancing sensemaking for collaborative web search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1771-1780.
57. Sharoda A Paul and Madhu C Reddy. 2010. Understanding together: sensemaking in collaborative information seeking. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. ACM, 321-330.
58. Peter Pirolli and Stuart Card. 1999. Information foraging. Psychological review 106, 4(1999), 643.
59. Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S Bernstein. 2014. Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 75-85.
60. Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets.. In ICWSM.
61. Daniel M. Russell, Mark J. Stefik, Peter Pirolli, and Stuart K. Card. 1993. The Cost Structure of Sensemaking. In Proceedings of the INTERACT'93 and CHI'93 Conference on Human Factors in Computing Systems (CHI'93). ACM, New York, NY, USA, 269-276. DOI: http://dx.doi.org/10.1145/169059.169209
62. Bongwon Suh, Gregorio Convertino, Ed H Chi, and Peter Pirolli. 2009. The singularity is not near: slowing growth of Wikipedia. In Proceedings of the 5th International Symposium on Wikis and Open Collaboration. ACM, 8.
63. Jaime Teevan, Daniel J Liebling, and Walter S Lasecki. 2014. Selfsourcing personal tasks. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. ACM, 2527-2532.
64. Karl E Weick. 1964. Reduction of cognitive dissonance through task enhancement and effort expenditure. The Journal of Abnormal and Social Psychology 68, 5(1964), 533.
65. Karl E. Weick. 1995. Sensemaking in organizations. Vol. 3. Sage.
66. Ryen W White, Mikhail Bilenko, and Silviu Cucerzan. 2007. Studying the use of popular destinations to enhance web search interaction. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 159-166.
67. Haoqi Zhang, Edith Law, Rob Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz. 2012. Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 217-226.