Crowdsourced knowledge acquisition: Towards hybrid-genre workflows

International Journal on Semantic Web and Information Systems

Volumn 9, Issue 3, 2013, Pages 14-41

  Sabou, Marta ^a   Scharl, Arno ^a   Föls, Michael ^b  

^a MODUL UNIVERSITY VIENNA   (Austria)

^b VIENNA UNIVERSITY OF ECONOMICS AND BUSINESS   (Austria)

Author keywords

Climate Change; CrowdFlower; Crowdsourcing; Games with a Purpose; Human Computation; Knowledge Acquisition; Mechanised Labour; Workflows

Indexed keywords

CLIMATE CHANGE;

CROWDFLOWER; CROWDSOURCING; GAMES WITH A PURPOSE; HUMAN COMPUTATION; MECHANISED LABOUR; WORK-FLOWS;

KNOWLEDGE ACQUISITION;

EID: 84898927778     PISSN: 15526283     EISSN: 15526291     Source Type: Journal    
DOI: 10.4018/ijswis.2013070102     Document Type: Article

References (50)

1. Berners-Lee, T., Hendler, J., & Lassila, O. (2001). The Semantic Web. Scientific American, 284(5), 34-43. doi:10. 1038/scientificamerican0501-34 PMID:11396337
2. Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., et al. (2010). Soylent: A word processor with a crowd inside. In Proceedings of the 23rd ACM Symposium on User Interface Software and Technology (pp. 313-322)
3. Bontcheva, K., Cunningham, H., Roberts, I., Roberts, A., Tablan, V., Aswani, N., & Gorrell, G. (2013). GATE Teamware: A web-based, collaborative text annotation framework. Language Resources and Evaluation. doi:10. 1007/s10579-013-9215-6
4. Brew, A., Greene, D., & Cunningham, P. (2010). Using crowdsourcing and active learning to track sentiment in online media. In Proceedings of the European Conference on Artificial Intelligence (pp. 145-150)
5. Callison-Burch, C. (2009). Fast, cheap, and creative: Evaluating translation quality using Amazon's mechanical Turk. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 286-295)
6. Celino, I., Contessa, S., Corubolo, M., Dell'Aglio, D., Della Valle, E., Fumeo, S., & Krüger, T. (2012). Linking smart cities datasets with human computation-The case of UrbanMatch. In Proceedings of the International Semantic Web Conference (pp. 34-49)
7. Chamberlain, J., Fort, K., Kruschwitz, U., Lafourcade, M., & Poesio, M. (2013). Using games to create language resources: Successes and limitations of the approach. In I. Gurevych, & K. Jungi, The people's web meets NLP. Collaboratively constructed language resources. Springer. doi:10. 1007/978-3-642-35085-6-1
8. Chklovski, T. (2005). Collecting paraphrase corpora from volunteer contributors. In Proceedings of the 3rd International Conference on Knowledge Capture (pp. 115-120)
9. Dai, P. Mausam & Weld, D. S. (2010). Decision-theoretic control of crowd-sourced workflows. In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 130-135). AAAI
10. Demartini, G., Difallah, D. E., & Cudré-Mauroux, P. (2012). ZenCrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In Proceedings of the 21st International Conference on World Wide Web (pp. 469-478). ACM
11. Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web. Communications of the ACM, 54(4), 86-96. doi:10. 1145/1924421. 1924442
12. Eckert, K., Niepert, M., Niemann, C., Buckner, C., Allen, C., & Stuckenschmidt, H. (2010). Crowdsourcing the assembly of concept hierarchies. In Proceedings of the 10th Annual Joint Conference on Digital Libraries (pp. 139-148). ACM
13. Feigenbaum, E. A. (1977). The art of artificial intelligence: Themes and case studies of knowledge engineering. In Proceedings of the 5th International Joint Conference of Artificial Intelligence (pp. 1014-1029)
14. Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37(2), 413-420. doi:10. 1162/ COLI-A-00057
15. Heath, T., & Bizer, C. (2011). Linked data: Evolving the web into a global data space (1st edition). Synthesis lectures on the Semantic Web: Theory and technology. Morgan & Claypool
16. Hees, J., Roth-Berghofer, T., Biedert, R., Adrian, B., & Dengel, A. (2011). BetterRelations: Using a game to rate linked data triples. In J. Bach and S. Edelkamp Proceedings of the 34th Annual German Conference on Artificial Intelligence (pp. 134-138). Springer
17. Hoffmann, L. (2009). Crowd control. Communications of the ACM, 52(3), 16-17. doi:10. 1145/1467247. 1467254
18. Hong, J., & Baker, C. F. (2011). How good is the crowd at "real" WSD? In Proceedings of the 5th Linguistic Annotation Workshop (pp. 30-37)
19. Hovy, E. (2010). Annotation. In Tutorial abstracts of ACL
20. Howe, J. (2009). Crowdsourcing: Why the power of the crowd is driving the future of business. Retrieved from http://crowdsourcing. typepad. com/
21. Kawrykow, A., Roumanis, G., Kam, A., Kwak, D., Leung, C., Wu, C., & Zarour, E., & Phylo players. (2012). Phylo: A citizen science approach for improving multiple sequence alignment. PLoS ONE, 7(3), 31362. doi:10. 1371/journal. pone. 0031362 PMID:22412834
22. Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 453-456). ACM
23. Kittur, A., Smus, B., & Kraut, R. (2011). CrowdForge: Crowdsourcing complex work. In Proceedings of the Conference on Human Factors in Computing Systems (pp. 1801-1806). ACM
24. Laws, F., Scheible, C., & Schutze, H. (2011). Active learning with Amazon mechanical Turk. In Proceeding of the Conference on Empirical Methods in NLP (pp. 1546-1556)
25. Lieberman, H., Smith, D. A., & Teeters, A. (2007). Common Consensus: a web-based game for collecting commonsense goals. In Proceedings of the Workshop on Common Sense and Intelligent User Interfaces held in conjunction with the 2007 International Conference on Intelligent User Interfaces IUI
26. Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010). TurKit: Human computation algorithms on mechanical Turk. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (pp. 57-66). ACM
27. Liu, W., Weichselbraun, A., Scharl, A., & Chang, E. (2005). Semi-Automatic ontology extension using spreading activation. In Proceedings of the 5th International Conference on Knowledge Management (pp. 145-153). Springer
28. Markotschi, T., & Völker, J. (2010). Guess what Human intelligence for mining linked data. In Proceedings of the Workshop on Knowledge Injection into and Extraction from Linked Data at the International Conference on Knowledge Engineering and Knowledge Management (EKAW-2010)
29. Negri, M., Bentivogli, L., Mehdad, Y., Giampiccolo, D., & Marchetti, A. (2011). Divide and conquer: Crowdsourcing the creation of cross-lingual textual entailment corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 670-679)
30. Poesio, M., Kruschwitz, U., Chamberlain, J., Robaldo, L., & Ducceschi, L. (2013). (in press). Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems. doi:10. 1145/2448116. 2448119
31. Quinn, A. J., & Bederson, B. B. (2011). Human computation: A survey and taxonomy of a growing field. In Proceedings of Human Factors in Computing Systems (pp. 1403-1412). ACM. doi:10. 1145/1978942. 1979148
32. Rafelsberger, W., & Scharl, A. (2009). Games with a purpose for social networking platforms. In C. Cattuto et al. Proceedings of the 20th ACM Conference on Hypertext and Hypermedia (pp. 193-197). ACM
33. Sarasua, C., Simperl, E., & Noy, N. F. (2012). CrowdMap: Crowdsourcing ontology alignment with microtasks. In P. Cudré-Mauroux, J. Heflin, E. Sirin, T. Tudorache, & J. Euzenat (Eds. ), Proceedings of the 11th International Conference on The Semantic Web (pp. 525-541)
34. Scharl, A., Sabou, M., & Föls, M. (2012). Climate quiz-A web application for eliciting and validating knowledge from social networks. In G. Bressan & R. M. Silveira (Eds. ), Proceedings of the 18th Brazilian Symposium on Multimedia and the Web (pp. 189-192). ACM
35. Scharl, A., & Weichselbraun, A. (2008). An automated approach to investigating the online media coverage of US presidential elections. Journal of Information Technology & Politics, 5(1), 121-132. doi:10. 1080/19331680802149582
36. Siorpaes, K., & Hepp, M. (2008). Games with a purpose for the Semantic Web. IEEE Intelligent Systems, 23(3), 50-60. doi:10. 1109/MIS. 2008. 45
37. Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and fast-but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 254-263)
38. Strohmaier, M., Helic, D., Benz, D., Körner, C. & Kern, R. (2012). Evaluation of folksonomy induction algorithms. ACM Transactions on Intelligent Systems and Technology, 3(4), Article 74
39. Thaler, S., Simperl, E., & Siorpaes, K. (2011) SpotTheLink: A game for ontology alignment. In Proceedings of the 6th Conference for Professional Knowledge Management (pp. 246-253)
40. Thaler, S., Simperl, E., & Wölger, S. (2012). An experiment in comparing human-computation techniques. IEEE Internet Computing, 16(5), 52-58. doi:10. 1109/MIC. 2012. 67
41. Tudorache, T., Nyulas, C. I., Noy, N. F., & Musen, M. A. (2013). WebProtégé: A collaborative ontology editor and knowledge acquisition tool for the web. [IOS Press. ]. Semantic Web Journal, 4(1), 89-99. PMID:23807872
42. Vickrey, D., Bronzan, A., Choi, W., Kumar, A., Turner-Maier, J., Wang, A., & Koller, D. (2008). Online word games for semantic data collection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 533-542). ACL
43. von Ahn, L., & Dabbish, L. (2008). Designing games with a purpose. Communications of the ACM, 51(8), 58-67. doi:10. 1145/1378704. 1378719
44. von Ahn, L., Mihir, K., & Blum, M. (2006). Verbosity: A game for collecting common-sense facts. In R. Grinter, T. Rodden, P. Aoki, E. Cutrell, R. Jeffries, & G. Olson (Eds. )ACM Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 75-78). ACM
45. Waitelonis, J., Ludwig, N., Knuth, M., & Sack, H. (2011). WhoKnows-Evaluating linked data heuristics with a quiz that cleans up DBpedia. International Journal of Interactive Technology and Smart Education, 8(4), 236-248. doi:10. 1108/17415651111189478
46. Wang, A., Hoang, C. D. V., & Kan, M. Y. (2012). Perspectives on crowdsourcing annotations for natural language processing. Language Resources and Evaluation. Published online. DOI: 10. 1007/ s10579-012-9176-1
47. Weichselbraun, A., Wohlgenannt, G., & Scharl, A. (2010). Refining non-taxonomic relation labels with external structured data to support ontology learning. Data & Knowledge Engineering, 69(8), 763-778. doi:10. 1016/j. datak. 2010. 02. 010
48. Wiggins, A., & Crowston, K. (2011). From conservation to crowdsourcing: A typology of citizen science. In Proceedings of the 44th Hawaii International Conference on System Science (HICSS-44). IEEE Computer Society
49. Wohlgenannt, G., Weichselbraun, A., Scharl, A., & Sabou, M. (2012). Dynamic integration of multiple evidence sources for ontology learning. Journal of Information and Data Management, 3(3), 243-254
50. Wolf, L., Knuth, M., Osterhoff, J., & Sack, H. (2011). RISQ! Renowned individuals semantic quiz: A jeopardy like quiz game for ranking facts. In C. Ghidini (Eds. ), ACM Proceedings of the 7th International Conference on Semantic Systems (pp. 71-78)