Mechanical Turk as an ontology engineer? Using microtasks as a component of an ontology-engineering workflow

Proceedings of the 5th Annual ACM Web Science Conference, WebSci'13

Volumn volume, Issue , 2013, Pages 262-271

  Noy, Natalya F ^a   Mortensen, Jonathan ^a   Alexander, Paul R ^a   Musen, Mark A ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

Amazon Mechanical Turk; Crowdsourcing; Human computation; Ontology; Semantic Web

Indexed keywords

ONTOLOGY; SEMANTIC WEB;

AMAZON MECHANICAL TURKS; CROWDSOURCING; HUMAN COMPUTATION; LOGICAL CORRECTNESS; ONTOLOGY ENGINEERING; ONTOLOGY EVALUATIONS; TIME-CONSUMING TASKS; UNDERGRADUATE STUDENTS;

STUDENTS;

EID: 84883102297     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2464464.2464482     Document Type: Conference Paper

References (35)

1. Alexander, P. R., Nyulas, C. I., Tudorache, T., Whetzel, T., Noy, N. F., and Musen, M. A. Semantic infrastructure to enable collaboration in ontology development. In International Workshop on Semantic Technologies for Information-Integrated Collaboration (STIIC 2011) (Philadelphia, PA, USA, 2011).
2. Auer, S., Dietzold, S., and Riechert, T. OntoWiki - a tool for social, semantic collaboration. In Fifth International Semantic Web Conference, ISWC, vol. LNCS 4273, Springer (Athens, GA, 2006).
3. Bernstein, M., Little, G., Miller, R., Hartmann, B., Ackerman, M., Karger, D., Crowell, D., and Panovich, K. Soylent: a word processor with a crowd inside. In The 23d annual ACM symposium on user interface software and technology, ACM (2010), 313-322.
4. Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., and Popovi?, Z. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756-760.
5. Demartini, G., Difallah, D. E., and Cudr-Mauroux, P. Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In 21st World Wide Web Conference WWW2012 (Lyon, France, 2012), 469-478.
6. Evermann, J., and Fang, J. Evaluating ontologies: Towards a cognitive measure of quality. Information Systems 35 (2010), 391403.
7. Ghidini, C., Kump, B., Lindstaedt, S., Mahbub, N., Pammer, V., Rospocher, M., and Serafini, L. Moki: The enterprise modelling wiki. In European Semantic Web Conference (ESWC-2009), Springer Berlin /Heidelberg (Heraklion, Greece, 2009), 831-835.
8. GOConsortium. Creating the Gene Ontology resource: design and implementation. Genome Res 11, 8 (2001), 1425-33.
9. Green, P., and Rosemann, M. Integrated process modeling: An ontological evaluation. Information Systems 25, 2 (2000), 73-87.
10. Haendel, M., Neuhaus, F., Osumi-Sutherland, D., Mabee, P., Mejino, J., Mungall, C., and Smith, B. Carothe common anatomy reference ontology. Anatomy Ontologies for Bioinformatics (2008), 327-349.
11. Hausenblas, M., Troncy, R., Raimond, Y., and Brger, T. Interlinking multimedia: How to apply linked data principles to multimedia fragments. In WWW 2009 Workshop: Linked Data on the Web (2009).
12. Kittur, A., Chi, E., and Suh, B. Crowdsourcing user studies with Mechanical Turk. In 26th annual SIGCHI conference on human factors in computing systems (2008), 453-456.
13. Lin, C. H., Mausam, and Weld, D. S. Dynamically switching between synergistic workows for crowdsourcing. In Twenty-Sixth AAAI Conference on Artificial Intelligence (2012).
14. Markotschi, T., and Völker, J. GuessWhat?! - Human Intelligence for Mining Linked Data. In Proceedings of the Workshop on Knowledge Injection into and Extraction from Linked Data at EKAW (2010).
15. Mason, W., and Watts, D. Financial incentives and the "Performance of Crowds". In ACM SIGKDD workshop on human computation, ACM (2009), 77-85.
16. McCann, R., Shen, W., and Doan, A. Matching schemas in online communities: A Web 2.0 approach. In The 24th International Conference on Data Engineering (ICDE-08) (Cancun, Mexico, 2008).
17. Minder, P., Seuken, S., Bernstein, A., and Zollinger, M. Crowdmanager-combinatorial allocation and pricing of crowdsourcing tasks with time constraints. In Workshop on Social Computing and User Generated Content in conjunction with ACM Conference on Electronic Commerce (ACM-EC 2012) (Valencia, Spain, 2012), 1-18.
18. Musen, M. A., Noy, N. F., Shah, N. H., Whetzel, P. L., Chute, C. G., Storey, M.-A., Smith, B., and team, T. N. The national center for biomedical ontology. Journal of American Medical Informatics Association 19 (2012), 190-195.
19. Niles, I., and Pease, A. Towards a standard upper ontology. In The 2nd International Conference on Formal Ontology in Information Systems (FOIS-2001) (Ogunquit, Maine, 2001).
20. Noy, N. F., Griffith, N., and Musen, M. A. Collecting community-based mappings in an ontology repository. In 7th International Semantic Web Conference (ISWC 2008) (Karlsruhe, Germany, 2008).
21. Noy, N. F., Mortensen, J., Alexander, P. R., and Musen, M. A. Ontology engineering through microtask crowdsourcing. Under review (2013).
22. Quinn, A., and Bederson, B. Human computation: a survey and taxonomy of a growing field. In Annual Conference on Human Factors in Computing Systems (CHI 2011), ACM (Vancouver, BC, 2011), 1403-1412.
23. Raddick, M., Bracey, G., Gay, P., Lintott, C., Murray, P., Schawinski, K., Szalay, A., and Vandenberg, J. Galaxy zoo: exploring the motivations of citizen science volunteers. arXiv preprint arXiv:0909.2925 (2009).
24. Sarasua, C., Simperl, E., and Noy, N. F. Crowdmap: Crowdsourcing ontology alignment with microtasks. In 11th International Semantic Web Conference (ISWC), Springer (Boston, MA, 2012).
25. Schwarz, N. Self-reports: How the questions shape the answers. American Psychologist 54, 2 (1999), 93-105.
26. Sebastian, A., Noy, N. F., Tudorache, T., and Musen, M. A. A generic ontology for collaborative ontology-development workflows. In 16th International Conference on Knowledge Engineering and Knowledge Management (EKAW 2008), Springer (Catania, Italy, 2008).
27. Simperl, E., Norton, B., and Vrandecic, D. Crowdsourcing tasks in linked data management. In 2nd workshop on consuming Linked Data COLD2011 co-located with the 10th International Semantic Web Conference ISWC 2011 (Bonn, Germany, 2011).
28. Tanur, J. M. Questions about Questions: Inquiries Into the Coqnitive Bases of Surveys. Russell Sage Foundation Publications, 1992.
29. Thaler, S., Siorpaes, K., and Simperl, E. SpotTheLink: A Game for Ontology Alignment. In 6th Conference for Professional Knowledge Management (2011).
30. Tudorache, T., Nyulas, C., Noy, N. F., and Musen, M. A. Webprotégé: A distributed ontology editor and knowledge acquisition tool for the web. Semantic Web Journal 11-165 (2011).
31. von Ahn, L., and Dabbish, L. Labeling images with a computer game. In SIGCHI conference on Human factors in computing systems, ACM Press New York, NY, USA (2004), 319-326.
32. Wang, J., Ghose, A., and Ipeirotis, P. Bonus, disclosure, and choice: What motivates the creation of high-quality paid reviews? In Thirty Third International Conference on Information Systems (ICIS) (Orlando, FL, 2012).
33. Waterhouse, T. P. Pay by the bit: an information-theoretic metric for collective human judgment. In Conference on Computer supported cooperative work (CSCW), ACM (2013), 623-638.
34. Whetzel, P. L., Noy, N. F., Shah, N. H., Alexander, P. R., Nyulas, C. I., Tudorache, T., and Musen, M. A. Bioportal: Enhanced functionality via new web services from the national center for biomedical ontology to access and use ontologies in software applications. Nucleic Acids Research (NAR) 39, Web Server issue (2011), W541-5.
35. Zhdanova, A., and Shvaiko, P. Community-driven ontology matching. In 3rd European Semantic Web Conference (Budva, Montenegro, 2006), 3449.