Task Workflow Design and its impact on performance and volunteers' subjective preference in Virtual Citizen Science

International Journal of Human Computer Studies

Volumn 104, Issue , 2017, Pages 50-63

  Sprinks, James ^a   Wardlaw, Jessica ^a   Houghton, Robert ^a   Bamford, Steven ^a   Morley, Jeremy ^a,b  

^a UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^b ORDNANCE SURVEY   (United Kingdom)

Author keywords

Citizen Science; Engagement; Interface design; Task workflow

Indexed keywords

HARDWARE;

CITIZEN SCIENCE; ENGAGEMENT; INTERFACE DESIGNS; SCIENTIFIC RESEARCHES; SCIENTIFIC RESULTS; TASK WORKFLOW; USER SATISFACTION; WORKFLOW DESIGNS;

ENGINEERING;

EID: 85015658157     PISSN: 10715819     EISSN: 10959300     Source Type: Journal    
DOI: 10.1016/j.ijhcs.2017.03.003     Document Type: Article

References (49)

1. Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H.R., Bertino, E., Dustdar, S., Quality Control in Crowdsourcing Systems: Issues and Directions. IEEE Internet Comput. 17 (2013), 76–81.
2. Cai, C.J., Iqbal, S.T., Teevan, J., 2016. Chain Reactions: The Impact of Order on Microtask Chains. In: Proceedings of the 34th Annual ACM Conference on Human Factors in Computing Systems. ACM, San Jose, USA, p. Pre-press.
3. Cheng, J., Teevan, J., Iqbal, S.T., Bernstein, M.S., 2015. Break It Down: A Comparison of Macro- and Microtasks. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15. ACM, New York, NY, USA, pp. 4061–4064. doi: 10.1145/2702123.2702146.
4. Chung-Yan, G.A., The nonlinear effects of job complexity and autonomy on job satisfaction, turnover, and psychological well-being. J. Occup. Health Psychol. 15 (2010), 237–251.
5. Cox, J., Oh, E.Y., Simmons, B., Lintott, C., Masters, K., Greenhill, A., Graham, G., Holmes, K., Defining and measuring success in online citizen science: a case study of Zooniverse Projects. Comput. Sci. Eng. 17 (2015), 28–41, 10.1109/MCSE.2015.65.
6. Crowston, K., Fagnot, I., 2008. The Motivational Arc of Massive Virtual Collaboration. Proceedings IFIP WG 95 Work. Conference Virtuality Soc. Massive Virtual Communities.
7. Curtis, V., Online citizen science games: opportunities for the biological sciences. Appl. Transl. Genom. Glob. Shar. Genom. Knowl. Free Mark. 3 (2014), 90–94.
8. Deterding, S., Sicart, M., Nacke, L., O'Hara, K., Dixon, D., Gamification. Using Game-design Elements in Non-gaming Contexts. CHI ’11 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’11, 2011, ACM, New York, NY, USA, 2425–2428.
9. Dodd, N.G., Ganster, D.C., The interactive effects of variety, autonomy, and feedback on attitudes and performance. J. Organ. Behav. 17 (1996), 329–347.
10. Dow, S., Kulkarni, A., Klemmer, S., Hartmann, B., 2012. Shepherding the Crowd Yields Better Work. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12. ACM, New York, NY, USA, pp. 1013–1022. doi: 10.1145/2145204.2145355.
11. Dubinsky, A.J., Skinner, S.J., Impact of job characteristics on retail salespeople's reactions to their jobs. J. Retail 60 (1984), 35–62.
12. Eveleigh, A., Jennett, C., Blandford, A., Brohan, P., Cox, A.L., 2014. Designing for Dabblers and Deterring Drop-outs in Citizen Science. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, CHI ’14. ACM, New York, NY, USA, pp. 2985–2994.
13. Eveleigh, A., Jennett, C., Lynn, S., Cox, A.L., 2013. I Want to Be a Captain! I Want to Be a Captain!: Gamification in the Old Weather Citizen Science Project. In: Proceedings of the First International Conference on Gameful Design, Research, and Applications, Gamification ’13. ACM, New York, NY, USA, pp. 79–82.
14. Farell, B., Pelli, D.G., Psychophysical methods, or how to measure a threshold and why. Vision Research: A Practical Guide to Laboratory Methods, 1999, Oxford University Press, New York.
15. Freitag, A., Meyer, R., Whiteman, L., Strategies employed by Citizen Science programs to increase the credibility of their data. Citiz. Sci. Theory Pract., 2016, 1.
16. Gerhart, B., How important are dispositional factors as determinants of job satisfaction Implications for job design and other personnel programs. J. Appl. Psychol. 72 (1987), 366–373.
17. Ghani, J.A., Deshpande, S.P., Task characteristics and the experience of optimal flow in human—computer interaction. J. Psychol. 128 (1994), 381–391.
18. Hackman, J.R., Oldham, G.R., Development of the job diagnostic survey. J. Appl. Psychol. 60 (1975), 159–170.
19. Hand, E., Citizen science: people power. Nat. News 466 (2010), 685–687.
20. Harrower, M., Sheesley, B., Designing better map interfaces: a framework for panning and zooming. Trans. GIS 9 (2005), 77–89.
21. Hennon, C.C., Knapp, K.R., Schreck, C.J., Stevens, S.E., Kossin, J.P., Thorne, P.W., Hennon, P.A., Kruk, M.C., Rennie, J., Gadéa, J.-M., Striegl, M., Carley, I., Cyclone center: can citizen scientists improve tropical cyclone intensity records. Bull. Am. Meteorol. Soc., 2014.
22. Huang, C.-Y., Distributed manufacturing execution systems: a workflow perspective. J. Intell. Manuf. 13 (2002), 485–497.
23. Hutt, H., Everson, R., Grant, M., Love, J., Littlejohn, G., 2013. How clumpy is my image Evaluating crowdsourced annotation tasks. In: Proceedings of the 13th UK Workshop on Computational Intelligence (UKCI), pp. 136–143. doi: 10.1109/UKCI.2013.6651298.
24. Iacovides, I., Jennett, C., Cornish-Trestrail, C., Cox, A.L., Do games attract or sustain engagement in citizen science: A study of volunteer motivations. CHI ’13 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’13., 2013, ACM, New York, NY, USA, 1101–1106.
25. Jordan, R., Crall, A., Gray, S., Phillips, T., Mellor, D., 2015. Citizen Science as a Distinct Field of Inquiry. BioScience biu217.
26. Kittur, A., Nickerson, J.V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., Horton, J., 2013. The Future of Crowd Work. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ’13. ACM, New York, NY, USA, pp. 1301–1318.
27. Kulkarni, A., Can, M., Hartmann, B., 2012. Collaboratively Crowdsourcing Workflows with Turkomatic. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12. ACM, New York, NY, USA, pp. 1003–1012. doi: 10.1145/2145204.2145354.
28. Kulkarni, A.P., Can, M., Hartmann, B., Turkomatic: automatic recursive task and workflow design for mechanical turk. CHI ’11 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’11, 2011, ACM, New York, NY, USA, 2053–2058, 10.1145/1979742.1979865.
29. Lahav, O., Naim, A., Buta, R.J., Corwin, H.G., de Vaucouleurs, G., Dressler, A., Huchra, J.P., van den Bergh, S., Raychaudhury, S., Sodré, L., Storrie-Lombardi, M.C., Galaxies, human eyes, and artificial neural networks. Science 267 (1995), 859–862.
30. Lasecki, W.S., Marcus, A., Rzeszotarski, J.M., Bigham, J.P., 2014. Using Microtask Continuity to Improve Crowdsourcing. Carnegie Mellon Univ. Hum.-Comput. Interact. Inst. - Tech. Rep. CMU-HCII-14-100.
31. Lintott, C., Schawinski, K., Bamford, S., Slosar, A., Land, K., Thomas, D., Edmondson, E., Masters, K., Nichol, R.C., Raddick, M.J., Szalay, A., Andreescu, D., Murray, P., Vandenberg, J., Galaxy Zoo 1: data release of morphological classifications for nearly 900 000 galaxies. Mon. Not. R. Astron. Soc. 410 (2011), 166–178.
32. Mankowski, T.A., Slater, S.J., Slater, T.F., An interpretive study of meanings citizen scientists make when participating in Galaxy Zoo. Contemp. Issues Educ. Res. CIER 4 (2011), 25–42.
33. Mao, A., Kamar, E., Chen, Y., Horvitz, E., Schwamb, M.E., Lintott, C.J., Smith, A.M., 2013. Volunteering Versus Work for Pay: Incentives and Tradeoffs in Crowdsourcing. In: Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing.
34. Nov, O., Arazy, O., Anderson, D., 2011. Technology-Mediated Citizen Science Participation: A Motivational Model. In: Proceedings of the AAAI International Conference of Weblogs Soc. Media. pp. 249–256.
35. Oldham, G.R., Hackman, J.R., Not what it was and not what it will be: the future of job design research. J. Organ. Behav. 31 (2010), 463–479.
36. Pelli, D.G., Farell, B., Psychophysical methods. Handbook of Optics, 2010, McGraw-Hill, New York, 3.1–3.12.
37. Ponciano, L., Brasileiro, F., Simpson, R., Smith, A., Volunteers' engagement in human computation for astronomy projects. Comput. Sci. Eng. 16 (2014), 52–59.
38. Prather, E.E., Cormier, S., Wallace, C.S., Lintott, C., Raddick, M.J., Smith, A., Measuring the conceptual understandings of citizen scientists participating in Zooniverse Projects: a first approach. Astron. Educ. Rev., 2013, 12.
39. Prestopnik, N.R., Crowston, K., 2012. Citizen science system assemblages: understanding the technologies that support crowdsourced science. In: Proceedings of the 2012 iConference, iConference ’12. ACM, New York, NY, USA. pp. 168–176.
40. Raddick, J., Bracey, G., Gay, P.L., Lintott, C.J., Cardamone, C., Murray, P., Schawinski, K., Szalay, A.S., Vandenberg, J., Galaxy Zoo: motivations of citizen scientists. Astron. Educ. Rev., 2013, 12.
41. Raddick, M.J., Bracey, G., Gay, P.L., Lintott, C.J., Murray, P., Schawinski, K., Szalay, A.S., Vandenberg, J., 2009. Galaxy Zoo: Exploring the Motivations of Citizen Science Volunteers. ArXiv09092925 Astro-Ph Physicsphysics.
42. Reed, J., Raddick, M.J., Lardner, A., Carney, K., 2013. An Exploratory Factor Analysis of Motivations for Participating in Zooniverse, a Collection of Virtual Citizen Science Projects. In: Proceedings of the 46th Hawaii International Conference on System Sciences (HICSS). pp. 610–619.
43. Reed, J., Rodriguez, W., Rickhoff, A., 2012. A framework for defining and describing key design features of virtual citizen science projects. In: Proceedings of the 2012 iConference, iConference ’12. ACM, New York, NY, USA, pp. 623–625.
44. Rotman, D., Preece, J., Hammock, J., Procita, K., Hansen, D., Parr, C., Lewis, D., Jacobs, D., 2012. Dynamic changes in motivation in collaborative citizen-science projects. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12. ACM, New York, NY, USA. pp. 217–226.
45. Schmidt, M.-T., Building workflow business objects. Patel, D.D., Sutherland, D.J., Miller, J., (eds.) Business Object Design and Implementation II, 1998, Springer, London, 64–76.
46. Sprinks, J., Morley, J., Bamford, S., Houghton, R., 2015a. Keeping citizen scientists interested: the importance of task workflow design. In: Proceedings of the Citizens Science Conference. San Jose, CA.
47. Sprinks, J., Morley, J.G., Houghton, R., Bamford, S., 2015b. The Impact of Task Workflow Design on VGI Citizen Science Platforms. GIS Res. UK 2015 Leeds UK.
48. Swanson, A., Kosmala, M., Lintott, C., Simpson, R., Smith, A., Packer, C., Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna. Sci. Data, 2, 2015, 150026.
49. Tinati, R., Van Kleek, M., Simperl, E., Luczak-Rösch, M., Simpson, R., Shadbolt, N., 2015. Designing for citizen data analysis: a cross-sectional case study of a multi-domain citizen science platform. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15. ACM, New York, NY, USA, pp. 4069–4078. doi: 10.1145/2702123.2702420.