Crowdsourcing for usability testing

Proceedings of the ASIST Annual Meeting

Volumn 49, Issue 1, 2012, Pages

  Liu, Di ^a   Bias, Randolph G ^a   Lease, Matthew ^a   Kuipers, Rebecca ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Crowd sourcing; Mechanical Turk; Remote usability testing; Usability; Usability testing

Indexed keywords

EID: 84878567481     PISSN: None     EISSN: 15508390     Source Type: Journal    
DOI: 10.1002/meet.14504901100     Document Type: Article

References (34)

1. Amazon 2011. Amazon Mechanical Turk. http://aws.amazon.com/mturk/ #pricing. Visited April 2012.
2. Bailey, G. D. 1993. Iterative methodology and designer training in human-computer interface design, INTERCHI '93, 198-205.
3. Bias, R. G., &Huang, S. C. 2010. Remote, remote, remote, remote usability testing. Proceedings of the Society for Technical Communication Summit, May, Dallas.
4. nd edition: Update for the Internet age. San Francisco: Morgan Kaufmann.
5. Caraway, B., 2010. Online labour markets: an inquiry into oDesk providers. In Work Organizations, Labour and Globalisation. Vol. 4, Number 2. Autumn, 2010 (111-125).
6. Chen, D., &Dolan, W. 2011. Collecting Highly Parallel Data for Paraphrase Evaluation. In Proc. of the Association for Computational Linguistics (ACL).
7. Crowd Flower 2011. FAQ - Self-Service - CrowdFlower. http://api. crowdflower.com/solutions/self-service. Visited April 2012.
8. Downs, J. S., Holbrook, M. B., Sheng, S., &Cranor, L. F. 2010 Are your participants gaming the system? Screening Mechanical Turk Workers. In CHI '10: Proceedings of the 28th international conference on Human factors in computing systems, ACM, pp. 2399-2402. 2010
9. Frei, B., 2009 "Paid crowdsourcing: Current state &progress toward mainstream business use," Whitepaper Produced by Smartsheet.com, 2009.
10. Heymann P. &Garcia-Molina H. (2011). Turkalytics: analytics for human computation. Proceedings of the 20th international conference on World wide web (WWW), pp. 477-486, 2011
11. Rzeszotarski, J.M. &Kittur, A. 2011. Instrumenting the crowd: using implicit behavioral measures to predict task performance. Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST), 13-22, 2011.
12. Howe, J. 2006. The Rise of Crowdsourcing.
13. Howe, J. 2008, Crowdsourcing: A Definition. http://crowdsourcing.typepad. com. Visited April 2012.
14. Howe, J. 2008. Crowdsourcing: Now with a real business model! http://www.wired.com/epicenter/2008/12/crowdsourcing-n. Visited April 2012.
15. Ipeirotis, P. 2010a. Demographics of Mechanical Turk. CeDER-10-01 working paper, New York University.
16. Ipeirotis, P. 2010b Be a Top Mechanical Turk Worker: You Need $5 and 5 Minutes. http://behind-the-enemy-lines.blogspot.com/2010/10/be-top- mechanical-turk-worker-you-need.html. Visited April 2012.
17. Ipeirotis, P., Provost, F., &Wang, J. 2010. Quality Management on Amazon Mechanical Turk. In HCOMP '10, New York, NY, USA, 2010.
18. ISO 9241-11 Ergonomic requirements for office work with visual display teminals (VDT)s - Part 11 Guidance on Usability, International Standard, 1998
19. Kapelner, A., &Chandler, D. 2010. Preventing Satisficing in Online Surveys: A "Kapcha" to Ensure Higher Quality Data. In CrowdConf Proceedings, 2010.
20. Kittur, A., Chi, E. H., &Suh, B. 2008. Crowdsourcing user studies with Mechanical Turk. In the Proceedings of the ACM CHI Conference.
21. Kochhar, S., Mazzocchi, S., &Paritosh P. 2010 The anatomy of a large-scale human computation engine. In HCOMP '10, New York, NY, USA, 2010.
22. Mason, W. and Watts, D.J. 2010. Financial incentives and the performance of crowds. In ACM SIGKDD Explorations Newsletter v. 11 no. 2, pp.100-108, 2010.
23. Mason, W. and Suri, S. 2010. A Guide to Conducting Behavioral Research on Amazon's Mechanical Turk. In Social Science Research Network, pre-print. 2010.
24. Molich, R., &Dumas, J. 2008. Comparative usability evaluation (CUE-4). Journal of Behaviour &Information Technology, 27, 3, 263-281.
25. Nielsen, J. 2003. Usability 101: Introduction to Usability. http://www.useit.com/alertbox/20030825.html. Visited April 2012.
26. Nielsen, J., &Mack, R. L. (Eds.) 1994. Usability inspection methods. New York: Wiley and Sons, Inc.
27. Quinn, A. J., &Bederson, B. B., 2011 Human Computation: A Survey and Taxonomy of a Growing Field. In Proceedings of CHI 2011, May 7-12, 2011, Vancouver, BC, Canada.
28. Ross, J., Irani, I., Silberman, M. Six, Zaldivar, A., &Tomlinson, B. 2010. "Who are the Crowdworkers?: Shifting Demographics in Amazon Mechanical Turk". In: CHI EA 2010. (2863-2872).
29. Rubin, J., &Chisnell, D., 2008. Handbook of usability testing: How to plan, design, and conduct effective tests. Hoboken, NJ: Wiley.
30. Spool, J., &Schroeder, W. 2001. Test web sites: five users is nowhere enough. In Proc. CHI2001 Extended Abstract. Seattle: ACM Press. 285-286
31. uTest 2011a. About Us. http://www.utest.com/company/. Visited April 2012.
32. uTest 2011b. How It Works. http://www.utest.com/how-it-works/agile- testing/. Visited April 2012.
33. uTest 2011c. Pricing Information. http://www.utest.com/pricing. Visited April 2012.
34. Winsor, J. 2009. Crowdsourcing: What it means for innovation. http://www.businessweek.com/innovate/content/jun2009/id20090615-946326.html Visited April 2012.