Economic and Ethical Background of Crowdsourcing for Speech

Crowdsourcing for Speech Processing: Applications to Data Collection, Transcription and Assessment

Volumn , Issue , 2013, Pages 303-334

  Adda, Gilles ^a   Mariani, Joseph J ^a,b   Besacier, Laurent ^c   Gelas, Hadrien ^c,d  

^a UFR 919 Laboratoire d'Informatique Pour la Mécanique et les Sciences de l'Ingénieur   (France)

^b IMMI CNRS   (France)

^c UNIV GRENOBLE ALPES   (France)

^d DDL CNRS   (France)

Author keywords

Amazon mechanical turk (MTurk); Crowdsourcing; Economic issues; Ethical issues; Legal issues; Speech processing; Under resourced languages; Workers

Indexed keywords

EID: 84886342228     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9781118541241.ch11     Document Type: Chapter

References (59)

1. Abate S, Menzel W and Tafila B (2005) An amharic speech corpus for large vocabulary continuous speech recognition. Interspeech, pp. 67-76, Lisbon.
2. Adda G and Mariani J (2010) Language resources and Amazon Mechanical Turk: legal, ethical and other issues. LISLR2010, Legal Issues for Sharing Language Resources Workshop, LREC2010, Malta.
3. Adda G, Sagot B, Fort K and Mariani JJ (2011) Crowdsourcing for language resource development: critical analysis of Amazon Mechanical Turk overpowering use. Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics (L&TC 2011), pp. 304-308, Poznan, Poland.
4. Akerlof GA (1970) The market for "lemons": quality uncertainty and the market mechanism. Quarterly Journal of Economics 84(3), 488-500.
5. Albright P (2009) Is crowdsourcing an opportunity or threat? http://www.computer.org/portal/web/ buildyourcareer/crowdsourcing (accessed 17 October 2012).
6. Arthur C (2006) What is the 1% rule? The Guardian.
7. Bederson BB and Quinn AJ (2011) Web workers unite! addressing challenges of online laborers. Proceedings of the 2011 Annual Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '11), pp. 97-106. ACM.
8. Berment V (2004) Méthodes pour informatiser les langues et les groupes de langues "peu dotées." PhD thesis, Université Joseph Fourier, Grenoble.
9. Biewald L (2010) Better crowdsourcing through automated methods for quality control. SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation.
10. Callison-Burch C and DredzeM(2010) Creating speech and language data with amazon's mechanical turk. CSLDAMT '10: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, CA.
11. Chamberlain J, Poesio M and Kruschwitz U (2008) Phrase detectives: a web-based collaborative annotation game. Proceedings of the International Conference on Semantic Systems (I-Semantics'08), Graz.
12. Chen DL and Dolan WB (2011) Building a persistent workforce on mechanical turk for multilingual data collection. Proceedings of The 3rd Human Computation Workshop (HCOMP 2011).
13. Chiarella S (2011) Cooking with Sharon. http://mechanicalturk.typepad.com/blog/2011/07/ cooking-with-sharon-tip-3-manage-your-reputation.html (accessed 17 October 20102).
14. Chilton LB, Horton JJ, Miller RC and Azenkot S (2010) Task search in a human computation market. Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10), pp. 1-9.
15. Draxler C (1997) WWWTranscribe-a modular transcription system based on the World Wide Web. Proceedings of Eurospeech, pp. 1691-1694, Rhodes.
16. DraxlerCand SteffenA(2005) Ph@ttsessionz: recording 1000 adolescent speakers in schools in germany Proceedings of Interspeech, pp. 1597-1600, Lisbon.
17. Schultz T and Kirchhoff K (eds) (2006) Multilingual Speech Processing. Academic Press, Berlington, MA.
18. Faridani S, Hartmann B and Ipeirotis PG (2011) Whats the right price? pricing tasks for finishing on time. Proceedings of the 3rd Human Computation Workshop (HCOMP 2011).
19. Felstiner A (2011) Working the crowd: employment and labor law in the crowdsourcing industry. Berkeley Journal of Employment and Labor Law 32(1), 143-204.
20. Fort K, Adda G and Cohen KB (2011) Amazon Mechanical Turk: gold mine or coal mine? Computational Linguistics (editorial) 37(2), 413-420.
21. Frei B (2009) Paid crowdsourcing: current state & progress towards mainstream business use. Smartsheet white paper. http://www.smartsheet.com/files/haymaker/Paid CrowdsourcingSept2009- ReleaseVersion-Smartsheet.pdf (accessed 17 October 2012).
22. Gelas H, Besacier L and Pellegrino F (2012) Developments of swahili resources for an automatic speech recognition system SLTU.
23. Gneezy U and Rustichini A (2000) Pay enough or don't pay at all. Quarterly Journal of Economics 115(3), 791- 810.
24. Gruenstein E, Mcgraw I and Sutherl A (2009) A self-transcribing speech corpus: collecting continuous speech with an online educational game the Speech and Language Technology in Education (SLaTE) Workshop,Warwickshire, England.
25. Holmstrom B (1979) Moral hazard and observability. Bell Journal of Economics 10(1), 74-91.
26. Ipeirotis P (2008) Why People Participate on Mechanical Turk, Now Tabulated. http://www.behind-theenemy- lines.com/2008/09/why-people-participate-on-mechanical.html (accessed 17 October 2012).
27. Ipeirotis P (2010a) Be a Top Mechanical TurkWorker: You Need $5 and 5 Minutes. http://www.behind-theenemy- lines.com/2010/10/be-top-mechanical-turk-worker-you-need.html (accessed 17 October 2012).
28. Ipeirotis P (2010b) Demographics of Mechanical Turk CeDER Working Papers http://hdl.handle.net/ 2451/29585. CeDER-10-01 (accessed 17 October 2012).
29. Ipeirotis P (2010c) A Plea to Amazon: Fix Mechanical Turk! http://behind-the-enemy-lines .blogspot.com/2010/10/plea-to-amazon-fix-mechanical-turk.html (accessed 17 October 2012).
30. Ipeirotis P (2011a) Do Mechanical Turk Workers Lie about Their Location? http://www.behind-theenemy- lines.com/2011/03/do-mechanical-turk-workers-lie-about.html (accessed 17 October 2012).
31. Ipeirotis P (2011b) Does Lack of Reputation Help the Crowdsourcing Industry? http://www.behind-theenemy- lines.com/2011/11/does-lack-of-reputation-help.html (accessed 17 October 2012).
32. Ipeirotis P (2011c) Pay Enough or Don't Pay at All. http://www.behind-the-enemy-lines.com/ 2011/05/pay-enough-or-dont-pay-at-all.html (accessed 17 October 2012).
33. Ipeirotis P (2012a) Mechanical Turk vs. Odesk: My Experiences. http://www.behind-the-enemy-lines.com/2012/02/mturk-vs-odesk-my-experiences.html (accessed 17 October 2012).
34. Ipeirotis P (2012b) The Need for Standardization in Crowdsourcing. http://www.behind-the-enemylines. com/2012/02/need-for-standardization-in.html (accessed 17 October 2012).
35. Ipeirotis P (2012c) Philippines: The Country That Never Sleeps. http://www.behind-the-enemylines. com/2012/04/when-is-world-working-odesk-edition-or.html (accessed 17 October 2012).
36. Kittur A, Smus B, Khamkar S and Kraut RE (2011) Crowdforge: crowdsourcing complex work. Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11), pp. 43-52.
37. Kochhar S, Mazzocchi S and Paritosh P (2010) The anatomy of a large-scale human computation engine. Proceedings of Human Computation Workshop at the 16th ACM SIKDD Conference on Knowledge Discovery and Data Mining, KDD 2010, Washington, DC.
38. Le V and Besacier L (2009) Automatic speech recognition for under-resourced languages: application to vietnamese language. Audio, Speech, and Language Processing, IEEE Transactions on 17(8), 1471-1482.
39. Little G, Chilton LB, Goldman M and Miller RC (2010) Turkit: human computation algorithms on mechanical turk. Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (UIST '10), pp. 57-66.
40. Marge M, Banerjee S and Rudnicky AI (2010) Using the Amazon Mechanical Turk for transcription of spoken language. IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 5270- 5273, Dallas, TX.
41. Mariani J (2011) Ethical dimension of crowdsourcing. Special session on Crowdsourcing, Interspeech'2011, Firenze.
42. Mariani J, Besnier JM, Bordé J, Cornu JM, Farge M, Ganascia JG, Haton JP and Serverin E (2009) Pour une éthique de la recherche en Sciences et Technologies de l'Information et de la Communication (STIC). Technical report, COMETS-CNRS. http://www.cnrs.fr/fr/organisme/ethique/comets/avis.htm (accessed 17 October 2012).
43. Marten L (2006) Swahili in The Encyclopedia of Languages and Linguistics, 2nd edn, vol. 12 (ed. K Brown). Elsevier, Oxford, pp. 304-308.
44. McGraw I, Gruenstein A and Sutherland A (2009) A self-labeling speech corpus: Collecting spoken words with an online educational game. Proceedings of Interspeech, pp. 3031-3034, Brighton, UK.
45. Novotney S and Callison-Burch C (2010) Cheap, fast and good enough: automatic speech recognition with non-expert transcription. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (HLT '10), pp. 207-215, Los Angeles, CA.
46. Parent G and Eskenazi M (2010) Toward better crowdsourced transcription: transcription of a year of the let's go bus information system data. Proceedings of IEEE Workshop on Spoken Language Technology, pp. 312-317, Berkeley, CA.
47. Quinn AJ and Bederson BB (2011) Human computation: a survey and taxonomy of a growing field. Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems (CHI EA '11), pp. 1403-1412.
48. Rashid A, Weckers J and Lucas E (2009) Software engineering, ethics in a digital world. IEEE Computing Now. http://www2.computer.org/portal/web/computingnow/0709/theme/co1_ softengethics (accessed 17 October 2012).
49. Ross J, Irani L, Silberman MS, Zaldivar A and TomlinsonB(2010) Who are the crowdworkers?: shifting demographics in mechanical turk. Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). ACM, New York.
50. Ross J, Zaldivar A, Irani L and Tomlinson B (2009) Who Are the Turkers?Worker Demographics in Amazon Mechanical Turk Social Code Report 2009-01. http://www.ics.uci.edu/~jwross/pubs/SocialCode- 2009-01.pdf (accessed 17 October 2012).
51. Scheffler T, Roller R and Reithinger N (2011) Speecheval: a domain-independent user simulation platform for spoken dialog system evaluation. IWSDS 2011, Granada.
52. Silberman MS, Ross J, Irani L and Tomlinson B (2010) Sellers' problems in human computation markets. Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10), pp. 18-21.
53. SnowR, O'Connor B, Jurafsky D and NgAY(2008) Cheap and fast-but is it good? evaluating non-expert annotations for natural language tasks. Proceedings of EMNLP 2008.
54. Virginia Commonwealth University (2009) VCU Institutional Review Board Written Policies and Procedures. http://www.research.vcu.edu/irb/wpp/flash/XVII-2.htm (accessed 17 October 2012).
55. von Ahn L (2006) Games with a purpose. IEEE Computer Magazine, pp. 96-98.
56. Wessel F and Ney H (2005) Unsupervised training of acoustic models for large vocabulary continuous speech recognition. IEEE Transactions on Speech and Audio Processing 13(1), 23-31.
57. Wolfson SM and LeaseM(2011) Look before you leap: Legal pitfalls of crowdsourcing. Proceedings of the American Society for Information Science and Technology 48(1), 1-10.
58. Zittrain J (2008a) The Future of the Internet-And How to Stop It. Yale University Press, New Haven, CT.
59. Zittrain J (2008b) Ubiquitous human computing. Philosophical Transaction of the Royal Society A 366(1881), 3813- 3821.