Reliable crowdsourcing for multi-class labeling using coding theory

IEEE Journal on Selected Topics in Signal Processing

Volumn 8, Issue 4, 2014, Pages 667-679

  Vempaty, Aditya ^a   Varshney, Lav R ^b,c   Varshney, Pramod K ^a  

^a Syracuse University   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

^c University of Illinois at Urbana Champaign   (United States)

Author keywords

Crowdsourcing; error control codes; multi class labeling; quality assurance

Indexed keywords

COMPUTER PROGRAMMING; QUALITY ASSURANCE;

AMAZON MECHANICAL TURKS; CLASSIFICATION PERFORMANCE; CROWDSOURCING; DECODING ALGORITHM; ERROR CONTROL CODE; IMPROVING SYSTEMS; MULTI-CLASS LABELING; REAL DATA SETS;

CODES (SYMBOLS);

EID: 84904601689     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2014.2316116     Document Type: Article

References (27)

1. A. Vempaty, L. R. Varshney, and P. K. Varshney, "Reliable classification by unreliable crowds," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP '13), May 2013, pp. 5558-5562.
2. J. Marschak and R. Radner, Economic Theory of Teams. New Haven, CT, USA: Yale Univ. Press, 1972.
3. D. Bollier, The Future of Work: What It Means for Individuals, Businesses, Markets and Governments. Washington, DC, USA: The Aspen Inst., 2011.
4. D. Tapscott and A. D. Williams, Wikinomics: How Mass Collaboration Changes Everything. New York, NY, USA: Portfolio Penguin, 2006, expanded ed.
5. D. Tapscott and A. D. Williams, Macrowikinomics: Rebooting Business and the World. New York, NY, USA: Portfolio Penguin, 2010.
6. P. K. Varshney, Distributed Detection and Data Fusion. New York, NY, USA: Springer-Verlag, 1997.
7. P. G. Ipeirotis, F. Provost, and J. Wang, "Quality management on Amazon Mechanical Turk," in Proc. ACM SIGKDD Workshop Human Comput. (HCOMP'10), Jul. 2010, pp. 64-67.
8. M. Lease, "On quality control and machine learning in crowdsourcing," in Proc. AAAI Workshop Human Comput. (HCOMP'11), Aug. 2011, pp. 97-102.
9. L. R. Varshney, "Privacy and reliability in crowdsourcing service delivery," in Proc. SRII Global Conf. '12, Jul. 2012, pp. 55-60.
10. J. Ross, L. Irani, M. S. Silberman, A. Zaldivar, and B. Tomlinson, "Who are the crowdworkers? Shifting demographics in Mechanical Turk," in Proc. 28th SIGCHI Conf. Hum. Factors Comput. Syst. (CHI '10), Apr. 2010, pp. 2863-2872.
11. J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor, "Are your participants gaming the system? Screening Mechanical Turk workers," in Proc. 28th SIGCHI Conf. Hum. Factors Comput. Syst. (CHI '10), Apr. 2010, pp. 2399-2402.
12. S. Branson, C. Wah, F. Schroff, B. Babenko, P. Welinder, P. Perona, and S. Belongie, "Visual recognition with humans in the loop," in Computer Vision-ECCV '10, ser. Lecture Notes in Computer Science, L. Buttyán, V. Gligor, and D. Westhoff, Eds. Berlin, Germany: Springer, 2010, vol. 6314, pp. 438-451.
13. T.-Y. Wang, Y. S. Han, P. K. Varshney, and P.-N. Chen, "Distributed fault-tolerant classification in wireless sensor networks," IEEE J. Sel. Areas Commun., vol. 23, no. 4, pp. 724-734, Apr. 2005.
14. C. Yao, P.-N. Chen, T.-Y. Wang, Y. S. Han, and P. K. Varshney, "Performance analysis and code design for minimum Hamming distance fusion in wireless sensor networks," IEEE Trans. Inf. Theory, vol. 53, no. 5, pp. 1716-1734, May 2007. (Pubitemid 46736812)
15. A. J. Quinn and B. B. Bederson, "Human computation: A survey and taxonomy of a growing field," in Proc. Annu. Conf. Human Factors Comput. Syst. (CHI '11), May 2011, pp. 1403-1412.
16. E. Law and L. von Ahn, Human Computation. San Rafael, CA, USA: Morgan & Claypool, 2011.
17. L. R. Varshney, "Participation in crowd systems," in Proc. 50th Annu. Allerton Conf. Commun. Control Comput., Oct. 2012, pp. 996-1001.
18. W. Mason and D. J. Watts, "Financial incentives and the "performance of crowds"," in Proc. ACM SIGKDD Workshop Human Comput. (HCOMP'09), Jun. 2009, pp. 77-85.
19. S.-W. Huang and W.-T. Fu, "Don't hide in the crowd! increasing social transparency between peer workers improves crowdsourcing outcomes," in Proc. SIGCHI Conf. Human Factors Comput. Syst. (CHI '13), Apr. 2013, pp. 621-630.
20. G.-J. Qi, C. C. Aggarwal, J. Han, and T. Huang, "Mining collective intelligence in diverse groups," in Proc. 22nd Int. Conf. World Wide Web (WWW'13), May 2013, pp. 1041-1052.
21. L. Berti-Equille, A. D. Sarma, X. Dong, A. Marian, and D. Srivastava, "Sailing the information ocean with awareness of currents: Discovery and application of source dependence," in Proc. 4th Bienn. Conf. Innov. Data Syst. Res. (CIDR), Jan. 2009.
22. D. A. Grier, "Error identification and correction in human computation: Lessons from the WPA," in Proc. AAAI Workshop Human Comput. (HCOMP'11), Aug. 2011, pp. 32-36.
23. D. R. Karger, S. Oh, and D. Shah, "Budget-optimal crowdsourcing using low-rank matrix approximations," in Proc. 49th Annu. Allerton Conf. Commun. Control Comput., Sep. 2011, pp. 284-291.
24. D. R. Karger, S. Oh, and D. Shah, "Iterative learning for reliable crowdsourcing systems," in Advances in Neural Information Processing Systems 24, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, Eds. Cambridge, MA, USA: MIT Press, 2011, pp. 1953-1961.
25. D. R. Karger, S. Oh, and D. Shah, "Efficient crowdsourcing for multiclass labeling," in Proc. ACM SIGMETRICS Int. Conf. Meas. Model. Comput. Syst., Jun. 2013, pp. 81-92.
26. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng, "Cheap and fast-But is it good?: Evaluating non-expert annotations for natural language tasks," in Proc. Conf. Empir. Meth. Natural Lang. Process. (EMNLP'08), Oct. 2008, pp. 254-263.
27. J. Rocker, C. M. Yauch, S. Yenduri, L.A. Perkins, and F. Zand, "Paperbased dichotomous key to computer based application for biological indentification," J. Comput. Sci. Coll., vol. 22, no. 5, pp. 30-38, May 2007.