Privacy and reliability in crowdsourcing service delivery

Annual SRII Global Conference, SRII

Volumn , Issue , 2012, Pages 55-60

  Varshney, Lav R ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CROWDSOURCING; FUSION RULE; SERVICE DELIVERY; STOCHASTIC PERTURBATIONS; THRESHOLD CONDITION;

MATHEMATICAL MODELS;

RELIABILITY;

EID: 84870774489     PISSN: 21660778     EISSN: 21660786     Source Type: Conference Proceeding    
DOI: 10.1109/SRII.2012.17     Document Type: Conference Paper

References (32)

1. Y. Benkler, The Wealth of Networks: How Social Production Transforms Markets and Freedom. New Haven, CT: Yale University Press, 2006.
2. D. Tapscott and A. D. Williams, Wikinomics: How Mass Collaboration Changes Everything, expanded ed. New York: Portfolio Penguin, 2006.
3. D. Tapscott and A. D. Williams, Macrowikinomics: Rebooting Business and the World. New York: Portfolio Penguin, 2010.
4. E. H. Bokelberg and L. R. Varshney, "Liquid: Harnessing the passion of the crowd within the enterprise, " 2012, submitted.
5. J. Ross, L. Irani, M. S. Silberman, A. Zaldivar, and B. Tomlinson, "Who are the crowdworkers? shifting demographics in Mechanical Turk, " in Proc. 28th SIGCHI Conf. Hum. Factors Comput. Syst. (CHI 2010), Apr. 2010, pp. 2863-2872.
6. J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor, "Are your participants gaming the system? screening Mechanical Turk workers, " in Proc. 28th SIGCHI Conf. Hum. Factors Comput. Syst. (CHI 2010), Apr. 2010, pp. 2399-2402.
7. P. G. Ipeirotis, F. Provost, and J. Wang, "Quality management on Amazon Mechanical Turk, " in Proc. ACM SIGKDD Workshop Human Comput. (HCOMP'10), Jul. 2010, pp. 64-67.
8. M. Lease, "On quality control and machine learning in crowdsourcing, " in Proc. AAAI Workshop Human Comput. (HCOMP'11), Aug. 2011, pp. 97-102.
9. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng, "Cheap and fast- but is it good?: Evaluating non-expert annotations for natural language tasks, " in Proc. Conf. Empirical Meth. Natural Language Process. (EMNLP'08), Oct. 2008, pp. 254-263.
10. J. Le, A. Edmonds, V. Hester, and L. Biewald, "Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution, " in Proc. SIGIR 2010 Workshop Crowdsourcing for Search Eval. (CSE 2010), Jul. 2010, pp. 17-20.
11. M. Marge, S. Banerjee, and A. I. Rudnicky, "Using the Amazon Mechanical Turk for transcription of spoken language, " in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP 2010, Mar. 2010, pp. 5270-5273.
12. R. Kern, H. Thies, and G. Satzger, "Statistical quality control for humanbased electronic services, " in Service-Oriented Computing, ser. Lecture Notes in Computer Science, P. P. Maglio, M. Weske, J. Yang, and M. Fantinato, Eds. Berlin: Springer, 2010, vol. 6470, pp. 243-257.
13. H. J. Jung and M. Lease, "Improving consensus accuracy via zscore and weighted voting, " in Proc. AAAI Workshop Human Comput. (HCOMP'11), Aug. 2011, pp. 113-118.
14. D. R. Karger, S. Oh, and D. Shah, "Budget-optimal crowdsourcing using low-rank matrix approximations, " in Proc. 49th Annu. Allerton Conf. Commun. Control Comput., Sep. 2011, pp. 284-291.
15. D. R. Karger, S. Oh, and D. Shah, "Iterative learning for reliable crowdsourcing systems, " in Advances in Neural Information Processing Systems 24, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, Eds. Cambridge, MA: MIT Press, 2011, pp. 1953-1961.
16. Y.-A. Sun, S. Roy, and G. D. Little, "Beyond independent agreement: A tournament selection approach for quality assurance of human computation tasks, " in Proc. AAAI Workshop Human Comput. (HCOMP'11), Aug. 2011, pp. 113-118.
17. G. D. Forney Jr., "Exponential error bounds for erasure, list, and decision feedback schemes, " IEEE Trans. Inf. Theory, vol. IT-14, no. 2, pp. 206-220, Mar. 1968.
18. G. Little and Y.-A. Sun, "Human OCR: Insights from a complex human computation process, " in CHI Workshop on Crowdsourcing Human Comput., May 2011.
19. D. A. Grier, "Error identification and correction in human computation: Lessons from the WPA, " in Proc. AAAI Workshop Human Comput. (HCOMP'11), Aug. 2011, pp. 32-36.
20. L. von Ahn, "Human computation, " Ph.D. dissertation, Carnegie Mellon University, 2005.
21. E. Bursztein, S. Bethard, C. Fabry, J. C. Mitchell, and D. Jurafsky, "How good are humans at solving CAPTCHAs? a large scale evaluation, " in Proc. 2010 IEEE Symp. Security Privacy, May 2010, pp. 399-413.
22. V. C. Raykar and S. Yu, "Eliminating spammers and ranking annotators for crowdsourced labeling tasks, " J. Mach. Learn. Res., vol. 13, pp. 491-518, Feb. 2012.
23. M. Bernstein, E. H. Chi, L. Chilton, B. Hartmann, A. Kittur, and R. C. Miller, "Crowdsourcing and human computation: systems, studies and platforms, " in Proc. 2011 Annu. Conf. Hum. Factors Comput. Syst. (CHI 2011), May 2011, pp. 53-56.
24. E. D. Karnin, E. Walach, and T. Drory, "Crowdsourcing in the document processing practice, " in Current Trends in Web Engineering, ser. Lecture Notes in Computer Science, F. Daniel and F. M. Facca, Eds. Berlin: Springer, 2010, vol. 6385, pp. 408-411.
25. X. Xiao, Y. Tao, and M. Chen, "Optimal random perturbation at multiple privacy levels, " in Proc. 35th Int. Conf. Very Large Data Bases (VLDB'09), Aug. 2009, pp. 814-825.
26. D. Y. Zhang, Y. Zeng, L. Wang, H. Li, and Y. Geng, "Modeling and evaluating information leakage caused by inferences in supply chains, " Comput. Ind., vol. 62, no. 3, pp. 351-363, Apr. 2011.
27. W. H. Pierce, Failure-Tolerant Computer Design. New York: Academic Press, 1965.
28. C. Dwork, "Differential privacy: A survey of results, " in Theory and Applications of Models of Computation, ser. Lecture Notes in Computer Science, M. Agrawal, D. Du, Z. Duan, and A. Li, Eds. Berlin: Springer, 2008, vol. 4978, pp. 1-19.
29. A. Kolupaev and J. Ogijenko, "CAPTCHAs: Humans vs. bots, " IEEE Security Privacy, vol. 6, no. 1, pp. 68-70, Jan.-Feb. 2008.
30. M. Mézard and A. Montanari, Information, Physics, and Computation. Oxford: Oxford University Press, 2009.
31. P. K. Varshney, Distributed Detection and Data Fusion. New York: Springer-Verlag, 1997.
32. M. Mézard and A. Montanari, "Reconstruction on trees and spin glass transition, " J. Stat. Phys., vol. 124, no. 6, pp. 1317-1350, Sep. 2006.