Dynamic and goal-based quality management for human-based electronic services

International Journal of Cooperative Information Systems

Volumn 21, Issue 1, 2012, Pages 3-29

  Kern, Robert ^a   Thies, Hans ^b   Zirpins, Christian ^a   Satzger, Gerhard ^a  

^a KARLSRUHE INSTITUTE OF TECHNOLOGY   (Germany)

^b SAP RESEARCH   (Germany)

Author keywords

Crowdsourcing; eServices; human computation; people services; statistical quality control

Indexed keywords

EID: 84858841346     PISSN: 02188430     EISSN: None     Source Type: Journal    
DOI: 10.1142/S0218843012400011     Document Type: Conference Paper

References (28)

1. P. G. Ipeirotis, Analyzing the amazon mechanical turk marketplace, XRDS 17 (2010) 16-21.
2. R. Kern, C. Zirpins and S. Agarwal, Managing quality of Human-Based eServices, in Service-Oriented Computing-ICSOC 2008 Workshops, ICSOC 2008 International Workshops, Sydney, Australia, December 1st, 2008, Revised Selected Papers, eds. G. Feuerlicht and W. Lamersdorf, Lecture Notes in Computer Science, Vol. 5472 (Springer, 2009), pp. 304-309.
3. R. Kern, H. Thies, C. Bauer and G. Satzger, Quality assurance for human-based electronic services: A decision matrix for choosing the right approach, in Current Trends in Web Engineering, eds. F. Daniel and F. Facca, Lecture Notes in Computer Science, Vol. 6385 (Springer Berlin/Heidelberg, 2010), pp. 421-424.
4. D. Montgomery, Introduction to Statistical Quality Control, 6th edn. (Wiley & Sons, New York, NY, USA, 2008).
5. J. Juran and A. Godfrey, Juran's Quality Handbook, 5th edn. (McGraw-Hill, New York, NY, USA, 2000).
6. R. Kern, H. Thies and G. Satzger, Statistical quality control for human-based electronic services, in Service-Oriented Computing-8th International Conference, ICSOC 2010, San Francisco, CA, USA, December 7-10, 2010, eds. P. P. Maglio, M. Weske, J. Yang and M. Fantinato, Lecture Notes in Computer Science, Vol. 6470 (Springer, 2010), pp. 243-257.
7. H. Dodge and M. Torrey, Additional continuous sampling inspection plans, Ind. Qual. Control (7) (1951) 7-12.
8. G. J. Lieberman and H. Solomon, Multi-Level continuous sampling plans, Ann. Math. Stat. 26(4) (1955) 686-704.
9. D. T. Gosh, An optimum continuous sampling plan CSP-2 with k = i to minimise the amount of inspection when incoming quality p follows a distribution, Indian J. Stat. 58(1) (1996) 105-117.
10. H.-J. Mittag and H. Rinne, Statistical Methods of Quality Assurance (Chapman and Hall/CRC, 1993).
11. R. Wang and C. Chen, Minimum average fraction inspected for continuous sampling plan CSP-1 under inspection error, J. Appl. Stat. 24(5) (1997) 539-548. (Pubitemid 127661898)
12. R. Kern, C. Bauer, H. Thies and G. Satzger, Validating results of human-based electronic services leveraging multiple reviewers, in Proc. 16th Americas Conf. Information Systems (AMCIS), (AIS Electronic Library, 2010).
13. R. L. Graham, D. E. Knuth and O. Patashnik, Concrete Mathematics: A Foundation for Computer Science, 2nd edn. (Addison-Wesley Professional, 1994).
14. D. Bermbach, R. Kern, P. Wichmann, S. Rath, C. Zirpins, D. Bermbach and C. Zir-pins, An extendable toolkit for managing quality of human-based electronic services, in Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence (2011).
15. D. Lopresti, Optical character recognition errors and their effects on natural language processing, Int. J. Doc. Anal. Recognit. (IJDAR) 12(3) (2009) 141-151.
16. J. Ross, L. Irani, M. Silberman, A. Zaldivar and B. Tomlinson, Who are the crowd-workers?: Shifting demographics in mechanical turk, in Proc. 28th Int. Conf. Extended Abstracts on Human Factors in Computing Systems (2010), pp. 2863-2872.
17. R. Snow, B. O'Connor, D. Jurafsky and A. Y. Ng, Cheap and fast-but is it good? evaluating non-expert annotations for natural language tasks, in EMNLP'08: Proc. Conf. Empirical Methods in Natural Language Processing (Association for Computational Linguistics, Stroudsburg, USA, 2008), pp. 254-263.
18. A. Sorokin and D. Forsyth, Utility data annotation with amazon mechanical turk, in CVPRW'08: Proc. Conf. Computer Vision and Pattern Recognition Workshops (IEEE Computer Society, Washington, WA, USA, June 2008), pp. 1-8.
19. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma and J. Movellan, Whose vote should count more: Optimal integration of labels from labelers of unknown expertise, in Advances in Neural Information Processing Systems, Vol. 22 (2009), pp. 2035-2043.
20. G. Little, L. B. Chilton, M. Goldman and R. C. Miller, Turkit: Human computation algorithms on mechanical turk, in Proc. 23nd Annual ACM Symposium on User Interface Software and Technology, UIST '10 (Association for Computing Machinery, New York, NY, USA, 2010), pp. 57-66.
21. P. Ipeirotis, F. Provost and J. Wang, Quality management on amazon mechanical turk in Proc. ACM SIGKDD Workshop on Human Computation (Association for Computing Machinery, New York, NY, USA, 2010), pp. 64-67.
22. A. Dawid and A. Skene, Maximum likelihood estimation of observer Error-Rates using the EM algorithm, J. R. Stat. Soc. 28(1) (1979) 20-28.
23. V. Raykar, S. Yu, L. Zhao, G. Valadez, C. Florin, L. Bogoni and L. Moy, Learning from crowds, J. Mach. Learn. Res. 99 (2010) 1297-1322.
24. J. de A. Condorcet, Essai sur l'application de l'analyse a la Probabilite des Decisions Rendues a la Pluralite des voix, 1st edn. (Imprimerie royale, Paris, France, 1785).
25. G. Latif-Shabgahi, J. Bass and S. Bennett, A taxonomy for software voting algorithms used in safety-critical systems, IEEE Trans. Reliab. 53(3) (2004) 319-328.
26. N. Littlestone and M. K. Warmuth, The weighted majority algorithm, Inf. Comput. 108 (1994) 212-261.
27. M. Revow, C. K. I. Williams and G. E. Hinton, Using generative models for handwritten digit recognition, IEEE Trans. Pattern Anal. Mach. Intell. 18(6) (1996) 592-606. (Pubitemid 126773895)
28. J. Marschak and R. Radner, Economic Theory of Teams, Chapter 1, Cowles Foundation Discussion Papers 59a, Cowles Foundation for Research in Economics, Yale University (1958).