Quality control in crowdsourcing: An objective measurement approach to identifying and correcting rater effects in the social evaluation of products and services

AAAI Spring Symposium - Technical Report

Volumn SS-12-06, Issue , 2012, Pages 32-37

  Lange, Rense ^a   Lange, Xavier ^b  

^a Integrated Knowledge Systems   (United States)

^b Tureus Inc   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COHERENT MEASUREMENT; CONTROL MECHANISM; CORRECTIVE ACTIONS; CROWDSOURCING; CUSTOMER FEEDBACK; DATA SETS; OBJECTIVE MEASUREMENT; PRODUCTS AND SERVICES; SOCIAL EVALUATION;

QUALITY CONTROL;

EID: 84864965194     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (13)

1. Bond, T.G., and Fox, C.M. 2007. Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
2. Churchill, G.A. 1979. A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16, 1, 64-73.
3. Engelhard, G. 1997. Constructing rater and task banks for performance assessments. Journal of Outcome Measurement, 1(1), 19-33.
4. Lange, R. 2007. Binary Items and Beyond: A Simulation of Computer Adaptive Testing Using the Rasch Partial Credit Model. In: Smith, E. and Smith, R. (Eds.) Rasch Measurement: Advanced and Specialized Applications. Pp. 148-180, Maple Grove, MN: JAM Press.
5. Lange, R., Moran, J., Greiff, W.R., and Ferro, L. 2004. A Probabilistic Rasch Analysis of Question Answering Evaluations. HLT-NAACL 2004: 65-72.
6. Linacre, J. M. 1989. Many-facet Rasch measurement. Chicago: MESA Press.
7. Linacre, J.M. 1997. Judging plans and facets. Mesa Research Note #3. (consulted on September 4, 2011, http://www.rasch.org/rn3.htm)
8. Linacre, J. M. 2011 Facets computer program for many-facet Rasch measurement, version 3.68.1. Beaverton, Oregon: Winsteps.com.
9. Pesudovs, K., Garamendi, E., Keeves, J.P., and Elliot, D.B. 2003. ADVS for Cataract Surgery Outcomes. Investigative Ophthalmology & Visual Science, 44, 2892-2899.
10. Rasch, G. 1960/1980. Probabilistic models for some intelligence and attainment tests.(Copenhagen, Danish Institute for Educational Research), expanded edition (1980) with foreword and afterword by B.D. Wright. Chicago: The University of Chicago Press.
11. Sorokin, A., and Forsyth, D. 2008: Utility data annotation with amazon mechanical Turk. First IEEE Workshop on Internet Vision at CVPR 08, Anchorage, Alaska.
12. van der Linden, W. J. 2000. Constrained adaptive testing with shadow tests. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 27-52). Boston: Kluwer.
13. Wright, B. D., & Masters, G. N. 1982. Rating scale analysis. Chicago: MESA Press.