Pay by the bit: An information-theoretic metric for collective human judgment

Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW

Volumn , Issue , 2013, Pages 623-637

  Waterhouse, Tamsyn P ^a  

^a GOOGLE INC   (United States)

Author keywords

Crowdsourcing; Human computation; Information theory

Indexed keywords

COLLECTION PROCESS; CONDITIONAL MUTUAL INFORMATION; CONDITIONAL PROBABILITIES; CROWDSOURCING; HUMAN COMPUTATION; INFORMATION MEASURES; MUTUAL INFORMATIONS; POINTWISE MUTUAL INFORMATION;

COMPUTER SUPPORTED COOPERATIVE WORK; INFORMATION THEORY; INTERACTIVE COMPUTER SYSTEMS;

MULTIVARIABLE SYSTEMS;

EID: 84874866677     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2441776.2441846     Document Type: Conference Paper

References (34)

1. Beal, M. J. Variational algorithms for approximate Bayesian inference. PhD thesis, Gatsby Computational Neuroscience Unit, University College London, 2003.
2. Card, S. K., Moran, T. P., and Newell, A. The model human processor: An engineering model of human performance. In Handbook of Perception and Human Performance, K. R. Boff, L. Kaufman, and J. P. Thomas, Eds., vol. 2: Cognitive Processes and Performance. Wiley-Interscience, 1986, ch. 45, 1-35.
3. Card, S. W., and Mohan, C. K. Ensemble selection for evolutionary learning using information theory and price's theorem. In Proc. GECCO 2006, ACM Press (2006), 1587-1588.
4. Chen, Y., Dimitrov, S., Sami, R., Reeves, D. M., Pennock, D. M., Hanson, R. D., Fortnow, L., and Gonen, R. Gaming prediction markets: Equilibrium strategies with a market maker. Algorithmica 58, 4 (2010), 930-969.
5. Chi, E. H., and Mytkowicz, T. Understanding the efficiency of social tagging systems using information theory. In Proc. HT 2008, ACM Press (2008), 81-88.
6. Dawid, A. P., and Skene, A. M. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics (1979), 20-28.
7. Dempster, A. P., Laird, N. M., and Rubin, D. B. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39, 1 (1977), 1-38.
8. Dow, S., Kulkarni, A., Bunge, B., Nguyen, T., Klemmer, S., and Hartmann, B. Shepherding the crowd: managing and providing feedback to crowd workers. In Ext. Abstracts CHI 2011, ACM Press (2011), 1669-1674.
9. Gelfand, A. E., and Smith, A. F. M. Sampling-based approaches to calculating marginal densities. Journal of the American Statistical Association 85, 410 (June 1990), 398-409.
10. Geman, S., and Geman, D. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6 (1984), 721-741.
11. Hanson, R. Logarithmic market scoring rules for modular combinatorial information aggregation. The Journal of Prediction Markets 1, 1 (2007), 3-15.
12. Hartley, H. O. Maximum likelihood estimation from incomplete data. Biometrics 14, 2 (June 1958), 174-194.
13. Hastings, W. K. Monte carlo sampling methods using markov chains and their applications. Biometrika 57, 1 (1970), 97-109.
14. Hick, W. E. On the rate of gain of information. Quarterly Journal of Experimental Psychology 4 (1952), 11-26.
15. Ipeirotis, P. G., Provost, F., and Wang, J. Quality management on Amazon Mechanical Turk. In Proc. HCOMP 2010, ACM Press (2010), 64-67.
16. Kochhar, S., Mazzocchi, S., and Paritosh, P. The anatomy of a large-scale human computation engine. In Proc. HCOMP 2010, ACM Press (2010), 10-17.
17. Krippendorff, K. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement 30, 1 (1970), 61-70.
18. Lazo, A. V., and Rathie, P. On the entropy of continuous probability distributions. IEEE Transactions on Information Theory 24, 1 (1978), 120-122.
19. Lintott, C. J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., Raddick, M. J., Nichol, R. C., Szalay, A., Andreescu, D., Murray, P., and van den Berg, J. Galaxy Zoo: Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society 389, 3 (2008), 1179-1189.
20. Paritosh, P. Human computation must be reproducible. In Proc. First International Workshop on Crowdsourcing Web Search (2012), 20-25.
21. Quinn, A. J., and Bederson, B. B. Human computation: a survey and taxonomy of a growing field. In Proc. CHI 2011, ACM Press (2011), 1403-1412.
22. Shannon, C. E. A mathematical theory of communication. The Bell System Technical Journal 27, 3 (1948), 379-423. Part 1 of 2.
23. Shannon, C. E., and Weaver, W. The Mathematical Theory of Communication. University of Illinois Press, 1949.
24. Shaw, A. D., Horton, J. J., and Chen, D. L. Designing incentives for inexpert human raters. In Proc. CSCW 2011, ACM Press (2011), 275-284.
25. Sheng, V. S., Provost, F., and Ipeirotis, P. G. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proc. KDD 2008, ACM Press (2008), 614-622.
26. Simpson, E., Roberts, S., Psorakis, I., and Smith, A. Dynamic Bayesian combination of multiple imperfect classifiers. ArXiv e-prints (June 2012).
27. Tanner, M. A., and Wong, W. H. The calculation of posterior distributions by data augmentation. Journal of the American Statistical Association 82, 398 (June 1987), 528-540.
28. Walsh, B. Markov chain Monte Carlo and Gibbs sampling, 2004. Lecture Notes for EEB 581.
29. Welinder, P., Branson, S., Belongie, S., and Perona, P. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems, vol. 23, NIPS (2010), 2424-2432.
30. Welinder, P., and Perona, P. Online crowdsourcing: rating annotators and obtaining cost-effective labels. In Proc. CVPR 2010, IEEE (2010), 25-32.
31. Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., and Movellan, J. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems 22 (2009), 2035-2043.
32. Willett, K. W., Lintott, C. J., et al. In preparation.
33. Yan, Y., Rosales, R., Fung, G., and Dy, J. Active learning from crowds. In Proc. ICML 2011, IMLS (2011), 1161-1168.
34. Yang-Peláez, J., and Flowers, W. C. Information content measures of visual displays. In Proc. InfoVis 2000, IEEE (2000), 99-103.