How to grade a test without knowing the answers - A Bayesian graphical model for adaptive crowdsourcing and aptitude testing

Proceedings of the 29th International Conference on Machine Learning, ICML 2012

Volumn 2, Issue , 2012, Pages 1183-1190

  Bachrach, Yoram ^a   Minka, Tom ^a   Guiver, John ^a   Graepel, Thore ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ACTIVE LEARNING; ADAPTIVE TESTING; BAYESIAN GRAPHICAL MODELS; CROWDSOURCING; EFFICIENT RESOURCE ALLOCATION; PROBABILISTIC GRAPHICAL MODELS; STATIC TESTS;

ABILITY TESTING; LEARNING SYSTEMS;

STATISTICAL TESTS;

EID: 84867117736     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. Anastasi, A., Urbina, S., et al. Psychological testing. Prentice Hall Upper Saddle River, NJ, 1997.
2. Bachrach, Y., Graepel, T., Kasneci, G., Kosinski, M., and Van Gael, J. Crowd IQ - aggregating opinions to boost performance. In AAMAS, 2012.
3. de Caritat, M.J.A.N. et al. Essai sur l'application de l'analyse à la probabilité des décisions rendues à la pluralité des voix. L'imprimerie royale, 1785.
4. DiPalantino, D. and Vojnovic, M. Crowdsourcing and allpay auctions. In ACM EC, 2009.
5. Gao, X.A., Bachrach, Y., Key, P., and Graepel, T. Quality expectation-variance tradeoffs in crowdsourcing contests. AAAI, 2012.
6. Getoor, L., Friedman, N., Koller, D., Pfeffer, A., and Taskar, B. 5 probabilistic relational models. Statistical relational learning, pp. 129, 2007.
7. Hambleton, R.K., Swaminathan, H., and Rogers, H.J. Fundamentals of item response theory, volume 2. 1991.
8. Herbrich, R., Minka, T., and Graepel, T. Trueskill: A bayesian skill rating system. NIPS, 2007.
9. Kasneci, G., Van Gael, J., Herbrich, R., and Graepel, T. Bayesian knowledge corroboration with logical rules and user feedback. In ECML/PKDD, 2010.
10. Koller, D. and Friedman, N. Probabilistic Graphical Models: Principles and Techniques. 2009.
11. Kosinski, M., Bachrach, Y., Kasneci, G., Van-Gael, J., and Graepel, T. Crowd IQ: Measuring the intelligence of crowdsourcing platforms. In ACM Web Sciences, 2012.
12. Lease, M. and Kazai, G. Overview of the trec 2011 crowd-sourcing track (conference notebook). In TREC, 2011.
13. Lyle, J.A. Collective problem solving: Are the many smarter than the few? 2008.
14. MacKay, David J.C. Information-based objective functions for active data selection. Neural Computation, 1992.
15. Minka, T., Winn, J.M., Guiver, J.P., and Knowles, D.A. Infer.NET 2.4, 2010.
16. Minka, Tom and Winn, John. Gates. NIPS, 21, 2008.
17. Minka, T.P. A family of algorithms for approximate Bayesian inference. PhD thesis, 2001.
18. Pearl, J. Probabilistic reasoning in intelligent systems : networks of plausible inference. 1988.
19. Pennock, D.M. and Sami, R. Computational aspects of prediction markets, 2007.
20. Raven, J.C. Standard progressive matrices plus.
21. Raykar, V.C., Yu, S., Zhao, L.H., Valadez, G.H., Florin, C., Bogoni, L., and Moy, L. Learning from crowds. JMLR, 2010.
22. Sen, A. Social choice theory. Handbook of mathematical economics, 3:1073-1181, 1986.
23. Welinder, P., Branson, S., Belongie, S., and Perona, P. The multidimensional wisdom of crowds. In NIPS, 2010.
24. Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., and Movellan, J. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. NIPS, 22:2035-2043, 2009.
25. Woolley, A.W., Chabris, C.F., Pentland, A., Hashmi, N., and Malone, T.W. Evidence for a collective intelligence factor in the performance of human groups. Science, 330 (6004):686, 2010.
26. Yan, Y., Rosales, R., Fung, G., and Dy, J. Active learning from crowds. In ICML, 2011.