Near-optimally teaching the crowd to classify

31st International Conference on Machine Learning, ICML 2014

Volumn 2, Issue , 2014, Pages 1355-1378

  Singla, Adish ^a   Bogunovic, Ilija ^b   Bartók, Gábor ^a   Karbasi, Amin ^a   Krause, Andreas ^a  

^a ETH ZURICH   (Switzerland)

^b EPFL   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; DISTANCE EDUCATION; LEARNING ALGORITHMS; LEARNING SYSTEMS; STOCHASTIC SYSTEMS;

AMAZON MECHANICAL TURKS; CLASSIFICATION RULES; ERROR PROBABILITIES; EXPONENTIAL REDUCTION; LINEAR SEPARATORS; OBJECTIVE FUNCTIONS; ON-LINE EDUCATION; TEACHING ALGORITHMS;

STOCHASTIC MODELS;

EID: 84919930088     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. Anthony, Martin, Brightwell, Graham, Cohen, Dave, and Shawe-Taylor, John. On exact specification by examples. In Proc. of COLT, pp. 311-318. ACM, 1992.
2. Balbach, F. J. and Zeugmann, T. Recent developments in algorithmic teaching. In Proceedings of the 3rd International Conference on Language and Automata Theory and Applications, pp. 1-18, 2009.
3. Basu, S. and Christensen, J. Teaching classification boundaries to humans. In Proc. of AAAI, 2013.
4. Burnashev, M. V. and Zigangirov, K. S. An interval estimation problem for controlled observations. In Proceedings of Problems of Information Transmission, 1974.
5. Coursera. https://www.coursera.org/.
6. CUBAM. Caltech UCSD binary annotation model. https://github.com/welinder/cubam.
7. Dalvi, N., Dasgupta, A., Kumar, R., and Rastogi, V. Aggregating crowdsourced binary ratings. In Proc. of WWW, pp. 285-294, 2013.
8. Doliwa, T., Simon, H. U., and Zilles, S. Recursive teaching dimension, learning complexity, and maximum classes. In Proc. of ALT, pp. 209-223, 2010.
9. Dow, S., Gerber, E., and Wong, A. A pilot study of using crowds in the classroom. In Proceedings of Human Factors in Computing Systems (CHI), 2013.
10. Du, Jun and Ling, Charles X. Active teaching for inductive learners. In Proceedings of the Eleventh SIAM International Conference on Data Mining, pp. 851-861, 2011.
11. Goldman, S. A. and Kearns, M. J. On the complexity of teaching. Journal of Computer and System Sciences, 50: 303-314, 1992.
12. Gomes, R., Welinder, P., Krause, A., and Perona, P. Crowdclustering. In Proc. of NIPS, 2011.
13. Imagenet. http://www.image-net.org/.
14. Karp, Richard M. and Kleinberg, Robert. Noisy binary search and its applications. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, 2007.
15. Krause, Andreas and Golovin, Daniel. Submodular function maximization. In Tractability: Practical Approaches to Hard Problems (to appear). Cambridge University Press, February 2014.
16. Lindsey, Robert, Mozer, Michael, Huggins, William J, and Pashler, Harold. Optimizing instructional policies. In Proc. of NIPS, pp. 2778-2786, 2013.
17. Lintott, C. J., Schawinski, K., Slosar, A., Land, K., Bam- ford, S., Thomas, D., Raddick, M. J., Nichol, R. C., Szalay, A., Andreescu, D., Murray, P., and Vandenberg, J. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society, 389: 1179-1189, 2008.
18. Lo, Chi-Yuan, Matoušek, Jiří, and Steiger, William. Algorithms for ham-sandwich cuts. Discrete & Computational Geometry, 11(1):433-452, 1994.
19. MTurk. Mechanical Turk platform, https://www.mturk.com/.
20. Nemhauser, G.L., Wolsey, L.A., and Fisher, M. An analysis of the approximations for maximizing submodular set functions. Mathematical Programming, 14:265-294, 1978.
21. Nowak, Robert D. The geometry of generalized binary search. IEEE Transactions on Information Theory, 57 (12):7893-7906, 2011.
22. Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Y. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 254-263, 2008.
23. Sorokin, A. and Forsyth, D. Utility data annotation with amazon mechanical turk. In First IEEE Workshop on Internet Vision, 2008.
24. Sullivan, Brian L, Wood, Christopher L, Iliff, Marshall J, Bonney, Rick E, Fink, Daniel, and Kelling, Steve, ebird: A citizen-based bird observation network in the biological sciences. Biological Conservation, 142(10): 2282-2292, 2009.
25. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
26. Weld, D. S, Adar, E., Chilton, L., Hoffmann, R., and Horvitz, E. Personalized online education a crowdsourcing challenge. Workshop on Human Computation, 2012.
27. Welinder, P., Branson, S., Belongie, S., and Perona, P. The multidimensional wisdom of crowds. In Proc. of NIPS, 2010.
28. Zhu, Xiaojin. Machine teaching for bayesian learners in the exponential family. In Proc. of NIPS, pp. 1905-1913, 2013.
29. Zilles, S., Lange, S., Holte, R., and Zinkevich, M. Models of cooperative teaching and learning. Journal of Machine Learning Research, 12:349-384, 2011.