메뉴 건너뛰기




Volumn 46, Issue 4, 2016, Pages 543-576

Learning from crowdsourced labeled data: a survey

Author keywords

Crowdsourcing; Ground truth inference; Label quality; Learning from crowds; Learning model quality; Multiple noisy labeling

Indexed keywords

CROWDSOURCING; INFERENCE ENGINES; OPEN SYSTEMS; VIRTUAL REALITY;

EID: 84976512003     PISSN: 02692821     EISSN: 15737462     Source Type: Journal    
DOI: 10.1007/s10462-016-9491-9     Document Type: Article
Times cited : (195)

References (132)
  • 2
    • 67349095304 scopus 로고    scopus 로고
    • Handbook of numerical analysis
    • Bernardi C, Maday Y (1997) Handbook of numerical analysis. Spectr Methods 5:209–485
    • (1997) Spectr Methods , vol.5 , pp. 209-485
    • Bernardi, C.1    Maday, Y.2
  • 4
    • 84993167633 scopus 로고    scopus 로고
    • Bragg J, Weld DS, et al (2013) Crowdsourcing multi-label classification for taxonomy creation. In: First AAAI conference on human computation and crowdsourcing, pp 25–33
  • 5
    • 0035478854 scopus 로고    scopus 로고
    • Random forests
    • Breiman L (2001) Random forests. Mach Learn 45(1):5–32
    • (2001) Mach Learn , vol.45 , Issue.1 , pp. 5-32
    • Breiman, L.1
  • 7
    • 0000046054 scopus 로고    scopus 로고
    • Identifying mislabeled training data
    • Brodley CE, Friedl MA (1999) Identifying mislabeled training data. J Artif Intell Res 11:131–167
    • (1999) J Artif Intell Res , vol.11 , pp. 131-167
    • Brodley, C.E.1    Friedl, M.A.2
  • 8
    • 84993151982 scopus 로고    scopus 로고
    • Buckley C, Lease M, Smucker MD, Jung HJ, Grady C, Buckley C, Lease M, Smucker MD, Grady C, Lease M, et al (2010) Overview of the trec 2010 relevance feedback track (notebook). In: The nineteenth text retrieval conference (TREC) notebook
  • 11
    • 33745838229 scopus 로고    scopus 로고
    • The pascal recognising textual entailment challenge
    • Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, Springer
    • Dagan I, Glickman O, Magnini B (2006) The pascal recognising textual entailment challenge. In: Quiñonero-Candela J, Dagan I, Magnini B, d’Alché-Buc F (eds) Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, Springer, pp 177–190
    • (2006) Quiñonero-Candela J , pp. 177-190
    • Dagan, I.1    Glickman, O.2    Magnini, B.3    Dagan, I.4    Magnini, B.5    d’Alché-Buc, F.6
  • 13
    • 0003102944 scopus 로고
    • Maximum likelihood estimation of observer error-rates using the em algorithm
    • Dawid AP, Skene AM (1979) Maximum likelihood estimation of observer error-rates using the em algorithm. Appl Stat 28(1):20–28
    • (1979) Appl Stat , vol.28 , Issue.1 , pp. 20-28
    • Dawid, A.P.1    Skene, A.M.2
  • 15
    • 79953645302 scopus 로고    scopus 로고
    • Crowdsourcing systems on the world-wide web
    • Doan A, Ramakrishnan R, Halevy AY (2011) Crowdsourcing systems on the world-wide web. Commun ACM 54(4):86–96
    • (2011) Commun ACM , vol.54 , Issue.4 , pp. 86-96
    • Doan, A.1    Ramakrishnan, R.2    Halevy, A.Y.3
  • 20
    • 76649132306 scopus 로고    scopus 로고
    • Eagle N (2009) txteagle: mobile crowdsourcing. In: Aykin N (ed) Internationalization, design and global development, Springer, pp 447–456
    • Eagle N (2009) txteagle: mobile crowdsourcing. In: Aykin N (ed) Internationalization, design and global development, Springer, pp 447–456
  • 25
    • 84899651693 scopus 로고    scopus 로고
    • Classification in the presence of label noise: a survey
    • Frénay B, Verleysen M (2014) Classification in the presence of label noise: a survey. IEEE Trans Neural Netw Learn Syst 25(5):845–869
    • (2014) IEEE Trans Neural Netw Learn Syst , vol.25 , Issue.5 , pp. 845-869
    • Frénay, B.1    Verleysen, M.2
  • 26
    • 0035371148 scopus 로고    scopus 로고
    • An adaptive version of the boost by majority algorithm
    • Freund Y (2001) An adaptive version of the boost by majority algorithm. Mach Learn 43(3):293–318
    • (2001) Mach Learn , vol.43 , Issue.3 , pp. 293-318
    • Freund, Y.1
  • 27
    • 84876057388 scopus 로고    scopus 로고
    • A survey on instance selection for active learning
    • Fu Y, Zhu X, Li B (2013) A survey on instance selection for active learning. Knowl Inf Syst 35(2):249–283
    • (2013) Knowl Inf Syst , vol.35 , Issue.2 , pp. 249-283
    • Fu, Y.1    Zhu, X.2    Li, B.3
  • 32
    • 84929471718 scopus 로고    scopus 로고
    • Incremental learning for ν -support vector regression
    • Gu B, Sheng VS, Wang Z, Ho D, Osman S, Li S (2015) Incremental learning for ν -support vector regression. Neural Netw 67:140–150
    • (2015) Neural Netw , vol.67 , pp. 140-150
    • Gu, B.1    Sheng, V.S.2    Wang, Z.3    Ho, D.4    Osman, S.5    Li, S.6
  • 33
    • 70849126253 scopus 로고    scopus 로고
    • The unreasonable effectiveness of data
    • Halevy A, Norvig P, Pereira F (2009) The unreasonable effectiveness of data. IEEE Intell Syst 24(2):8–12
    • (2009) IEEE Intell Syst , vol.24 , Issue.2 , pp. 8-12
    • Halevy, A.1    Norvig, P.2    Pereira, F.3
  • 35
    • 84929208117 scopus 로고    scopus 로고
    • Demographic estimation from face images: human vs. machine performance
    • Han H, Otto C, Liu X, Jain A (2014) Demographic estimation from face images: human vs. machine performance. IEEE Trans Pattern Anal Mach Intell 37(6):1148–1161
    • (2014) IEEE Trans Pattern Anal Mach Intell , vol.37 , Issue.6 , pp. 1148-1161
    • Han, H.1    Otto, C.2    Liu, X.3    Jain, A.4
  • 36
    • 33847246935 scopus 로고    scopus 로고
    • The rise of crowdsourcing
    • Howe J (2006) The rise of crowdsourcing. Wired Mag 14(6):1–4
    • (2006) Wired Mag , vol.14 , Issue.6 , pp. 1-4
    • Howe, J.1
  • 40
    • 84867415955 scopus 로고    scopus 로고
    • Convex formulations of learning from crowds
    • Kajino H, Kashima H (2012) Convex formulations of learning from crowds. Trans Jan Soc Artif Intell 27:133–142
    • (2012) Trans Jan Soc Artif Intell , vol.27 , pp. 133-142
    • Kajino, H.1    Kashima, H.2
  • 42
    • 84896842331 scopus 로고    scopus 로고
    • Budget-optimal task allocation for reliable crowdsourcing systems
    • Karger DR, Oh S, Shah D (2014) Budget-optimal task allocation for reliable crowdsourcing systems. Oper Res 62(1):1–24
    • (2014) Oper Res , vol.62 , Issue.1 , pp. 1-24
    • Karger, D.R.1    Oh, S.2    Shah, D.3
  • 48
    • 84922928478 scopus 로고    scopus 로고
    • Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention
    • Kurve A, Miller DJ, Kesidis G (2015) Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention. IEEE Trans Knowl Data Eng 27(3):794–809
    • (2015) IEEE Trans Knowl Data Eng , vol.27 , Issue.3 , pp. 794-809
    • Kurve, A.1    Miller, D.J.2    Kesidis, G.3
  • 50
    • 84993118608 scopus 로고    scopus 로고
    • Li H, Yu B (2014) Error rate bounds and iterative weighted majority voting for crowdsourcing
    • Li H, Yu B (2014) Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv:1411.4086
  • 51
    • 84906932246 scopus 로고    scopus 로고
    • Towards a general rule for identifying deceptive opinion spam. In: Proceedings of the 52nd annual meeting of the association for computational linguistics
    • Li J, Ott M, Cardie C, Hovy E (2014) Towards a general rule for identifying deceptive opinion spam. In: Proceedings of the 52nd annual meeting of the association for computational linguistics, ACL
    • (2014) ACL
    • Li, J.1    Ott, M.2    Cardie, C.3    Hovy, E.4
  • 52
    • 84922280902 scopus 로고    scopus 로고
    • Segmentation-based image copy-move forgery detection scheme
    • Li J, Li X, Yang B, Sun X (2015) Segmentation-based image copy-move forgery detection scheme. IEEE Trans Inf Forensics Secur 10(3):507–518
    • (2015) IEEE Trans Inf Forensics Secur , vol.10 , Issue.3 , pp. 507-518
    • Li, J.1    Li, X.2    Yang, B.3    Sun, X.4
  • 56
    • 84957964666 scopus 로고    scopus 로고
    • Detecting multiple stochastic network motifs in network data
    • Liu K, Cheung WK, Liu J (2015) Detecting multiple stochastic network motifs in network data. Knowl Inf Syst 42(1):49–74
    • (2015) Knowl Inf Syst , vol.42 , Issue.1 , pp. 49-74
    • Liu, K.1    Cheung, W.K.2    Liu, J.3
  • 60
    • 84931059008 scopus 로고
    • Contextual correlates of semantic similarity
    • Miller GA, Charles WG (1991) Contextual correlates of semantic similarity. Lang Cognit Process 6(1):1–28
    • (1991) Lang Cognit Process , vol.6 , Issue.1 , pp. 1-28
    • Miller, G.A.1    Charles, W.G.2
  • 62
    • 84942293165 scopus 로고    scopus 로고
    • A unified statistical framework for crowd labeling
    • Muhammadi J, Rabiee HR, Hosseini A (2015) A unified statistical framework for crowd labeling. Knowl Inf Syst 45(2):271–294
    • (2015) Knowl Inf Syst , vol.45 , Issue.2 , pp. 271-294
    • Muhammadi, J.1    Rabiee, H.R.2    Hosseini, A.3
  • 67
    • 84939876669 scopus 로고    scopus 로고
    • Transfer learning for Bayesian discovery of multiple Bayesian networks
    • Oyen D, Lane T (2015) Transfer learning for Bayesian discovery of multiple Bayesian networks. Knowl Inf Syst 43(1):1–28
    • (2015) Knowl Inf Syst , vol.43 , Issue.1 , pp. 1-28
    • Oyen, D.1    Lane, T.2
  • 68
    • 77956031473 scopus 로고    scopus 로고
    • A survey on transfer learning
    • Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359
    • (2010) IEEE Trans Knowl Data Eng , vol.22 , Issue.10 , pp. 1345-1359
    • Pan, S.J.1    Yang, Q.2
  • 69
    • 0028741835 scopus 로고
    • Voting algorithms
    • Parhami B (1994) Voting algorithms. IEEE Trans Reliab 43(4):617–629
    • (1994) IEEE Trans Reliab , vol.43 , Issue.4 , pp. 617-629
    • Parhami, B.1
  • 71
    • 84942249246 scopus 로고    scopus 로고
    • Class imbalance revisited: a new experimental setup to assess the performance of treatment methods
    • Prati RC, Batista GEAPA, Silva DF (2015) Class imbalance revisited: a new experimental setup to assess the performance of treatment methods. Knowl Inf Syst 45(1):247–270
    • (2015) Knowl Inf Syst , vol.45 , Issue.1 , pp. 247-270
    • Prati, R.C.1    Batista, G.E.A.P.A.2    Silva, D.F.3
  • 75
    • 84857856268 scopus 로고    scopus 로고
    • Eliminating spammers and ranking annotators for crowdsourced labeling tasks
    • Raykar VC, Yu S (2012) Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J Mach Learn Res 13:491–518
    • (2012) J Mach Learn Res , vol.13 , pp. 491-518
    • Raykar, V.C.1    Yu, S.2
  • 78
    • 84878784643 scopus 로고    scopus 로고
    • Learning from multiple annotators: distinguishing good from random labelers
    • Rodrigues F, Pereira F, Ribeiro B (2013) Learning from multiple annotators: distinguishing good from random labelers. Pattern Recogn Lett 34(12):1428–1436
    • (2013) Pattern Recogn Lett , vol.34 , Issue.12 , pp. 1428-1436
    • Rodrigues, F.1    Pereira, F.2    Ribeiro, B.3
  • 81
    • 84870522605 scopus 로고    scopus 로고
    • Active learning literature survey
    • Settles B (2010) Active learning literature survey. Univ Wis Madison 52(55–66):11
    • (2010) Univ Wis Madison , vol.52 , Issue.55-66 , pp. 11
    • Settles, B.1
  • 83
    • 84965134289 scopus 로고    scopus 로고
    • Double or nothing: multiplicative incentive mechanisms for crowdsourcing
    • Shah NB, Zhou D (2015) Double or nothing: multiplicative incentive mechanisms for crowdsourcing. In: Advances in neural information processing systems, vol 28. pp 1–9
    • (2015) Advances in neural information processing systems , vol.28 , pp. 1-9
    • Shah, N.B.1    Zhou, D.2
  • 88
    • 65449137414 scopus 로고
    • Knowledge discovery in large image databases: dealing with uncertainties in ground truth
    • Smyth P, Burl MC, Fayyad UM, Perona P (1994) Knowledge discovery in large image databases: dealing with uncertainties in ground truth. In: KDD workshop, pp 109–120
    • (1994) KDD workshop
    • Smyth, P.1    Burl, M.C.2    Fayyad, U.M.3    Perona, P.4
  • 98
    • 0036565589 scopus 로고    scopus 로고
    • An instance-weighting method to induce cost-sensitive trees
    • Ting KM (2002) An instance-weighting method to induce cost-sensitive trees. IEEE Trans Knowl Data Eng 14(3):659–665
    • (2002) IEEE Trans Knowl Data Eng , vol.14 , Issue.3 , pp. 659-665
    • Ting, K.M.1
  • 101
    • 84904601689 scopus 로고    scopus 로고
    • Reliable crowdsourcing for multi-class labeling using coding theory
    • Vempaty A, Varshney LR, Varshney PK (2014) Reliable crowdsourcing for multi-class labeling using coding theory. IEEE J Sel Top Signal Process 8(4):667–679
    • (2014) IEEE J Sel Top Signal Process , vol.8 , Issue.4 , pp. 667-679
    • Vempaty, A.1    Varshney, L.R.2    Varshney, P.K.3
  • 104
    • 51749107030 scopus 로고    scopus 로고
    • recaptcha: Human-based character recognition via web security measures
    • Von Ahn L, Maurer B, McMillen C, Abraham D, Blum M (2008) recaptcha: Human-based character recognition via web security measures. Science 321(5895):1465–1468
    • (2008) Science , vol.321 , Issue.5895 , pp. 1465-1468
    • Von Ahn, L.1    Maurer, B.2    McMillen, C.3    Abraham, D.4    Blum, M.5
  • 105
    • 84873165828 scopus 로고    scopus 로고
    • Efficiently scaling up crowdsourced video annotation
    • Vondrick C, Patterson D, Ramanan D (2013) Efficiently scaling up crowdsourced video annotation. Int J Comput Vis 101(1):184–204
    • (2013) Int J Comput Vis , vol.101 , Issue.1 , pp. 184-204
    • Vondrick, C.1    Patterson, D.2    Ramanan, D.3
  • 106
    • 65749118363 scopus 로고    scopus 로고
    • Graphical models, exponential families, and variational inference
    • Wainwright MJ, Jordan MI (2008) Graphical models, exponential families, and variational inference. Found Trends Mach Learn 1(1–2):1–305
    • (2008) Found Trends Mach Learn , vol.1 , Issue.1-2 , pp. 1-305
    • Wainwright, M.J.1    Jordan, M.I.2
  • 107
    • 85006266573 scopus 로고    scopus 로고
    • Man vs. machine: practical adversarial detection of malicious crowdsourcing workers
    • USENIX Association, CA
    • Wang G, Wang T, Zheng H, Zhao BY (2014) Man vs. machine: practical adversarial detection of malicious crowdsourcing workers. In: 23rd USENIX security symposium, USENIX Association, CA
    • (2014) In: 23rd USENIX security symposium
    • Wang, G.1    Wang, T.2    Zheng, H.3    Zhao, B.Y.4
  • 110
    • 0002949285 scopus 로고    scopus 로고
    • The problem with noise and small disjuncts
    • Weiss GM, Hirsh H (1998) The problem with noise and small disjuncts. In: ICML, pp 574–578
    • (1998) ICML
    • Weiss, G.M.1    Hirsh, H.2
  • 111
    • 77956536129 scopus 로고    scopus 로고
    • Online crowdsourcing: rating annotators and obtaining cost-effective labels
    • IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW): IEEE
    • Welinder P, Perona P (2010) Online crowdsourcing: rating annotators and obtaining cost-effective labels. In: The 2010 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW), IEEE, pp 25–32
    • (2010) The , vol.2010 , pp. 25-32
    • Welinder, P.1    Perona, P.2
  • 113
    • 84961291654 scopus 로고    scopus 로고
    • A rapid learning algorithm for vehicle classification
    • Wen X, Shao L, Xue Y, Fang W (2015) A rapid learning algorithm for vehicle classification. Inf Sci 295:395–406
    • (2015) Inf Sci , vol.295 , pp. 395-406
    • Wen, X.1    Shao, L.2    Xue, Y.3    Fang, W.4
  • 115
  • 116
    • 84881234522 scopus 로고    scopus 로고
    • A probabilistic model of active learning with multiple noisy oracles
    • Wu W, Liu Y, Guo M, Wang C, Liu X (2013) A probabilistic model of active learning with multiple noisy oracles. Neurocomputing 118:253–262
    • (2013) Neurocomputing , vol.118 , pp. 253-262
    • Wu, W.1    Liu, Y.2    Guo, M.3    Wang, C.4    Liu, X.5
  • 122
    • 0003585730 scopus 로고    scopus 로고
    • An introduction to Bayesian inference in econometrics
    • Wiley, New York
    • Zellner A (1996) An introduction to Bayesian inference in econometrics. Wiley, New York
    • (1996)
    • Zellner, A.1
  • 123
    • 84882759297 scopus 로고    scopus 로고
    • Research on crowdsourcing quality control strategies and evaluation algorithm
    • Zhang Z, Pang J, Xie X (2013) Research on crowdsourcing quality control strategies and evaluation algorithm. Chin J Comput 8:1636–1649
    • (2013) Chin J Comput , vol.8 , pp. 1636-1649
    • Zhang, Z.1    Pang, J.2    Xie, X.3
  • 128
    • 85027958625 scopus 로고    scopus 로고
    • Active learning with imbalanced multiple noisy labeling
    • Zhang J, Wu X, Shengs VS (2015d) Active learning with imbalanced multiple noisy labeling. IEEE Trans Cybern 45(5):1081–1093
    • (2015) IEEE Trans Cybern , vol.45 , Issue.5 , pp. 1081-1093
    • Zhang, J.1    Wu, X.2    Shengs, V.S.3
  • 129
    • 84963731561 scopus 로고    scopus 로고
    • Multi-class ground truth inference in crowdsourcing with clustering
    • Zhang J, Sheng V, Wu J, Wu X (2016) Multi-class ground truth inference in crowdsourcing with clustering. IEEE Trans Knowl Data Eng 28(4):1080–1085
    • (2016) IEEE Trans Knowl Data Eng , vol.28 , Issue.4 , pp. 1080-1085
    • Zhang, J.1    Sheng, V.2    Wu, J.3    Wu, X.4


* 이 정보는 Elsevier사의 SCOPUS DB에서 KISTI가 분석하여 추출한 것입니다.