Trick or treat: Putting peer prediction to the test

EC 2014 - Proceedings of the 15th ACM Conference on Economics and Computation

Volumn , Issue , 2014, Pages 507-524

  Gao, Xi Alice ^a   Mao, Andrew ^a   Chen, Yiling ^a   Adams, Ryan Prescott ^a  

^a HARVARD UNIVERSITY   (United States)

Author keywords

hidden markov models; online behavioral experiment; peer prediction

Indexed keywords

HIDDEN MARKOV MODELS;

BEHAVIORAL EXPERIMENT; MULTIPLAYERS; ON-LINE EXPERIMENTS; PREDICTION MECHANISMS; REPEATED GAMES; STRATEGIC BEHAVIOR; SUBJECTIVE INFORMATION;

FORECASTING;

EID: 84903137143     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2600057.2602865     Document Type: Conference Paper

References (25)

1. Asim Ansari, Ricardo Montoya, and Oded Netzer. 2012. Dynamic learning in behavioral games: A hidden Markov mixture of experts approach. Quantitative Marketing and Economics 10, 4 (2012), 475-503.
2. Jesse Chandler, Pam Mueller, and Gabriele Paolacci. 2013. Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods (2013). DOI:http://dx.doi.org/10.3758/s13428-013-0365-7
3. Anirban Dasgupta and Arpita Ghosh. 2013. Crowdsourced Judgement Elicitation with Endogenous Proficiency. ACM International World Wide Web Conference (WWW) (2013).
4. Xi Alice Gao, Yoram Bachrach, Peter Key, and Thore Graepel. 2012. Quality Expectation-Variance Tradeoffs in Crowdsourcing Contests. In AAAI.
5. John J Horton, David G Rand, and Richard J Zeckhauser. 2011. The online laboratory: conducting experiments in a real labor market. Experimental Economics 14, 3 (Feb. 2011), 399-425.
6. L K John, G Loewenstein, and Drazen Prelec. 2012. Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science 23, 5 (May 2012), 524-532.
7. R Jurca and B Faltings. 2009. Mechanisms for making crowds truthful. Journal of Artificial Intelligence Research 34, 1 (2009), 209-253.
8. Andrew Mao, Yiling Chen, Krzysztof Z. Gajos, David Parkes, Ariel D. Procaccia, and Haoqi Zhang. 2012. TurkServer: Enabling Synchronous and Longitudinal Online Experiments. In Proceedings of the 4th Workshop on Human Computation (HCOMP'12).
9. W. Mason and S. Suri. 2012. Conducting Behavioral Research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (2012), 1-23.
10. Nolan Miller, Paul Resnick, and Richard Zeckhauser. 2005. Eliciting Informative Feedback: The Peer-Prediction Method. MANAGEMENT SCIENCE 51, 9 (1 Sept. 2005), 1359-1373. DOI:http://dx.doi.org/10.1287/mnsc.1050.0379
11. Charles Noussair and Marc Willinger. 2011. Mixed strategies in an unprofitable game: an experiment. Technical Report.
12. Drazen Prelec. 2004. A Bayesian truth serum for subjective data. Science 306, 5695 (Oct. 2004), 462-466.
13. Drazen Prelec and HS Seung. 2006. An algorithm that finds truth even if most people are wrong. (2006).
14. Lawrence R Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 2 (1989), 257-286.
15. Goran Radanovic and Boi Faltings. 2013. A Robust Bayesian Truth Serum for Nonbinary Signals. In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI'13). 833-839.
16. David G Rand. 2012. The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology 299, C (April 2012), 172-179.
17. Gideon Schwarz. 1978. Estimating the Dimension of a Model. The Annals of Statistics 6, 2 (1978), 461-464. DOI:http://dx.doi.org/10.2307/2958889
18. Reinhard Selten and Thorsten Chmura. 2008. Stationary concepts for experimental 2x2-games. The American Economic Review 98, 3 (2008), 938-966.
19. Jason Shachat, J Swarthout, and Lijia Wei. 2012. A hidden Markov model for the detection of pure and mixed strategy play in games. Available at SSRN (2012).
20. AD Shaw, JJ Horton, and DL Chen. 2011. Designing incentives for inexpert human raters. Proceedings of the ACM 2011 conference on Computer supported cooperative work (2011), 275-284.
21. Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. Information Theory, IEEE Transactions on 13, 2 (1967), 260-269.
22. Bo Waggoner and Yiling Chen. 2013. Information Elicitation Sans Verification. In Proceedings of the 3rd Workshop on Social Computing and User Generated Content (SC13).
23. Jens Witkowski, Yoram Bachrach, Peter Key, and David C. Parkes. 2013. Dwelling on the Negative: Incentivizing Effort in Peer Prediction. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP'13).
24. Jens Witkowski and David C. Parkes. 2012a. A Robust Bayesian Truth Serum for Small Populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI'12).
25. Jens Witkowski and David C Parkes. 2012b. Peer Prediction without a Common Prior. EC 2012 (April 2012), 1-18.