CROWDMOS: An approach for crowdsourcing mean opinion score studies

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2011, Pages 2416-2419

  Ribeiro, Flávio ^a   Florêncio, Dinei ^b   Zhang, Cha ^b   Seltzer, Michael ^b  

^a UNIVERSITY OF SÃO PAULO   (Brazil)

^b MICROSOFT RESEARCH   (United States)

Author keywords

crowdMOS; crowdsourcing; mean opinion score; mechanical turk; MOS; MUSHRA; subjective quality

Indexed keywords

CROWDMOS; CROWDSOURCING; MEAN OPINION SCORES; MECHANICAL TURKS; MOS; MUSHRA; SUBJECTIVE QUALITY;

PROCESSING; SIGNAL PROCESSING; SPEECH COMMUNICATION; STORMS; SUBJECTIVE TESTING;

QUALITY CONTROL;

EID: 80051607565     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2011.5946971     Document Type: Conference Paper

References (13)

1. "Methods for subjective determination of transmission quality," ITU-T Recommendation P.800, Aug. 1996.
2. "Methods for the subjective assessment of small impairments in audio systems including multichannel sound systems," ITU-R Recommendation BS.1116-1, Oct. 1997.
3. "Method for the subjective assessment of intermediate quality level of coding systems," ITU-R Recommendation BS.1534-1, Jan. 2001.
4. "Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs," ITU-T Recommendation P.862, Feb. 2001.
5. S. Pennock, "Accuracy of the perceptual evaluation of speech quality (PESQ) algorithm," Proc. Of MESAQIN, 2002.
6. Z. Qiao, L. Sun, and E. Ifeachor, "Case study of PESQ performance in live wireless mobile VoIP environment," in Proc. of PIMRC, 2008.
7. A. Kittur, E.H. Chi, and B. Suh, "Crowdsourcing user studies with Mechanical Turk," in Proc. ACM CHI 2008. ACM, 2008, pp. 453-456.
8. K.T. Chen, C.C. Wu, Y.C. Chang, and C.L. Lei, "A crowdsourceable QoE evaluation framework for multimedia content," in Proc. ACMMultimedia 2009. ACM, 2009, pp. 491-500.
9. F. Ribeiro, D. Florencio, and V. Nascimento, "Crowdsourcing Subjective Image Quality Evaluation," submitted.
10. F. Ribeiro and D. Florencio, "Region of Interest Determination Using Human Computation," submitted.
11. S. King and V. Karaiskos, "The Blizzard Challenge 2009," in Proc. of the Blizzard Challenge, 2009.
12. F. Ribeiro, D. Florencio, C. Zhang, and M. Seltzer, "crowdMOS Standalone Tools," available at http://research.microsoft.com/crowdmos/.
13. B. Efron, R. Tibshirani, and R.J. Tibshirani, An introduction to the bootstrap, Chapman & Hall/CRC, 1993.