Audio music similarity and retrieval: Evaluation power and stability

Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011

Volumn , Issue , 2011, Pages 597-602

  Urbano, Julián ^a   Martín, Diego ^a   Marrero, Mónica ^a   Morato, Jorge ^a  

^a UNIVERSIDAD CARLOS III DE MADRID   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

AUDIO MUSIC; EFFECTIVENESS MEASURE; RELEVANCE JUDGMENT; RELIABILITY LEVEL; RETRIEVAL SYSTEMS; SIGNIFICANT DIFFERENCES;

INFORMATION RETRIEVAL; STATISTICAL METHODS;

RELIABILITY;

EID: 84861089565     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (17)

1. C. Buckley and E.M. Voorhees, "Evaluating Evaluation Measure Stability, " ACM SIGIR, pp. 33-34, 2000.
2. J.S. Downie, "MIREX Next Generation, " music-ir email list, 2011. Available at: http://listes.ircam.fr/wws/info/music-ir.
3. J.S. Downie, A.F. Ehmann, M. Bay, and M.C. Jones, "The Music Information Retrieval Evaluation eXchange: Some Observations and Insights, " Advances in Music Information Retrieval, W.R. Zbigniew and A.A. Wieczorkowska, (eds.), Springer, pp. 93-115, 2010.
4. IMIRSEL, "MIREX 2009 Audio Music Similarity and Retrieval Results, " http://musicir.org/mirex/wiki/2009:Audio-Music-Similarity-and-Retrieval- Results.
5. K. Järvelin and J. Kekäläinen, "Cumulated Gain-Based Evaluation of IR Techniques, " ACM Transactions on Information Systems, 20:4, pp. 422-446, 2002.
6. J.H. Lee, "Crowdsourcing Music Similarity Judgments using Mechanical Turk, " ISMIR, pp. 183-188, 2010.
7. T. Sakai, "On the Reliability of Information Retrieval Metrics Based on Graded Relevance, " Information Processing and Management, 43:2, pp. 531-548, 2007.
8. M. Sanderson and J. Zobel, "Information Retrieval System Evaluation: Effort, Sensitivity, and Reliability, " ACM SIGIR, pp. 162-169, 2005.
9. M.A. Seaman, J.R. Levin, and R.C. Serlin, "New Developments in Pairwise Multiple Comparisons: Some Powerful and Practicable Procedures, " Psychological Bulletin, 110:3, pp. 577-586, 1991.
10. R. Typke, M. den Hoed, J. de Nooijer, F. Wiering, and R.C. Veltkamp, "A Ground Truth for Half a Million Musical Incipits, " Journal of Digital Information Management, vol. 3, no. 1, pp. 34-39, 2005.
11. R. Typke, R.C. Veltkamp, and F. Wiering, "A Measure for Evaluating Retrieval Techniques based on Partially Ordered Ground Truth Lists, " IEEE International Conference on Multimedia and Expo, pp. 1793-1796, 2006.
12. J. Urbano, "Information Retrieval Meta-Evaluation: Challenges and Opportunities in the Music Domain, " ISMIR, 2011.
13. J. Urbano, M. Marrero, D. Martín, and J. Lloréns, "Improving the Generation of Ground Truths based on Partially Ordered Lists, " ISMIR, pp. 285-290, 2010.
14. J. Urbano, J. Morato, M. Marrero, and D. Martín, "Crowdsourcing Preference Judgments for Evaluation of Music Similarity Tasks, " ACM SIGIR Workshop on Crowdsourcing for Search Evaluation, pp. 9-16, 2010.
15. E.M. Voorhees, "Topic Set Size Redux, " ACM SIGIR, pp. 806-807, 2009.
16. E.M. Voorhees and C. Buckley, "The Effect of Topic Set Size on Retrieval Experiment Error, " ACM SIGIR, pp. 316-323, 2002.
17. W. Webber, A. Moffat, J. Zobel, and T. Sakai, "Precision- At-Ten Considered Redundant, " ACM SIGIR, pp. 695-696, 2008.