A methodology for evaluating aggregated search results

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 6611 LNCS, Issue , 2011, Pages 141-152

  Arguello, Jaime ^a   Diaz, Fernando ^b   Callan, Jamie ^a   Carterette, Ben ^c  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b YAHOO RESEARCH   (United States)

^c UNIVERSITY OF DELAWARE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER SCIENCE; COMPUTERS;

MECHANICAL TURKS; RELIABLE ASSESSMENT; SEARCH SERVICES; STATED PREFERENCES; USER STUDY; WEB SEARCH RESULTS;

INFORMATION RETRIEVAL;

EID: 85015342057     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-20161-5_15     Document Type: Conference Paper

References (15)

1. Arguello, J., Diaz, F., Callan, J., Crespo, J.-F.: Sources of evidence for vertical selection. In: SIGIR 2009, pp. 315–322. ACM, New York (2009)
2. Arguello, J., Diaz, F., Paiement, J.-F.: Vertical selection in the presence of unlabeled verticals. In: SIGIR 2010, pp. 691–698. ACM, New York (2010)
3. Carterette, B., Bennett, P.N., Chickering, D.M., Dumais, S.T.: Here or there: preference judgments for relevance. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 16–27. Springer, Heidelberg (2008)
4. Cohen, J.: A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37–46 (1960)
5. Diaz, F.: Integration of news content into web results. In: WSDM 2009, pp. 182–191. ACM, New York (2009)
6. Diaz, F., Arguello, J.: Adaptation of offline vertical selection predictions in the presence of user feedback. In: SIGIR 2009, pp. 323–330. ACM, New York (2009)
7. Fleiss, J.: Measuring nominal scale agreement among many raters. Psychological Bulletin. 76(5), 378–382 (1971)
8. Kumar, R., Vassilvitskii, S.: Generalized distances between rankings. In: WWW 2010, pp. 571–580. ACM, New York (2010)
9. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)
10. Li, X., Wang, Y.-Y., Acero, A.: Learning query intent from regularized click graphs. In: SIGIR 2008, pp. 339–346. ACM, New York (2008)
11. Sanderson, M., Paramita, M.L., Clough, P., Kanoulas, E.: Do user preferences and evaluation measures line up? In: SIGIR 2010, pp. 555–562. ACM, New York (2010)
12. Schulze, M.: A new monotonic, clone-independent, reversal symmetric, and condorcet-consistent single-winner election method. Social Choice and Welfare (July 2010)
13. Sushmita, S., Joho, H., Lalmas, M., Villa, R.: Factors affecting click-through behavior in aggregated search interfaces. In: CIKM 2010, pp. 519–528. ACM, New York (2010)
14. Thomas, P., Hawking, D.: Evaluation by comparing result sets in context. In: CIKM 2006, pp. 94–101. ACM, New York (2006)
15. Zhu, D., Carterette, B.: An analysis of assessor behavior in crowdsourced preference judgements. In: SIGIR Workshop on Crowdsourcing for Search Evaluation, pp. 21– 26. ACM, New York (2010)