Understandability biased evaluation for information retrieval

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 9626, Issue , 2016, Pages 280-292

  Zuccon, Guido ^a  

^a QUEENSLAND UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS;

EVALUATION FRAMEWORK; EVALUATION MEASURES; MULTIPLE DIMENSIONS; RETRIEVAL SYSTEMS; RETRIEVED DOCUMENTS; SIMULATED EXPERIMENTS; SYSTEM EFFECTIVENESS; UNDERSTANDABILITY;

SEARCH ENGINES;

EID: 84962596553     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-319-30671-1_21     Document Type: Conference Paper

References (35)

1. Ahmed, O.H., Sullivan, S.J., Schneiders, A.G., McCrory, P.R.: Concussion information online: evaluation of information quality, content and readability of concussionrelated websites. Br. J. Sports Med. 46(9), 675–683 (2012)
2. Barry, C.L.: User-defined relevance criteria: an exploratory study. JASIS 45(3), 149–159 (1994)
3. Bruza, P.D., Zuccon, G., Sitbon, L.: Modelling the information seeking user by the decision they make. In: Proceedings of MUBE, pp. 5–6 (2013)
4. Carterette, B.: System effectiveness, user models, and user utility: a conceptualframework for investigation. In: Proceedings of SIGIR, pp. 903–912 (2011)
5. Clarke, C.L., Craswell, N., Soboroff, I., Ashkan, A.: A comparative analysis of cascade measures for novelty and diversity. In: Proceedings of WSDM, pp. 75–84 (2011)
6. Clarke, C.L., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I.: Novelty and diversity in information retrieval evaluation. In: Proceedings of SIGIR, pp. 659–666 (2008)
7. Collins-Thompson, K., Callan, J.: Predicting reading difficulty with statistical language models. JASIST 56(13), 1448–1462 (2005)
8. Cosijn, E., Ingwersen, P.: Dimensions of relevance. IP&M 36(4), 533–550 (2000)
9. Cuadra, C.A., Katter, R.V.: Opening the black box of ‘relevance’. J. Doc. 23(4), 291–303 (1967)
10. Eisenberg, M., Barry, C.: Order effects: a study of the possible influence of presentation order on user judgments of document relevance. JASIS 39(5), 293–300 (1988)
11. Friedman, D.B., Hoffman-Goetz, L., Arocha, J.F.: Health literacy and the world wide web: comparing the readability of leading incident cancers on the internet. Inf. Health Soc. Care 31(1), 67–87 (2006)
12. Goeuriot, L., Jones, G., Kelly, L., Leveling, J., Hanbury, A., Müller, H., Salanterä, S., Suominen, H., Zuccon, G.: ShARe/CLEF eHealth Evaluation Lab 2013, Task 3: Informationretrieval to address patients’ questions when reading clinical reports. In: Proceedings of CLEF (2013)
13. Goeuriot, L., Kelly, L., Lee, W., Palotti, J., Pecina, P., Zuccon, G., Hanbury, A., Gareth, H.M., Jones, J.F.: ShARe, CLEF eHealth Evaluation Lab 2014, Task 3: User-centred health information retrieval. In: Proceedings of CLEF Sheffield, UK (2014)
14. Larsson, P.: Classification into readability levels: implementation andevaluation. PhD thesis, Uppsala University (2006)
15. McCallum, D.R., Peterson, J.L.: Computer-based readability indexes. In: Proceedings of the ACM Conference, pp. 44–48 (1982)
16. Mizzaro, S.: Relevance: the whole history. JASIS 48(9), 810–832 (1997)
17. Moffat, A., Zobel, J.: Rank-biased precision for measurement of retrieval effectiveness. TOIS 27(1), 2 (2008)
18. Palotti, J., Zuccon, G., Goeuriot, L., Kelly, L., Hanbury, A., Jones, G.J., Lupu, M., Pecina, P.: Clef eHealth evaluation lab 2015, task 2: Retrieving informationabout medical symptoms. In: Proceedings of CLEF (2015)
19. Palotti, J., Zuccon, G., Hanbury, A.: The influence of pre-processing on the estimation of readability of web documents. In: Proceedings of CIKM (2015)
20. Rees, A.M., Schultz, D.G.: A field experimental approach to the study of relevance assessments in relation to document searching. Technical report, Case Western Reserve University (1967)
21. Robertson, S.E.: The probability ranking principle in IR. J. Doc. 33(4), 294–304 (1977)
22. Sakai, T., Song, R.: Evaluating diversified search results using per-intent graded relevance. In: Proceedings of SIGIR, pp. 1043–1052 (2011)
23. Saracevic, T.: The stratified model of information retrieval interaction: extension and applications. Proceedings of ASIS, vol. 34, pp. 313–327 (1997)
24. Schamber, L., Eisenberg, M.: Relevance: the search for a definition. In: Proceedings of ASIS (1988)
25. Smucker, M.D., Clarke, C.L.: Time-based calibration of effectiveness measures. In: Proceedings of SIGIR, pp. 95–104 (2012)
26. Walsh, T.M., Volsko, T.A.: Readability assessment of internet-based consumer health information. Respir. Care 53(10), 1310–1315 (2008)
27. Wiener, R.C., Wiener-Pla, R.: Literacy, pregnancy and potential oral health changes: the internetand readability levels. Matern. Child Health J. 1–6 (2013)
28. Xu, Y.C., Chen, Z.: Relevance judgment: what do information users consider beyond topicality? JASIST 57(7), 961–973 (2006)
29. Yan, X., Song, D., Li, X.: Concept-based document readability in domain specific information retrieval. In: Proceedings of CIKM, pp. 540–549 (2006)
30. Yilmaz, E., Aslam, J.A., Robertson, S.: A new rank correlation coefficient for information retrieval. In: Proceedings of SIGIR, pp. 587–594 (2008)
31. Zhai, C.X., Cohen, W.W., Lafferty, J.: Beyond independent relevance: methods and evaluation metrics forsubtopic retrieval. In: Proceedings of SIGIR, pp. 10–17 (2003)
32. Zhang, Y., Park, L.A., Moffat, A.: Click-based evidence for decaying weight distributions in search effectiveness metrics. Inf. Retrieval 13(1), 46–69 (2010)
33. Zhang, Y., Zhang, J., Lease, M., Gwizdka, J.: Multidimensional relevance modeling via psychometrics and crowdsourcing. In: Proceedings of SIGIR, pp. 435–444 (2014)
34. Zuccon, G., Koopman, B.: Integrating understandability in the evaluation of consumer health search engines. Proceedings of MedIR, pp. 32–35 (2014)
35. Zuccon, G., Koopman, B., Palotti, J.: Diagnose this if you can. In: Hanbury, A., Kazai, G., Rauber, A., Fuhr, N. (eds.) ECIR 2015. LNCS, vol. 9022, pp. 562–567. Springer, Heidelberg (2015)