The IR Task at the CLEF ehealth evaluation lab 2016: User-centred health information retrieval

CEUR Workshop Proceedings

Volumn 1609, Issue , 2016, Pages 15-27

  Zuccon, Guido ^a   Palotti, Joao ^b   Goeuriot, Lorraine ^c   Kelly, Liadh ^d   Lupu, Mihai ^b   Pecina, Pavel ^e   Müller, Henning ^f   Budaher, Julie ^c   Deacon, Anthony ^a  

^a QUEENSLAND UNIVERSITY OF TECHNOLOGY   (Australia)

^b VIENNA UNIVERSITY OF TECHNOLOGY   (Austria)

^c UNIV GRENOBLE ALPES   (France)

^d TRINITY COLLEGE DUBLIN   (Ireland)

^e CHARLES UNIVERSITY   (Czech Republic)

^f UNIVERSITY OF APPLIED SCIENCES WESTERN SWITZERLAND   (Switzerland)

Author keywords

Evaluation; Health Search

Indexed keywords

HEALTH; LABORATORIES; MEDICAL EDUCATION; WEBSITES;

CONSUMER HEALTHS; EVALUATION; EVALUATION METHODS; HEALTH INFORMATIONS; MEDICAL CONDITIONS; MEDICAL INFORMATION; MEDICAL KNOWLEDGE; SEARCH TECHNOLOGY;

SEARCH ENGINES;

EID: 84984783073     PISSN: 16130073     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (33)

1. L. Azzopardi. Query Side Evaluation: An Empirical Analysis of Effectiveness and Effort. In Proc. of SIGIR, 2009.
2. P. Bailey, A. Moffat, F. Scholer, and P. Thomas. User Variability and IR System Evaluation. In Proc. of SIGIR, 2015.
3. M. Benigeri and P. Pluye. Shortcomings of health information on the internet. Health promotion international, 18(4):381-386, 2003.
4. M. Coleman and T. L. Liau. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60:283-284, 1975.
5. G. V. Cormack, C. L. A. Clarke, and S. Buettcher. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32Nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 09, pages 758-759, New York, NY, USA, 2009. ACM.
6. G. V. Cormack, M. D. Smucker, and C. L. Clarke. Efficient and effective spam filtering and re-ranking for large web datasets. Information retrieval, 14(5):441-465, 2011.
7. S. Fox. Health topics: 80% of internet users look for health information online. Pew Internet & American Life Project, 2011.
8. L. Goeuriot, G. J. Jones, L. Kelly, J. Leveling, A. Hanbury, H. Müller, S. Salantera, H. Suominen, and G. Zuccon. Share/clef ehealth evaluation lab 2013, task 3: Information retrieval to address patients questions when reading clinical reports. CLEF 2013 Online Working Notes, 8138, 2013.
9. L. Goeuriot, L. Kelly, W. Lee, J. Palotti, P. Pecina, G. Zuccon, A. Hanbury, and H. M. Gareth J.F. Jones. ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: User-centred health information retrieval. In CLEF 2014 Evaluation Labs and Workshop: Online Working Notes, Sheffield, UK, 2014.
10. L. Goeuriot, L. Kelly, G. Zuccon, and J. Palotti. Building evaluation datasets for consumer-oriented information retrieval. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may 2016. European Language Resources Association (ELRA).
11. R. Gunning. The Technique of Clear Writing. McGraw-Hill, 1952.
12. A. Hanbury. Medical information retrieval: An instance of domain-specific search. In Proceedings of SIGIR 2012, pages 1191-1192, 2012.
13. D. Hiemstra and C. Hauff. Mirex: Mapreduce information retrieval experiments. arXiv preprint arXiv:1004.4489, 2010.
14. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422-446, 2002.
15. L. Kelly, L. Goeuriot, H. Suominen, A. Neveol, J. Palotti, and G. Zuccon. Overview of the CLEF eHealth Evaluation Lab 2016. In Information Access Evaluation. Multilinguality, Multimodality, and Visualization. Springer Berlin Heidelberg, 2016.
16. B. Koopman and G. Zuccon. Relevation! an open source system for information retrieval relevance assessment. arXiv preprint, 2013.
17. A. Lipani, G. Zuccon, L. Mihai, B. Koopman, and A. Hanbury. The impact of fixed-cost pooling strategies on test collection bias. In Proceedings of the 2016 International Conference on The Theory of Information Retrieval, ICTIR 16, New York, NY, USA, 2016. ACM.
18. C. Macdonald, R. McCreadie, R. L. Santos, and I. Ounis. From puppy to maturity: Experiences in developing terrier. Proc. of OSIR at SIGIR, pages 60-63, 2012.
19. D. McDaid and A. Park. Online health: Untangling the web. evidence from the bupa health pulse 2010 international healthcare survey. Technical report, 2011.
20. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst., 27(1):2:1-2:27, Dec. 2008.
21. J. Palotti, L. Goeuriot, G. Zuccon, and A. Hanbury. Ranking health web pages with relevance and understandability. In Proceedings of the 39th international ACM SIGIR conference on Research and development in information retrieval, 2016
22. J. Palotti, G. Zuccon, J. Bernhardt, A. Hanbury, and L. Goeuriot. Assessors Agreement: A Case Study across Assessor Type, Payment Levels, Query Variations and Relevance Dimensions. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 7th International Conference of the CLEF Association, CLEF16 Proceedings. Springer International Publishing, 2016.
23. J. Palotti, G. Zuccon, L. Goeuriot, L. Kelly, A. Hanburyn, G. J. Jones, M. Lupu, and P. Pecina. CLEF eHealth Evaluation Lab 2015, Task 2: Retrieving Information about Medical Symptoms. In CLEF 2015 Online Working Notes. CEUR-WS, 2015.
24. J. Palotti, G. Zuccon, and A. Hanbury. The influence of pre-processing on the estimation of readability of web documents. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM15, pages 1763-1766, New York, NY, USA, 2015. ACM.
25. L. Park and Y. Zhang. On the distribution of user persistence for rank-biased precision. In Proceedings of the 12th Australasian document computing symposium, pages 17-24, 2007.
26. J. Pomiḱalek. Removing boilerplate and duplicate content from web corpora. PhD thesis, Masaryk university, Faculty of informatics, Brno, Czech Republic, 2011.
27. W. Shen, J.-Y. Nie, X. Liu, and X. Liui. An investigation of the effectiveness of concept-based approach in medical information retrieval GRIUM@ CLEF2014eHealthTask 3. In Proceedings of the CLEF eHealth Evaluation Lab, 2014.
28. T. Strohman, D. Metzler, H. Turtle, and W. B. Croft. Indri: A language modelbased search engine for complex queries. In Proceedings of the International Conference on Intelligent Analysis, volume 2, pages 2-6. Citeseer, 2005.
29. R. W. White and E. Horvitz. Cyberchondria: studies of the escalation of medical concerns in web search. ACM TOIS, 27(4):23, 2009.
30. D. Zhu, S. T.-I. Wu, J. J. Masanz, B. Carterette, and H. Liu. Using discharge summaries to improve information retrieval in clinical domain. In Proceedings of the CLEF eHealth Evaluation Lab, 2013.
31. G. Zuccon. Understandability biased evaluation for information retrieval. In Advances in Information Retrieval, pages 280-292, 2016.
32. G. Zuccon and B. Koopman. Integrating understandability in the evaluation of consumer health search engines. In Medical Information Retrieval Workshop at SIGIR 2014, page 32, 2014.
33. G. Zuccon, B. Koopman, and J. Palotti. Diagnose this if you can: On the effectiveness of search engines in finding medical self-diagnosis information. In Advances in Information Retrieval, pages 562-567. Springer, 2015.