An eye-tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine

Journal of the American Society for Information Science and Technology

Volumn 63, Issue 9, 2012, Pages 1728-1746

  Balatsoukas, Panos ^a   Ruthven, Ian ^a  

^a UNIVERSITY OF STRATHCLYDE   (United Kingdom)

Author keywords

information seeking; judgment; user studies

Indexed keywords

DECISION MAKING PROCESS; EYE-TRACKING; GOOGLE SEARCH ENGINE; HUMAN REASONING; INFORMATION SEEKING; JUDGMENT; METHODOLOGICAL FRAMEWORKS; MULTI METHODS; RELEVANCE CRITERIA; RELEVANCE JUDGMENT; RESEARCH DESIGNS; USER STUDY; VISUAL BEHAVIOR;

INTERNET PROTOCOLS; RESEARCH; SEARCH ENGINES;

EYE MOVEMENTS;

EID: 84865204028     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.22707     Document Type: Article

References (36)

1. Aula, A., Majaranta, P., &, Raiha, K-J., (2005). Eye-tracking reveals the personal styles for search result evaluation. In, M.F. Constabile, &, F. Paterno, (Eds.), INTERACT 2005 (pp. 1058-1061). Berlin: Springer-Verlag.
2. Balatsoukas, P., Morris, A., &, O'Brien, A., (2009). An evaluation framework of user interaction with metadata surrogates. Journal of Information Science, 35 (3), 321-339.
3. Balatsoukas, P., &, Ruthven I., (2010). The use of relevance criteria during predictive judgment: An eye tracking approach. Proceedings of the American Society for Information Science and Technology, 47 (1), 1-10. doi: 10.1002/meet.14504701145.
4. Barry, C., &, Schamber, L., (1998). Users' criteria for relevance evaluation: A cross-situational comparison. Information Processing and Management, 34 (2/3), 219-236. (Pubitemid 128392791)
5. Borlund, P., (2003). The concept of relevance in IR. Journal of the American Society of Information Science and Technology, 54 (10), 913-925.
6. Borlund, P., &, Ingwersen, P., (1997). The development of a method for the evaluation of interactive information retrieval systems. Journal of Documentation, 53 (3), 225-250.
7. th International Conference on Human Factors in Computing Systems (pp. 21-30). New York: ACM.
8. Crystal, A., &, Greenberg, J., (2006). Relevance criteria identified by health information users during Web searches. Journal of the American Society for Information Science and Technology, 57 (10), 1368-2882.
9. Cutrell, E., &, Guan, Z., (2007). What are you looking for? An eye-tracking study of information usage in Web search. In Proceedings of the ACM HCI 2007 Conference on Human Factors in Computing Systems (pp. 407-415). New York: ACM.
10. Duchowski, A., (2007). Eye tracking methodology: Theory and practice. Berlin: Springer-Verlag.
11. Dumais, S., Buscher, G., &, Cutrell, E., (2010). Individual differences in gaze patterns for Web search. In Proceedings of the Third Symposium on Information Interaction in Context (pp. 185-194). New York: ACM.
12. Eisenberg, M.B., (1988). Measuring relevance judgments. Information Processing & Management, 24 (4), 373-389.
13. Ericsson, A., &, Simon, H., (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
14. Foster, A., &, Ford, N., (2003). Serendipity and information seeking: An empirical study. Journal of Documentation, 59, 321-340. (Pubitemid 36594169)
15. Gerjets, P., Kammerer, Y., &, Werner, B., (2011). Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking aloud protocols and eye tracking data. Learning and Instruction, 21 (2), 220-231.
16. Goldberg, J.H., Stimson, M., Lewenstein, M., Scott, N., &, Wichansky, A., (2002). Eye Tracking in Web search tasks: Design implications. In Proceedings of the Eye tracking Research and Applications Symposium (ETRA) (pp. 51-58). New York: ACM.
17. Granka, L., Joachims, T., &, Gay, G., (2004). Eye-tracking analysis of user behavior in WWW search. In, M. Sanderson, K. Jarvelin, J. Allan, &, P. Bruza, (Eds.), Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 478-479). New York: ACM.
18. Hughes, A., Wilkens, T., Wildemuth, B., &, Marchionini, G., (2003). Text or pictures? An eye tracking study of how people view digital video surrogates. In, E.M. Bakker, T.S. Huang, M.S. Lew, N. Sebe, &, X.S. Zhou, (Eds.), Proceedings of the Second International Conference CIVR 2003 Urbana-Champaign, IL, July 24-25 (pp. 271-280). Berlin: Springer-Verlag.
19. Lorigo, L, Haridasan, M., Brynjarsdottir, H., Xia, L., Joachims, T., Gay, G., Pan, B., (2008). Eye tracking and online search: Lessons learned and challenges ahead. Journal of the American Society for Information Science and Technology, 59 (7), 1041-1052. (Pubitemid 351731797)
20. Lorigo, L., Pan, B., Hembrooke, H., Joachims, T., Granka, L., &, Gay, G., (2006). The influence of task and gender on search and evaluation behavior using Google. Information Processing and Management, 42, 1123-1131. (Pubitemid 41828304)
21. Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., &, Granka, L., (2007). In Google we trust: Users' decisions on rank, position and relevance. Journal of Computer-Mediated Communication, 12, 801-823.
22. Papaeconomou, C., Zijlema, A., &, Ingwersen, P., (2008). Searchers' relevance judgments and criteria in evaluating Web pages in a learning style perspective. In Proceedings of the Second international symposium on Information Interaction in Context (IIX 2008) (pp. 123-132). New York: ACM Press.
23. Rayner, K., (2009). Eye movements and attention in reading, scene perception and visual search. The Quarterly Journal of Experimental Psychology, 62, 1457-1506.
24. Rele, S., &, Duchowski, A.T., (2005). Using eye tracking to evaluate alternative search results interfaces. Human Factors and Ergonomics Society Annual Meeting, 49 (15), 1459-1463. (Pubitemid 351742518)
25. Rieh, S.Y., (2002). Judgment of information quality and cognitive authority in the web. Journal of the American Society for Information Science and Technology, 53 (2), 145-161. (Pubitemid 34138427)
26. Rodden, K., Fu, X., Aula, A., &, Spiro, I., (2008). Eye-mouse coordination patterns on Web search pages. In Proceedings of the ACM CHI 2005 Conference on Human Factors in Computing Systems (pp. 2997-3002). New York: ACM Press.
27. Ruthven, I., (2008). Interactive information retrieval. Annual Review of Information Science and Technology, 42, 43-92.
28. Saracevic, T., &, Su, L., (1989). Modeling and measuring user-intermediary computer interaction in online searching: Design of a study. Proceedings of the Annual Meeting of the American Society for Information Science and Technology, 26, 75-80.
29. Savolainen, R., &, Kari, J., (2006). User-defined relevance criteria in Web searching. Journal of Documentation, 62 (6), 685-707. (Pubitemid 44704537)
30. Schamber, L., Eisenberg, M., &, Nilan, M., (1990). A re-examination of relevance: Towards a dynamic, situational definition. Journal of Information Processing and Management, 26 (6), 755-776.
31. Spink, A., &, Greisdorf, H., (2001). Regions and levels: Measuring and mapping users' relevance judgments. Journal of the American Society for Information Science and Technology, 52 (2), 161-173. (Pubitemid 32134188)
32. Spink, A., Greisdorf, H., &, Bateman, J., (1998). From highly relevant to not relevant: Examining different regions of relevance. Information Processing and Management, 34 (5), 599-621. (Pubitemid 128410409)
33. Tang, R., &, Solomon, P., (2001) Use of relevance criteria across stages of document evaluation: On the complementarity of experimental and naturalistic studies, Journal of the American Society for Information Science and Technology, 52 (8), 676-685. (Pubitemid 32551466)
34. Tombros, A., Ruthven, I., &, Jose, J., (2005). How users' assess Web pages for information seeking. Journal of the American society for Information Science and Technology, 56 (4), 327-344. (Pubitemid 40320108)
35. Wang, P., &, Soergel, D., (1998). A cognitive model of document use during a research project. Study I. Document selection. Journal of the American Society for Information Science and Technology, 49 (2), 115-133. (Pubitemid 128594911)
36. Zhang, Y., &, Yuelin, L., (2008). A user-centered functional metadata evaluation of moving image collections. Journal of the American Society for Information Science and Technology, 59 (8), 1331-1346.