An examination of two delivery modes for interactive search system experiments: Remote and laboratory

Conference on Human Factors in Computing Systems - Proceedings

Volumn , Issue , 2011, Pages 1531-1540

  Kelly, Diane ^a   Gyllstrom, Karl ^b  

^a UNIVERSITY OF NORTH CAROLINA   (United States)

^b UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Interactive search system evaluation; Online experiment; Remote usability study; Study mode

Indexed keywords

SAFETY DEVICES;

DELIVERY MODES; INTERACTIVE SEARCH; INTERACTIVE SEARCH SYSTEM EVALUATION; ON-LINE EXPERIMENTS; REMOTE USABILITY; REMOTE USABILITY STUDY; SEARCH SYSTEM; STUDY MODE; SYSTEM EVALUATION; USABILITY STUDIES;

LABORATORIES;

EID: 79958085438     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1978942.1979166     Document Type: Conference Paper

References (32)

1. Alonso, O. & Mizzaro, S. Relevance critieria for ecommerce: A crowdsourcing-based experimental analysis. In Proc. SIGIR 2009, ACM (2009), 760-761.
2. Alonso, O., Rose, D. E., & Stewart, B. Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2) (2008).
3. Amazon Mechanical Turk. https://www.mturk.com/
4. Andreasen, M. S., Nielsen, H. V., Schrøder, S. O., & Stage, J. What happened to remote usability testing? An empirical study of three methods. In Proc. CHI 2007, ACM Press, (2007), 1405-1414. (Pubitemid 47581732)
5. Anick, P. Using terminological feedback for web search refinement: A log based study. In Proc. SIGIR 2003, ACM Press, 2003, 88-95.
6. Birnbaum, M. H. Psychological Experiments on the Internet. London, UK: Academic Press, 2000.
7. Bruun, A., Gull, P., Hofmeister, L. & Stage, J. Let your users do the testing: A comparison of three remote asynchronous usability testing methods. In Proc. CHI 2009, ACM Press (2009), 1619-1628.
8. Campbell, D. T. & Stanley, J. C. Experimental and quasi-experimental designs for research. Chicago: Rand McNally, 1966.
9. Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. Are your participants gaming the system? Screening Mechanical Turk workers. In Proc. CHI 2010, ACM Press (2010), 2399-2402.
10. Dumais, S. T. & Belkin, N. J. The TREC Interactive Tracks: Putting the user into search. In TREC: Experiment and Evaluation in Information Retrieval (pp. 123-153), Cambridge: MIT Press, 2005.
11. ESP Online Game. http://www.gwap.com/gwap/gamesPreview/espgame/.
12. Hammontree, M., Weiler, M. & Nayak, N. Remote usability testing. Interactions 1(3) (1994), 21-25.
13. Hartson, H. R., Castillo, J.C., Kelso, J., & Neale, W. C. Remote evaluation: The network as an extension of the usability laboratory. In Proc. of CHI 1996, ACM Press (1996), 228-235.
14. Kazai, G., Milic-Frayling, N. & Costello, J. Towards methods for the collective gathering and quality control of relevance assessments. In Proc. SIGIR 2009, ACM Press (2009), 452-459.
15. Kelly, D. Methods for evaluating interactive information retrieval systems with users. Foundations & Trends in Information Retrieval, 3(1-2) (2009), 1-224.
16. Kelly, D., Gyllstrom, K., & Bailey, E. (2009). A comparison of term and query suggestion features for interactive searching. In Proc. of SIGIR 2009, ACM Press (2009), 371-378.
17. Kittur, A., Chi, E. H., & Suh, B. Crowdsourcing user studies with Mechanical Turk. In Proc. CHI 2008, ACM Press (2008), 453-456.
18. Lemur IR Toolkit. http://www.lemurproject.org/.
19. McFadden, E., Hager, D. R., Elie, C. J., & Blackwell, J. M. Remote usability evaluation: Overview and case studies. International Journal of Human-Computer Interaction, 14(3-4) (2002), 489-502. (Pubitemid 36611302)
20. Mulaik, S. A. Foundations of factor analysis. Boca Raton, FL: CRC Press, 1972.
21. Radlinski, F., Kurup, M. & Joachims, T. (2008). How does clickthrough data reflect retrieval quality? In Proc. of CIKM 2008, ACM Press (2008), 43-52.
22. Reips, U.-D. The Web's Experimental Psychology Lab. http://iscience. deusto.es/archive/ulf/Lab/WebExpPsyLab.html, 1995.
23. Robertson, S. E. On the history of evaluation in IR. Journal of Information Science, 34(4) (2008), 439-456.
24. Ross, J., Irani, L., Silberman, M. S., Saldivar, A., & Tomlinson, B. Who are the crowdworkers? Shifting demographics in Mechanical Turk. In Proc. CHI 2010, ACM Press (2010), 2863-2872.
25. Scholtz, J. Adaptation of traditional usability testing method for remote testing. In Proc. HICSS 2001, 1-8.
26. th edition). Newbury Park, CA: Sage Publications, 1991.
27. Spärck-Jones, K. Information Retrieval Experiment. London, UK: Butterworths & Co. Ltd, 1981.
28. Tague-Sutcliffe, J. M. The pragmatics of information retrieval experimentation, revisted. Information Processing & Management, 28(4) (1992), 467-490.
29. Toms, E. G., Freund, L. & Li, C. WISSE: The Web interactive information retrieval experimentation system prototype. Information Processing & Management, 40(5) (2004), 655-675.
30. von Ahn, L. Games with a purpose. IEEE Computer Magazine, June 2006, 96-98.
31. Voorhees, E. M. Overview of the TREC 2005 Robust Retrieval Track. Proc. of TREC-14 (2006).
32. Voorhees, E. M. & Harman, D. K. TREC: Experiment and Evaluation in Information Retrieval, Cambridge, MA: MIT Press (2005).