Understanding user satisfaction with intelligent assistants

CHIIR 2016 - Proceedings of the 2016 ACM Conference on Human Information Interaction and Retrieval

Volumn , Issue , 2016, Pages 121-130

  Kiseleva, Julia ^a   Williams, Kyle ^b   Jiang, Jiepu ^c   Awadallah, Ahmed Hassan ^d   Crook, Aidan C ^d   Zitouni, Imed ^d   Anastasakos, Tasos ^d  

^a EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b PENNSYLVANIA STATE UNIVERSITY   (United States)

^c UNIVERSITY OF MASSACHUSETTS   (United States)

^d MICROSOFT   (United States)

Author keywords

Intelligent assistant; Mobile search; Spoken dialogue system; User experience; User satisfaction; User study

Indexed keywords

INFORMATION RETRIEVAL; INTELLIGENT AGENTS; MOBILE DEVICES; MOBILE TELECOMMUNICATION SYSTEMS; SPEECH PROCESSING;

INTELLIGENT ASSISTANTS; MOBILE SEARCH; SPOKEN DIALOGUE SYSTEM; USER EXPERIENCE; USER SATISFACTION; USER STUDY;

WORLD WIDE WEB;

EID: 84974602953     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2854946.2854961     Document Type: Conference Paper

References (48)

1. M. Ageev, Q. Guo, D. Lagun, and E. Agichtein. Find it if you can: a game for modeling different types of web search success using interaction data. In SIGIR, 2011.
2. M. Ageev, D. Lagun, and E. Agichtein. Improving search result summaries by using searcher behavior data. In SIGIR, pages 13-22, 2013.
3. E. Agichtein, E. Brill, and S. T. Dumais. Improving web search ranking by incorporating user behavior information. In SIGIR, pages 19-26, 2006.
4. A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between ir effectiveness measures and user satisfaction. In SIGIR, pages 773-774, 2007.
5. P. Borlund. The iir evaluation model: a framework for evaluation of interactive information retrieval systems. Inf. Res. (IRES), 8(3), 2003.
6. A. Chuklin and P. Serdyukov. How query extensions reflect search result abandonments. In SIGIR, pages 1087-1088, 2012.
7. A. Chuklin and P. Serdyukov. Good abandonments in factoid queries. In WWW (Companion Volume), pages 483-484, 2012.
8. A. Diriye, R. White, G. Buscher, and S. T. Dumais. Leaving so soon?: understanding and predicting web search abandonment rationales. In CIKM, pages 1025-1034, 2012.
9. A. Dong, Y. Chang, Z. Zheng, G. Mishne, J. Bai, R. Zhang, K. Buchner, and C. L. F. Diaz. Towards recency ranking in web search. In WSDM, pages 11-20, 2010.
10. H. A. Feild, J. Allan, and R. Jones. Predicting searcher frustration. In SIGIR, pages 34-41, 2010.
11. S. Fox, K. Karnawat, M. Mydland, S. T. Dumais, and T. White. Evaluating implicit measures to improve web search. ACM Trans. Inf. Syst. (TOIS), 23(2):147-168, 2005.
12. Google Inc. Teens use voice search most, even in bathroom, google's mobile voice study finds, 2015. http://prn. to/1sfiQRr.
13. Q. Guo, H. Jin, D. Lagun, S. Yuan, and E. Agichtein. Towards estimating web search result relevance from touch interactions on mobile devices. In CHI Extended Abstracts 2013, pages 1821-1826, 2013.
14. A. Hassan. A semi-supervised approach to modeling web search satisfaction. In SIGIR, pages 275-284, 2012.
15. A. Hassan and R. W. White. Personalized models of search satisfaction. In CIKM, pages 2009-2018, 2013.
16. A. Hassan, R. Jones, and K. L. Klinkner. Beyond DCG: user behavior as a predictor of a successful search. In WSDM, pages 221-230, 2010.
17. A. Hassan, X. Shi, N. Craswell, and B. Ramsey. Beyond clicks: query reformulation as a predictor of search satisfaction. In CIKM, pages 2019-2028, 2013.
18. A. Hassan Awadallah, R. W. White, S. T. Dumais, and Y.-M. Wang. Struggling or exploring?: disambiguating long search sessions. In WSDM, pages 53-62, 2014.
19. L. P. Heck, D. Hakkani-Tür, M. Chinthakunta, G. Tür, R. Iyer, P. Parthasarathy, L. Stifelman, E. Shriberg, and A. Fidler. Multi-modal conversational search and browse. In SLAM@INTERSPEECH, pages 96-101, 2013.
20. P. Ingwersen and K. Järvelin. The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series). Springer-Verlag New York, 2005.
21. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst. (TOIS), 20(4):422-446, 2002.
22. K. Järvelin, S. L. Price, L. M. L. Delcambre, and M. L. Nielsen. Discounted cumulated gain based evaluation of multiple-query ir sessions. In ECIR, pages 4-15, 2008.
23. J. Jiang, W. Jeng, and D. He. How do users respond to voice input errors?: lexical and phonetic query reformulation in voice search. In SIGIR, pages 143-152, 2013.
24. J. Jiang, A. H. Awadallah, X. Shi, and R. W. White. Understanding and predicting graded search satisfaction. In WSDM, 2015.
25. J. Jiang, A. Hassan Awadallah, R. Jones, U. Ozertem, I. Zitouni, R. G. Kulkarni, and O. Z. Khan. Automatic online evaluation of intelligent assistants. In WWW, pages 506-516, 2015.
26. T. Joachims. Optimizing search engines using clickthrough data. In KDD, pages 133-142, 2002.
27. T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In SIGIR, pages 154-161, 2005.
28. D. Kelly. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval (FTIR), 3(1-2):1-224, 2009.
29. D. Kelly. When effort exceeds expectations: A theory of search task difficulty (keynote). In SCST'15: Proceedings of the First International Workshop on Supporting Complex Search Tasks, volume 1338 of CEUR Workshop Proceedings, 2015.
30. Y. Kim, A. Hassan, R. W. White, and Y.-M. Wang. Playing by the rules: mining query associations to predict search performance. In WSDM, pages 133-142, 2013.
31. J. Kiseleva, E. Crestan, R. Brigo, and R. Dittel. Modelling and detecting changes in user satisfaction. In CIKM, pages 1449-1458, 2014.
32. J. Kiseleva, J. Kamps, V. Nikulin, and N. Makarov. Behavioral dynamics from the serp's perspective: What are failed serps and how to fix them? In CIKM, 2015.
33. D. Lagun, C.-H. Hsieh, D. Webster, and V. Navalpakkam. Towards better measurement of attention and satisfaction in mobile search. In SIGIR, pages 113-122, 2014.
34. J. Li, S. B. Huffman, and A. Tokuda. Good abandonment in mobile and pc internet search. In SIGIR, pages 43-50, 2009.
35. M. F. McTear. Spoken dialogue technology: enabling the conversational user interface. ACM Computing Surveys (CSUR), 34(1):90-169, 2002.
36. M. Negri, M. Turchi, J. G. C. De Souza, and D. Falavigna. Quality estimation for automatic speech recognition. In COLING, pages 1813-1823, 2014.
37. T. Saracevic. Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science, 26:321-343, 1975.
38. T. Saracevic, P. B. Kantor, A. Y. Chamis, and D. Trivison. A study of information seeking and retrieving. I. background and methodology. II. users, questions and effectiveness. III. searchers, searches, overlap. Journal of the American Society for Information Science and Technology, 39:161-176; 177-196; 197-216, 1988.
39. M. Shokouhi and Q. Guo. From queries to cards: Re-ranking proactive card recommendations based on reactive search history. In SIGIR, pages 695-704, 2015.
40. G. Tür. Extending boosting for large scale spoken language understanding. Machine Learning (ML), 69(1):55-74, 2007.
41. G. Tür, Y.-Y. Wang, and D. Z. Hakkani-Tür. Techware: Spoken language understanding resources [best of the web]. IEEE Signal Process. Mag. (SPM), 30(3):187-189, 2013.
42. G. Tür, Y.-Y. Wang, and D. Z. Hakkani-Tür. Understanding spoken language. Computing Handbook, 3rd ed (41):1-17, 2014.
43. P. Vakkari. Task-based information searching. ARIST, 37:413-464, 2003.
44. W. Wahlster. Smartkom: Foundations of multimodal dialogue systems. Springer, 2006.
45. Y. Wang and E. Agichtein. Query ambiguity revisited: Clickthrough measures for distinguishing informational and ambiguous queries. In HLT-NAACL, pages 361-364, 2010.
46. B. Wildemuth, L. Freund, and E. G. Toms. Untangling search task complexity and difficulty in the context of interactive information retrieval studies. Journal of Documentation, 70:1118-1140, 2014.
47. T. Wilson. Models in information behaviour research. Journal of Documentation, 55(3):249-270, 1999.
48. E. Yilmaz, M. Verma, N. Craswell, F. Radlinski, and P. Bailey. Relevance and effort: An analysis of document utility. In CIKM, pages 91-100, 2014.