Automatic online evaluation of intelligent assistants

WWW 2015 - Proceedings of the 24th International Conference on World Wide Web

Volumn , Issue , 2015, Pages 506-516

  Jiang, Jiepu ^a   Awadallah, Ahmed Hassan ^b   Jones, Rosie ^b   Ozertem, Umut ^b   Zitouni, Imed ^b   Kulkarni, Ranjitha Gurunath ^b   Khan, Omar Zia ^b  

^a University of Massachusetts Amherst   (United States)

^b MICROSOFT   (United States)

Author keywords

Evaluation; Mobile search; Spoken dialog system; User experience; Voice Activated intelligent assistant

Indexed keywords

FORECASTING; INFORMATION RETRIEVAL; MOBILE DEVICES; MOBILE TELECOMMUNICATION SYSTEMS; WEBSITES; WORLD WIDE WEB;

EVALUATION; INTELLIGENT ASSISTANTS; MOBILE SEARCH; SPOKEN DIALOG SYSTEMS; USER EXPERIENCE;

SPEECH RECOGNITION;

EID: 84968741455     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2736277.2741669     Document Type: Conference Paper

References (33)

1. Ageev, M., Guo, Q., Lagun, D. and Agichtein, E. (2011). Find it if you can: A game for modeling different types of web search success using interaction data. Proc. SIGIR 11, 345-354.
2. Smith R.W. and Hipp, D.R. (1995). Spoken Natural Lan-guage Dialog Systems: A Practical Approach. Oxford Uni-versity Press.
3. Feild, H.A., Allan, J. and Jones, R. (2010). Predicting searcher frustration. Proc. SIGIR 10, 34-41.
4. Fox, S., Karnawat, K., Mydland, M., Dumais, S. and White, T. 2005. Evaluating implicit measures to improve web search. ACM TOIS, 23(2), 147-168.
5. Friedman, J., Hastie, T., Tibshirani, R. (2000). Additive lo-gistic regression: A statistical view of boosting (with discus-sion and a rejoinder by the authors). The Annals of Statistics.
6. Hassan, A. (2012). A semi-supervised approach to modeling web search satisfaction. Proc. SIGIR 12, 275-284.
7. Hassan, A., Jones, R. and Klinkner, K.L. (2010). Beyond DCG: user behavior as a predictor of a successful search. Proc. WSDM 10, 221-230.
8. Hassan, A., Shi, X., Craswell, N. and Ramsey, B. (2013). Be-yond clicks: query reformulation as a predictor of search sat-isfaction. Proc. CIKM 13, 2019-2028.
9. Hassan, A., Song, Y. and He, L. (2011). A task level metric for measuring web search satisfaction and its application on improving relevance estimation. Proc. CIKM 11, 125-134.
10. Hassan, A., White, R.W., Dumais, S.T. and Wang, Y.M. (2014). Proc. WSDM 14, 53-62.
11. Heck, L.P., Hakkani-Tör, D., Chinthakunta, M., Tör, G., Iyer, R., Parthasarathy, P., Stifelman, L., Shriberg, E. and Fidler, A. (2013). Multi-modal conversational search and browse. Proceedings of the First Workshop on Speech, Lan-guage and Audio in Multimedia, 96-101.
12. Huang, P.S., Kumar, K., Liu, C., Gong, Y. and Deng, L. (2013). Predicting speech recognition confidence using deep learning with word identity and score features. Proc. ICASSP, 7413-7417.
13. Huffman, S.B. and Hochster, M. (2007). How well does re-sult relevance predict session satisfaction? Proc. SIGIR 07, 567-574.
14. Järvelin, K. and Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM TOIS, 20(4), 422-446.
15. Järvelin, K., Price, S., Delcambre, L.L. and Nielsen, M. (2008). Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions. Proc. ECIR 08, 4-15.
16. Jeng, W., Jiang, J. and He, D. (2013). Users Perceived Diffi-culties and Corresponding Reformulation Strategies in Voice Search. Proc. HCIR 2013.
17. Jiang, J., Hassan Awadallah, A., Shi, X. and White, R.W. (2015). Understanding and Predicting Graded Search Satis-faction. Proc. WSDM 15, 57-66.
18. Jiang, J., He, D. and Allan, J. (2014). Searching, browsing, and clicking in a search session. Proc. SIGIR 14, 607-616.
19. Jiang, J., Jeng, W. and He, D. (2013). How do users respond to voice input errors? lexical and phonetic query reformula-Tion in voice search. Proc. SIGIR 13, 143-152.
20. Johnston, M., Bangalore, S., Vasireddy, G., Stent, A., Ehlen, P., Walker, M., Whittaker, S. and Maloor, P. (2002). MATCH: An architecture for multimodal dialogue systems. Proc. ACL 02, 376-383.
21. Kim, Y., Hassan, A., White, R.W. and Zitouni, I. (2014).comparing client and server dwell time estimates for click-level satisfaction prediction. Proc. SIGIR 14, 895-898.
22. Kim, Y., Hassan, A., White, R.W. and Zitouni, I. (2014). Modeling dwell time to predict click-level satisfaction. Proc. WSDM 14, 193-202.
23. Kotov, A., Bennett, P.N., White, R.W., Dumais, S.T. and Teevan, J. (2011). Modeling and analysis of cross-session search tasks. Proc. SIGIR 11, 5-14.
24. Niu, X. and Kelly, D. (2014). The use of query suggestions during information search. IPandM, 50(1), 218-234.
25. Philips, L. (1990). Hanging on the Metaphone.computer Language, 7(12), 39-44.
26. Shokouhi, M., Jones, R., Ozertem, U., Raghunathan, K. and Diaz, F. 2014. Mobile query reformulations. Proc. SIGIR 14, 1011-1014.
27. Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Van Ess-Dykema, C. and Meteer, M. (2000). Dialogue act modeling for automatic tag-ging and recognition of conversational speech.computa-Tional linguistics, 26(3), 339-373.
28. Traum, D.R. (2000). 20 questions on dialogue act taxono-mies. Journal of semantics, 17(1), 7-30.
29. Tur, G. and De Mori, R. (2011). Spoken language under-standing: Systems for extracting semantic information from speech. John Wiley & Sons.
30. Wahlster, W. (2006). SmartKom: foundations of multimodal dialogue systems. Springer.
31. Walker, M.A., Litman, D.J., Kamm, C.A. and Abella, A. (1997). PARADISE: A framework for evaluating spoken dia-logue agents. Proc. ACL 97, 271-280.
32. Wang, H., Song, Y., Chang, M.W., He, X., Hassan, A. and White, R.W. (2014). Modeling action-level satisfaction for search task satisfaction prediction. SIGIR 14, 123-132.
33. Young, S., Gasic, M., Thomson, B. and Williams, J.D. (2013). POMDP-based statistical spoken dialog systems: A review. Proc. IEEE, 101(5), 1160-1179.