Quantifying the User Experience

Quantifying the User Experience

Volumn , Issue , 2012, Pages

  Sauro, Jeff ^a   Lewis, James R ^b  

^a Measuring Usability LLC   (United States)

^b IBM   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85013689963     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/C2010-0-65192-3     Document Type: Book

References (304)

1. Lewis, J.R., Sauro, J., 2012. Excel and R Companion to "Quantifying the User Experience: Practical Statistics for User Research": Rapid Answers to over 100 Examples and Exercises. Create Space Publishers, Denver.
2. UPA. 2011. The Usability Professionals Association salary survey. Available at (accessed July 29, 2011). http://www.usabilityprofessionals.org/usability_resources/surveys/SalarySurveys.html.
3. Albert W., Tullis T., Tedesco D. Beyond the Usability Lab. 2010, Morgan Kaufmann, Boston.
4. Aykin N.M., Aykin T. Individual differences in human-computer interaction. Comput. Ind. Eng. 1991, 20:373-379.
5. Bisbort A. Escaping Flatland Available at, (accessed July 17, 2011). http://www.edwardtufte.com/tufte/advocate_1099.
6. Dickens J. The fresh cream cakes market: The use of qualitative research as part of a consumer research programme. Applied Marketing and Social Research 1987, 23-68. Wiley, New York. U. Bradley (Ed.).
7. Dumas J., Redish J.C. A Practical Guide to Usability Testing 1999, Intellect, Portland.
8. Gordon W., Langmaid R. Qualitative Market Research: A Practitioner's and Buyer's Guide 1988, Gower Publishing, Aldershot, England.
9. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs)-Part 11: Guidance on Usability (ISO 9241-11:1998(E)) 1998, ISO, ISO, Geneva.
10. Lewis J.R. Usability testing. Handbook of Human Factors and Ergonomics 2012, 1267-1312. Wiley, New York. G. Salvendy (Ed.).
11. Nielsen J. Success Rate: The Simplest Usability Metric Available at, (accessed July 10, 2011). http://www.useit.com/alertbox/20010218.html.
12. Reichheld F.F. The one number you need to grow. Harvard Bus. Rev. 2003, 81:46-54.
13. Reichheld F. The Ultimate Question: Driving Good Profits and True Growth 2006, Harvard Business School Press, Boston.
14. Rubin J. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests 1994, Wiley, New York.
15. Sauro J., Kindlund E. A method to standardize usability metrics into a single score. Proceedings of CHI 2005 2005, 401-409. ACM, Portland.
16. Sauro J. A Practical Guide to Measuring Usability 2010, Measuring Usability LLC, Denver.
17. Sauro J. How to Quantify Comments Available at, (accessed July 15, 2011). http://www.measuringusability.com/blog/quantify-comments.php.
18. Sauro J., Lewis J.R. Correlations among prototypical usability metrics: Evidence for the construct of usability. Proceedings of CHI 2009 2009, 1609-1618. ACM, Boston.
19. Schumacher R. The Handbook of Global User Research 2010, Morgan Kaufmann, Boston.
20. Scriven M. The methodology of evaluation. Perspectives of Curriculum Evaluation 1967, 39-83. Rand McNally, Chicago. R.W. Tyler, R.M. Gagne, M. Scriven (Eds.).
21. Tullis T., Albert B. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics 2008, Morgan Kaufmann, Boston.
22. Agresti A., Coull B. Approximate is better than "exact" for interval estimation of binomial proportions. Am. Stat. 1998, 52:119-126.
23. Blalock H.M. Social Statistics 1972, McGraw-Hill, New York.
24. Clopper C.J., Pearson E. The use of confidence intervals for fiducial limits illustrated in the case of the binomial. Biometrika 1934, 26:404-413.
25. Cordes R. The effects of running fewer subjects on time-on-task measures. Int. J. Hum. Comput. Interact. 1993, 5(4):393-403.
26. Laplace P.S. Theorieanalytique des probabilitites 1812, Courcier, Paris.
27. Lewis J.R. Binomial confidence intervals for small-sample usability studies. Advances in Applied Ergonomics: Proceedings of the 1st International Conference on Applied Ergonomics-ICAE '96 1996, 732-737. USA Publishing, Istanbul. G. Salvendy, A. Ozok (Eds.).
28. Lewis J.R., Sauro J. When 100% really isn't 100%: Improving the accuracy of small-sample estimates of completion rates. J. Usability Stud. 2006, 3(1):136-150.
29. Sauro J. Restoring Confidence in Usability Results Available at, (accessed July 29, 2011). http://www.measuringusability.com/conf_intervals.htm.
30. Sauro J., Lewis J.R. Estimating completion rates from small samples using binomial confidence intervals: Comparisons and recommendations. Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting 2005, 2100-2104. HFES, Santa Monica, CA.
31. Sauro J., Lewis J.R. Average task times in usability tests: What to report?. Proceedings of CHI 2010 2010, 2347-2350. ACM, Atlanta.
32. Smithson M. Confidence Intervals. Sage University Papers Series on Quantitative Applications in the Social Science, 07-140 2003, Sage, Thousand Oaks, CA.
33. Wilson E.B. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc. 1927, 22:209-212.
34. Agresti A., Franklin C.A. Statistics: The Art and Science of Learning from Data 2007, Pearson, New York.
35. Armitage P., Berry G., Matthews J.N.S. Statistical Methods in Medical Research 2002, Blackwell Science, Oxford, UK. 4th ed.
36. Bangor A., Kortum P.T., Miller J.T. An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 2008, 6:574-594.
37. Chapanis A. Some generalizations about generalization. Hum. Fact. 1988, 30:253-267.
38. Gonick L., Smith W. Cartoon Guide to Statistics 1993, Collins Reference, New York.
39. Lewis J.R. Testing small system customer set-up. Proceedings of the Human Factors Society 26th Annual Meeting 1982, 718-720. Human Factors Society, Santa Monica, CA.
40. Lewis J.R., Sauro J. The factor structure of the System Usability Scale. Human Centered Design, HCII 2009 2009, 94-103. Springer-Verlag, Berlin, Germany. M. Kurosu (Ed.).
41. Molich R., Kirakowski J., Sauro J., Tullis T. Comparative Usability Task Measurement Workshop (CUE-8) 2009, UPA, Portland, UPA 2009 Conference.
42. Rosenbaum S. Usability evaluations versus usability testing: When and why?. IEEE Trans. Prof. Commun. 1989, 32:210-216.
43. Sauro J. Estimating productivity: Composite operators for keystroke level modeling. Proceedings of the 13th International Conference on Human-Computer Interaction, HCII 2009 2009, 352-361. Springer-Verlag, Berlin. J.A. Jacko (Ed.).
44. Sauro J. Does better usability increase customer loyalty? Available at, (accessed October 2011). http://www.measuringusability.com/usability-loyalty.php.
45. Sauro J., Kindlund E. How long should a task take? Identifying specification limits for task times in usability tests. Proceeding of the Human Computer Interaction International Conference (HCII 2005) 2005, Mira Digital Publication, St. Louis.
46. Sauro J., Lewis J.R. Average task times in usability tests: What to report?. Proceedings of CHI 2010 2010, 2347-2350. ACM, Atlanta.
47. Agresti A., Caffo B. Simple and effective confidence intervals for proportions and differences of proportions result from adding two successes and two failures. Am. Stat. 2000, 54(4):280-288.
48. Agresti A., Coull B. Approximate is better than "exact" for interval estimation of binomial proportions. Am. Stat. 1998, 52:119-126.
49. Agresti A., Franklin C.A. Statistics: The Art and Science of Learning from Data 2007, Prentice-Hall, Upper Saddle River, NJ.
50. Agresti A., Min Y. Simple improved confidence intervals for comparing matched proportions. Stat. Med. 2005, 24:729-740.
51. Bland M. An Introduction to Medical Statistics 2000, Oxford University Press, Oxford, UK. third ed.
52. Box G.E.P. Non-normality and test on variance. Biometrika 1953, 40:318-355.
53. Campbell I. Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations. Stat. Med. 2007, 26:3661-3675.
54. 2 test of goodness of fit. Ann. Math. Stat. 1952, 23:315-345.
55. 2 tests. Biometrics 1954, 10:417-451.
56. Howell D. Statistical Methods for Psychology 2002, Thomson Learning, Andover, UK. fifth ed.
57. McNemar Q. Psychological Statistics 1969, Wiley, New York. fourth ed.
58. Pearson K. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philos. Mag. Ser. 1900, 5(50):157-175.
59. Satterthwaite F.E. An approximate distribution of estimates of variances components. Biometr. Bull. 1946, 2:110-114.
60. Sauro J. A Practical Guide to the System Usability Scale 2011, CreateSpace, Denver.
61. Sauro J. If you could only ask one question, use this one Available at, (accessed February 22, 2011). http://www.measuringusability.com/blog/single-question.php.
62. Welch B.L. The significance of the difference between two means when the population variances are unequal. Biometrika 1938, 29:350-362.
63. Agresti A., Coull B. Approximate is better than "exact" for interval estimation of binomial proportions. Am. Stat. 1998, 52:119-126.
64. Agresti A., Min Y. Simple improved confidence intervals for comparing matched proportions. Stat. Med. 2005, 24:729-740.
65. Alreck P.L., Settle R.B. The Survey Research Handbook 1985, Richard D. Irwin, Homewood, IL.
66. Bradley J.V. Probability; Decision; Statistics 1976, Prentice-Hall, Englewood Cliffs, NJ.
67. Brown F.E. Marketing Research: A Structure for Decision Making 1980, Addison-Wesley, Reading, MA.
68. Campbell D.T., Stanley J.C. Experimental and Quasi-experimental Designs for Research 1963, Rand McNally, Chicago.
69. Campbell I. Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations. Stat. Med. 2007, 26:3661-3675.
70. Churchill G.A. Marketing Research: Methodological Foundations 1991, Dryden Press, Ft. Worth, TX.
71. Cohen J. Statistical Power Analysis for the Behavioral Sciences 1988, Lawrence Erlbaum, Hillsdale, NJ. 2nd ed.
72. Cowles M. Statistics in Psychology: An Historical Perspective 1989, Lawrence Erlbaum, Hillsdale, NJ.
73. Diamond W.J. Practical Experiment Designs for Engineers and Scientists 1981, Lifetime Learning Publications, Belmont, CA.
74. Dumas J.S., Salzman M.C. Usability assessment methods. Reviews of Human Factors and Ergonomics 2006, Vol. 2:109-140. HFES, Santa Monica, CA. R.C. Williges (Ed.).
75. Kirakowski J. Summative usability testing: Measurement and sample size. Cost-Justifying Usability: An Update for the Internet Age 2005, 519-554. Elsevier, Amsterdam, Netherlands. R.G. Bias, D.J. Mayhew (Eds.).
76. Kraemer H.C., Thiemann S. How Many Subjects? Statistical Power Analysis in Research 1987, Sage Publications, Newbury Park, CA.
77. Landauer T.K. Behavioral research methods in human-computer interaction. Handbook of Human-Computer Interaction 1997, 203-227. Elsevier, Amsterdam, Netherlands. 2nd ed. M. Helander, T.K. Landauer, P. Prabhu (Eds.).
78. Lewis J.R. Usability testing. Handbook of Human Factors and Ergonomics 2012, 1267-1312. John Wiley, New York. 4th ed. G. Salvendy (Ed.).
79. Mayer R.E. From novice to expert. Handbook of Human-Computer Interaction 1997, 781-795. Elsevier, Amsterdam, Netherlands. 2nd ed. M.G. Helander, T.K. Landauer, P.V. Prabhu (Eds.).
80. Minium E.W., King B.M., Bear G. Statistical Reasoning in Psychology and Education 1993, John Wiley, New York. 3rd ed.
81. Myers J.L. Fundamentals of Experimental Design 1979, Allyn & Bacon, Boston.
82. Nielsen J. Usability testing. Handbook of Human Factors and Ergonomics 1997, 1543-1568. John Wiley, New York. 2nd ed. G. Salvendy (Ed.).
83. Parasuraman A. Nonprobability sampling methods. Marketing Research 1986, 498-516. Addison-Wesley, Reading, MA.
84. Salsburg D. The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century 2001, Henry Holt, New York.
85. Sauro J., Lewis J.R. Average task times in usability tests: What to report?. Proceedings of CHI 2010 2010, 2347-2350. ACM, Atlanta.
86. Scriven M. The methodology of evaluation. Perspectives of Curriculum Evaluation 1967, 39-83. Rand McNally, Chicago. R.W. Tyler, R.M. Gagne, M. Scriven (Eds.).
87. Stigler S.M. The History of Statistics: The Measurement of Uncertainty before 1900 1986, The Belknap Press of Harvard University Press, Cambridge, MA.
88. Walpole R.E. Elementary Statistical Concepts 1976, Macmillan, New York.
89. Wickens C.D. Commonsense statistics. Ergon. Des. 1998, 6(4):18-22.
90. Winer B.J., Brown D.R., Michels K.M. Statistical Principles in Experimental Design 1991, McGraw-Hill, New York. 3rd ed.
91. Agresti A. Simple capture-recapture models permitting unequal catch ability and variable sampling effort. Biometrics 1994, 50:494-500.
92. Al-Awar J., Chapanis A., Ford R. Tutorials for the first-time computer user. IEEE Trans. Prof. Commun. 1981, 24:30-37.
93. Baecker R.M. Themes in the early history of HCI-some unanswered questions. Interactions 2008, 15(2):22-27.
94. Barnum C., Bevan N., Cockton G., Nielsen J., Spool J., Wixon D. The "Magic Number 5": Is it enough for web testing?. Proceedings of CHI 2003 2003, 698-699. ACM, Ft. Lauderdale, FL.
95. Boehm B.W. Software Engineering Economics 1981, Prentice-Hall, Englewood Cliffs, NJ.
96. Bradley J.V. Probability; Decision; Statistics 1976, Prentice-Hall, Englewood Cliffs, NJ.
97. Bradley J.V. Robustness?. Br. J. Math. Stat. Psychol. 1978, 31:144-152.
98. Briand L.C., El Emam K., Freimut B.G., Laitenberger O. A comprehensive evaluation of capture-recapture models for estimating software defect content. IEEE Trans. Softw. Eng. 2000, 26(6):518-540.
99. Burnham K.P., Overton W.S. Estimation of the size of a closed population when capture probabilities vary among animals. Biometrika 1979, 65:625-633.
100. Caulton D.A. Relaxing the homogeneity assumption in usability testing. Behav. Inf. Technol. 2001, 20:1-7.
101. Chapanis A. Some generalizations about generalization. Hum. Fact. 1988, 30:253-267.
102. Coull B.A., Agresti A. The use of mixed logit models to reflect heterogeneity in capture-recapture studies. Biometrics 1999, 55:294-301.
103. Cowles M. Statistics in Psychology: An Historical Perspective 1989, Lawrence Erlbaum, Hillsdale, NJ.
104. Dalal S.R., Mallows C.L. Some graphical aids for deciding when to stop testing software. IEEE J. Sel. Areas Commun. 1990, 8(2):169-175.
105. Dorazio R.M. On selecting a prior for the precision parameter of Dirichlet process mixture models. J. Stat. Plan. Inference 2009, 139:3384-3390.
106. Dorazio R.M., Royle J.A. Mixture models for estimating the size of a closed population when capture rates vary among individuals. Biometrics 2003, 59:351-364.
107. Dumas J. The great leap forward: The birth of the usability profession (1988-1993). J. Usability Stud. 2007, 2(2):54-60.
108. Dumas J., Sorce J., Virzi R. Expert reviews: How many experts is enough?. Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting. Human Factors and Ergonomics Society 1995, 228-232.
109. Eick S.G., Loader C.R., Vander Wiel S.A., Votta L.G. How many errors remain in a software design document after inspection?. Proceedings of the 25th Symposium on the Interface 1993, 195-202.
110. Ennis D.M., Bi J. The beta-binomial model: Accounting for inter-trial variation in replicated difference and preference tests. J. Sens. Stud. 1998, 13:389-412.
111. Faulkner L. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behav. Res. Methods Instrum. Comput. 2003, 35:379-383.
112. Gould J.D. How to design usable systems. Handbook of Human-Computer Interaction 1988, 757-789. North-Holland, Amsterdam, Netherlands. M. Helander (Ed.).
113. Gould J.D., Boies S.J. Human factors challenges in creating a principal support office system: The Speech Filing System approach. ACM Trans. Inf. Syst. 1983, 1:273-298.
114. Gould, J.D., and Lewis, C., 1984. Designing for Usability: Key Principles and What Designers Think (Tech. Report RC-10317). IBM Corporation, Yorktown Heights, NY.
115. Gould J.D., Boies S.J., Levy S., Richards J.T., Schoonard J. The 1984 Olympic Message System: A test of behavioral principles of system design. Commun. ACM 1987, 30:758-769.
116. Guest G., Bunce A., Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods 2006, 18(1):59-82.
117. Hertzum M., Jacobsen N.J. The evaluator effect: A chilling fact about usability evaluation methods. Int. J. Hum. Comput. Interact. 2001, 13:421-443. (Corrected paper published in 2003, International Journal of Human-Computer Interaction 15, 183-204.).
118. Hornbæk K. Dogmas in the assessment of usability evaluation methods. Behav. Inf. Technol. 2010, 29(1):97-111.
119. Hwang W., Salvendy G. What makes evaluators to find more usability problems? A meta-analysis for individual detection rates. Human-Computer Interaction, Part I, HCII 2007 2007, 499-507. Springer-Verlag, Heidelberg, Germany. J. Jacko (Ed.).
120. Hwang W., Salvendy G. Integration of usability evaluation studies via a novel meta-analytic approach: What are significant attributes for effective evaluation?. Int. J. Hum. Comput. Interact. 2009, 25(4):282-306.
121. Hwang W., Salvendy G. Number of people required for usability evaluation: The 10±2 rule. Commun. ACM 2010, 53(5):130-133.
122. Jelinek F. Statistical Methods for Speech Recognition 1997, MIT Press, Cambridge, MA.
123. Kanis H. Estimating the number of usability problems. Appl. Ergon. 2011, 42:337-347.
124. Kennedy P.J. Development and testing of the operator training package for a small computer system. Proceedings of the Human Factors Society 26th Annual Meeting 1982, 715-717.
125. Law E.L., Hvannberg E.T. Analysis of combinatorial user effect in international usability tests. Proceedings of CHI 2004 2004, 9-16. ACM, Vienna, Austria.
126. Lewis J.R. Testing small system customer set-up. Proceedings of the Human Factors Society 26th Annual Meeting 1982, 718-720.
127. Lewis J.R. Sample sizes for usability studies: Additional considerations. Hum. Fact. 1994, 36:368-378.
128. Lewis, J.R., 2000. Evaluation of Problem Discovery Rate Adjustment Procedures for Sample Sizes from Two to Ten (Tech. Report 29.3362). IBM Corp, Raleigh, NC. Available at (accessed October 1, 2011). http://drjim.0catch.com/pcarlo5-ral.pdf.
129. Lewis J.R. Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. Int. J. Hum. Comput. Interact. 2001, 13:445-479.
130. Lewis J.R. Effect of level of problem description on problem discovery rates: Two case studies. Proceedings of the Human Factors and Ergonomics 50th Annual Meeting 2006, 2567-2571.
131. Lewis J.R. Sample sizes for usability tests: Mostly math, not magic. Interactions 2006, 13(6):29-33. (See corrected formula in Interactions 14 (1), 4.).
132. Lewis J.R. Usability testing. Handbook of Human Factors and Ergonomics 2012, 1267-1312. John Wiley, New York. 4th ed. G. Salvendy (Ed.).
133. Lewis J.R. Usability evaluation of a speech recognition IVR. Measuring the User Experience, Chapter 10: Case Studies 2008, 244-252. Morgan-Kaufman, Amsterdam, Netherlands. T. Tullis, B. Albert (Eds.).
134. Lewis J.R., Henry S.C., Mack R.L. Integrated office software benchmarks: A case study. Proceedings of the 3rd IFIP Conference on Human-Computer Interaction-INTERACT '90 1990, 337-343. Elsevier Science Publishers, Cambridge, UK.
135. Lewis J.R., Sauro J. When 100% really isn't 100%: Improving the accuracy of small-sample estimates of completion rates. J. Usability Test. 2006, 3(1):136-150.
136. Lindgaard G., Chattratichart J. Usability testing: What have we overlooked?. Proceedings of CHI 2007 2007, 1415-1424. ACM, San Jose, CA.
137. Manning C.D., Schütze H. Foundations of Statistical Natural Language Processing 1999, MIT Press, Cambridge, MA.
138. Medlock M.C., Wixon D., McGee M., Welsh D. The rapid iterative test and evaluation method: Better products in less time. Cost-Justifying Usability: An Update for the Internet Age 2005, 489-517. Elsevier, Amsterdam, Netherlands. R.G. Bias, D.J. Mayhew (Eds.).
139. Nielsen J. Finding usability problems through heuristic evaluation. Proceedings of CHI '92 1992, 373-380. ACM, Monterey, CA.
140. Nielsen J. Why you only need to test with 5 users. Alertbox. 2000, Available at www.useit.com/alertbox/20000319.html; (accessed January 26, 2011).
141. Nielsen J., Landauer T.K. A mathematical model of the finding of usability problems. Proceedings of INTERCHI'3 1993, 206-213. ACM, Amsterdam, Netherlands.
142. Nielsen J., Molich R. Heuristic evaluation of user interfaces. Proceedings of CHI '90 1990, 249-256. ACM, New York.
143. Perfetti C., Landesman L. Eight is not enough. 2001, Available at http://www.uie.com/articles/eight_is_not_enough/ (accessed February 18, 2011).
144. Schmettow M. Heterogeneity in the usability evaluation process. Proceedings of the 22nd British HCI Group Annual Conference on HCI 2008: People and Computers XXII: Culture, Creativity, Interaction 2008, vol. 1:89-98. ACM, Liverpool, UK.
145. Schmettow M. Controlling the usability evaluation process under varying defect visibility. Proceedings of the 2009 British Computer Society Conference on Human-Computer Interaction 2009, 188-197. ACM, Cambridge, UK.
146. Schnabel Z.E. The estimation of the total fish population of a lake. Am. Math. Mon. 1938, 45:348-352.
147. Smith D.C., Irby C., Kimball R., Verplank B., Harlem E. Designing the Star user interface. Byte 1982, 7(4):242-282.
148. Spool J., Schroeder W. Testing websites: Five users is nowhere near enough. CHI 2001 Extended Abstracts 2001, 285-286. ACM, New York.
149. Turner C.W., Lewis J.R., Nielsen J. Determining usability test sample size. The International Encyclopedia of Ergonomics and Human Factors 2006, 3084-3088. CRC Press, Boca Raton, FL. W. Karwowski (Ed.).
150. Virzi R.A. Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society 34th Annual Meeting 1990, 291-294.
151. Virzi R.A. Refining the test phase of usability evaluation: How many subjects is enough?. Hum. Fact. 1992, 34:457-468.
152. Virzi R.A. Usability inspection methods. Handbook of Human-Computer Interaction 1997, 705-715. Elsevier, Amsterdam, Netherlands. second ed. M.G. Helander, T.K. Landauer, P.V. Prabhu (Eds.).
153. Walia G.S., Carver J.C. Evaluation of capture-recapture models for estimating the abundance of naturally-occurring defects. Proceedings of ESEM '08 2008, 158-167. ACM, Kaiserslautern, Germany.
154. Walia G.S., Carver J.C., Nagappan N. The effect of the number of inspectors on the defect estimates produced by capture-recapture models. Proceedings of ICSE '08 2008, 331-340. ACM, Leipzig, Germany.
155. Whiteside J., Bennett J., Holtzblatt K. Usability engineering: Our experience and evolution. Handbook of Human-Computer Interaction 1988, 791-817. North-Holland, Amsterdam, Netherlands. M. Helander (Ed.).
156. Williams G. The Lisa computer system. Byte 1983, 8(2):33-50.
157. Wilson E.B. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc. 1927, 22:209-212.
158. Woolrych A., Cockton G. Why and when five test users aren't enough. Proceedings of IHM-HCI 2001 Conference 2001, vol. 2:105-108. CépadèusÉditions, Toulouse, France. J. Vanderdonckt, A. Blandford, A. Derycke (Eds.).
159. Wright P.C., Monk A.F. A cost-effective evaluation method for use by designers. Int. J. Man Mach. Stud. 1991, 35:891-912.
160. Aladwani A.M., Palvia P.C. Developing and validating an instrument for measuring user perceived web quality. Inf. Manag. 2002, 39:467-476.
161. Albert, W., Dixon, E., 2003. Is this what you expected? The use of expectation measures in usability testing. Paper presented at the Usability Professionals Association Annual Conference. UPA, Scottsdale, AZ.
162. Anastasi A. Psychological Testing 1976, Macmillan, New York.
163. Common Industry Format for Usability Test Reports (ANSI-NCITS 354-2001) 2001, ANSI, Author, Washington, DC.
164. Azzara C.V. Questionnaire Design for Business Research 2010, Tate Publishing, Mustang, OK.
165. Bangor A., Kortum P.T., Miller J.T. An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 2008, 24:574-594.
166. Bangor A., Kortum P.T., Miller J.T. Determining what individual SUS scores mean: Adding an adjective rating scale. J. Usability Studies 2009, 4(3):114-123.
167. Bargas-Avila J.A., Lötscher J., Orsini S., Opwis K. Intranet satisfaction questionnaire: development and validation of a questionnaire to measure user satisfaction with the intranet. Comput. Hum. Behav. 2009, 25:1241-1250.
168. Barnette J.J. Effects of stem and Likert response option reversals on survey internal consistency: If you feel the need, there is a better alternative to using those negatively worded stems. Educ. Psychol. Meas. 2000, 60:361-370.
169. Benedek, J., Miner, T., 2002. Measuring desirability: New methods for evaluating desirability in a usability lab setting. Paper presented at the Usability Professionals Association Annual Conference. UPA, Orlando, FL.
170. Borsci S., Federici S., Lauriola M. On the dimensionality of the system usability scale: A test of alternative measurement models. Cogn. Process. 2009, 10:193-197.
171. Brace I. Questionnaire Design: How to Plan, Structure, and Write Survey Material for Effective Market Research 2008, Kogan Page Limited, London. 2nd ed.
172. Brooke J. SUS: A "quick and dirty" usability scale. Usability Evaluation in Industry 1996, 189-194. Taylor & Francis, London. P. Jordan, B. Thomas, B. Weerdmeester (Eds.).
173. Cavallin H., Martin W.M., Heylighen A. How relative absolute can be: SUMI and the impact of the nature of the task in measuring perceived software usability. Artif. Intell. Soc. 2007, 22:227-235.
174. Cheung G.W., Rensvold R.B. Assessing extreme and acquiescence response sets in cross-cultural research using structural equations modeling. J. Cross-Cult. Psychol. 2000, 31:187-212.
175. Chin J.P., Diehl V.A., Norman K.L. Development of an instrument measuring user satisfaction of the human-computer interface. Proceedings of CHI 1988 1988, 213-218. ACM, Washington, DC.
176. Cordes R.E. Software ease of use evaluation using magnitude estimation. Proceedings of the Human Factors Society 1984, 157-160. HFS, Santa Monica, CA.
177. Cordes R.E. Use of Magnitude Estimation for Evaluating Product Ease of Use (Tech. Report 82.0135) 1984, IBM, Tucson, AZ.
178. Cortina J.M. What is coefficient alpha? An examination of theory and applications. J. Exp. Psychol. 1993, 78(1):98-104.
179. Courage C., Baxter K Understanding Your Users: A Practical Guide to User Requirements 2005, Morgan Kaufmann, San Francisco, CA.
180. Cronbach L.J. Response sets and test validity. Educ. Psychol. Meas. 1946, 6:475-494.
181. Davis D. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989, 13(3):319-339.
182. Embretson S.E., Reise S.P. Item Response Theory for Psychologists 2000, Lawrence Erlbaum Associates, Inc, Mahwah, NJ.
183. Finstad K. The system usability scale and non-native English speakers. J. Usability Stud. 2006, 1(4):185-188.
184. Finstad K. Response interpolation and scale sensitivity: Evidence against 5-point scales. J. Usability Stud. 2010, 5(3):104-110.
185. Finstad K. The usability metric for user experience. Interact. Comput. 2010, 22:323-327.
186. Grimm S.D., Church A.T. A cross-cultural study of response biases in personality measures. J. Res. Pers. 1999, 33:415-441.
187. Hassenzahl M. The effect of perceived hedonic quality on product appealingness. Int. J. Hum. Comput. Interact. 2001, 13(4):481-499.
188. Hassenzahl M. The interplay of beauty, goodness, and usability in interactive products. Hum. Comput. Interact. 2004, 19:319-349.
189. Hassenzahl M., Platz A., Burmester M., Lehner K. Hedonic and ergonomic quality aspects determine a software's appeal. Proceedings of CHI 2000 2000, 201-208. ACM, The Hague, Amsterdam.
190. Hollemans G. User satisfaction measurement methodologies: Extending the user satisfaction questionnaire. Proceedings of HCI International 1999 1999, 1008-1012. Lawrence Erlbaum, Mahwah, NJ.
191. Hornbæk K. Current practice in measuring usability: Challenges to usability studies and research. Int. J. Hum. Comput. Stud. 2006, 64(2):79-102.
192. Hornbæk K., Law E.L. Meta-analysis of correlations among usability measures. Proceedings of CHI 2007 2007, 617-626. ACM, San Jose, CA.
193. Ibrahim A.M. Differential responding to positive and negative items: The case of a negative item in a questionnaire for course and faculty evaluation. Psychol. Rep. 2001, 88:497-500.
194. Ergonomic requirements for office work with visual display terminals (VDTs), Part 11, Guidance on usability (ISO 9241-11:1998E) 1998, ISO, Author, Geneva, Switzerland.
195. Karn K., Little A., Nelson G., Sauro J., Kirakowski J., Albert W., Norman K. Subjective ratings of usability: Reliable or ridiculous?. Panel Presentation at the Usability Professionals Association Annual Conference 2008, UPA, Baltimore, MD.
196. Keiningham T.L., Cooil B., Andreassen T.W., Aksoy L. A longitudinal examination of Net Promoter and firm revenue growth. J. Mark. 2007, 71:39-51.
197. Kirakowski J. The Software Usability Measurement Inventory: Background and usage. Usability Evaluation in Industry 1996, 169-178. Taylor & Francis, London, UK, Also available at www.ucc.ie/hfrg/questionnaires/sumi/index.html (accessed October 1, 2011). P. Jordan, B. Thomas, B. Weerdmeester (Eds.).
198. Kirakowski J., Cierlik B. Measuring the usability of websites. Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting 1998, 424-428. HFES, Santa Monica, CA, Also available at www.wammi.com (accessed October 1, 2011).
199. Kirakowski J., Corbett M. SUMI: The software usability measurement inventory. Br. J. Educ. Technol. 1993, 24:210-212.
200. Kirakowski J., Dillon A. The Computer User Satisfaction Inventory (CUSI): Manual and Scoring Key 1988, Human Factors Research Group, University College of Cork, Cork, Ireland.
201. Kuniavsky M. Observing the User Experience: A Practitioner's Guide to User Research 2003, Morgan Kaufmann, San Francisco, CA.
202. LaLomia M.J., Sidowski J.B. Measurements of computer satisfaction, literacy, and aptitudes: A review. Int. J. Hum. Comput. Interact. 1990, 2:231-253.
203. Landauer T.K. Behavioral research methods in human-computer interaction. Handbook of Human-Computer Interaction 1997, 203-227. Elsevier, Amsterdam, Netherlands. second ed. M. Helander, T. Landauer, K., P. Prabhu (Eds.).
204. Lascu D., Clow K.E. Web site interaction satisfaction, Scale development consideration. J. Internet Commer. 2008, 7(3):359-378.
205. Lewis J.R. Psychometric Evaluation of a Post-study System Usability Questionnaire: The PSSUQ (Tech. Report 54.535) 1990, International Business Machines Corp, Boca Raton, FL.
206. Lewis J.R. Psychometric Evaluation of an After-Scenario Questionnaire for Computer Usability Studies: The ASQ (Tech. Report 54.541) 1990, International Business Machines Corp, Boca Raton, FL.
207. Lewis J.R. Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ. SIGCHI Bull. 1991, 23:78-81.
208. Lewis J.R. Psychometric evaluation of the Post-Study system usability questionnaire: The PSSUQ. Proceedings of the Human Factors Society 36th Annual Meeting 1992, 1259-1263. Human Factors Society, Santa Monica, CA.
209. Lewis J.R. Multipoint scales: Mean and median differences and observed significance levels. Int. J. Hum. Comput. Interact. 1993, 5:382-392.
210. Lewis J.R. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. Int. J. Hum. Comput. Interact. 1995, 7:57-78.
211. Lewis J.R. Tradeoffs in the design of the IBM computer usability satisfaction questionnaires. Proceedings of HCI International 1999 1999, 1023-1027. Lawrence Erlbaum, Mahwah, NJ.
212. Lewis J.R. Psychometric evaluation of the PSSUQ using data from five years of usability studies. Int. J. Hum. Comput. Interact. 2002, 14:463-488.
213. Lewis J.R. Practical Speech User Interface Design 2011, Taylor & Francis, Boca Raton, FL.
214. Lewis, J.R., 2012a. Predicting Net Promoter scores from System Usability Scale scores. Available at (accessed January 4, 2012). http://www.measuringusability.com/blog/nps-sus.php.
215. Lewis J.R. Usability testing. Handbook of Human Factors and Ergonomics 2012, 1267-1312. John Wiley, New York. 4th ed. G. Salvendy (Ed.).
216. Lewis J.R., Henry S.C., Mack R.L. Integrated office software benchmarks: A case study. Proceedings of the 3rd IFIP Conference on Human-Computer Interaction, INTERACT '90 1990, 337-343. Elsevier Science, Cambridge, UK. D. Diaper (Ed.).
217. Lewis J.R., Sauro J. The factor structure of the System Usability Scale. Human Centered Design, HCII 2009 2009, 94-103. Springer-Verlag, Heidelberg, Germany. M. Kurosu (Ed.).
218. Lucey N.M. More Than Meets the I: User-Satisfaction of Computer Systems (Unpublished thesis for Diploma in Applied Psychology) 1991, University College Cork, Cork, Ireland.
219. Lund A. USE Questionnaire Resource Page Available at, (accessed April 6, 2011). http://usesurvey.com.
220. Lund A. Measuring usability with the USE questionnaire. Usability and User Exp. Newslett. STC Usability SIG 2001, 8(2):1-4. Also available at www.stcsig.org/usability/newsletter/0110_measuring_with_use.html (accessed October 1, 2011).
221. Massaro D.W. Experimental Psychology and Information Processing 1975, Rand McNally College Publishing Company, Chicago.
222. McGee M. Usability magnitude estimation. Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting 2003, 691-695. HFES, Santa Monica, CA.
223. McGee M. Master usability scaling: Magnitude estimation and master scaling applied to usability measurement. Proceedings of CHI 2004 2004, 335-342. ACM, Vienna, Austria.
224. McSweeney R. SUMI: A Psychometric Approach to Software Evaluation (unpublished M.A. (Qual.) thesis in applied psychology) 1992, University College of Cork, Cork, Ireland, Also see: http://sumi.ucc.ie/sumipapp.html (accessed October 1, 2011).
225. Molich R., Kirakowski J., Sauro J., Tullis T. Comparative Usability Task Measurement Workshop (CUE-8). Workshop conducted at the UPA 2009 Conference in Portland, OR 2009.
226. Mussen P., Rosenzweig M.R., Aronson E., Elkind D., Feshbach S., Geiwitz J., Glickman S.E., Murdock B.B., Wertheimer M., Harvey L.O. Psychology: An Introduction 1977, D. C. Heath and Company, Lexington, MA. second ed.
227. Nunnally J.C. Psychometric Theory 1978, McGraw-Hill, New York.
228. Parasuraman A. Marketing Research 1986, Addison-Wesley, Reading, MA.
229. Preston C.C., Colman A.M. Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychol. 2000, 104:1-15.
230. Quilty L.C., Oakman J.M., Risko E. Correlates of the Rosenberg Self-Esteem Scale method effects. Structu. Equ. Modeling 2006, 13(1):99-117.
231. Reichheld F.F. The one number you need to grow. Harv. Bus. Rev. 2003, 81:46-54.
232. Reichheld F. The Ultimate Question: Driving Good Profits and True Growth 2006, Harvard Business School Press, Boston.
233. Reise S.P., Ainsworth A.T., Haviland M.G. Item response theory: Fundamentals, applications, and promise in psychological research. Curr. Dir. Psychol. Sci. 2005, 14(2):95-101.
234. Safar J.A., Turner C.W. Validation of a two factor structure of system trust. Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting 2005, 497-501. HFES, Santa Monica, CA.
235. Sauro J. Does better usability increase customer loyalty? Available at, (accessed April 7, 2011). http://www.measuringusability.com/usability-loyalty.php.
236. Sauro J. If you could only ask one question, use this one Available at, (accessed March 15, 2011). http://www.measuringusability.com/blog/single-question.php.
237. Sauro J. That's the worst website ever! Effects of extreme survey items Available at, (accessed March 24, 2011). http://www.measuringusability.com/blog/extreme-items.php.
238. Sauro J. Top-box scoring of rating scale data Available at, (accessed April 7, 2011). http://www.measuringusability.com/blog/top-box.php.
239. Sauro J. A Practical Guide to the System Usability Scale (SUS): Background, Benchmarks & Best Practices 2011, Measuring Usability LLC, Denver.
240. Sauro J. The Standardized Universal Percentile Rank Questionnaire (SUPR-Q) Available at, (accessed April 2, 2011). http://www.suprq.com/.
241. Sauro J., Dumas J.S. Comparison of three one-question, post-task usability questionnaires. Proceedings of CHI 2009 2009, 1599-1608. ACM, Boston.
242. Sauro J., Lewis J.R. Correlations among prototypical usability metrics: Evidence for the construct of usability. Proceedings of CHI 2009 2009, 1609-1618. ACM, Boston.
243. Sauro J., Lewis J.R. When designing usability questionnaires, does it hurt to be positive?. Proceedings of CHI 2011 2011, 2215-2223. ACM, Vancouver, Canada.
244. Schmettow M., Vietze W. Introducing item response theory for measuring usability inspection processes. Proceedings of CHI 2008 2008, 893-902. ACM, Florence, Italy.
245. Slaughter L., Harper B., Norman K. Assessing the equivalence of the paper and on-line formats of the QUIS 5.5. Proceedings of the 2nd Annual Mid-Atlantic Human Factors Conference 1994, 87-91. HFES, Washington, DC.
246. Spector P., Van Katwyk P., Brannick M., Chen P. When two factors don't reflect two constructs: How item characteristics can produce artifactual factors. J. Manag. 1997, 23(5):659-677.
247. Tedesco D.P., Tullis T.S. A comparison of methods for eliciting post-task subjective ratings in usability testing. Paper presented at the Usability Professionals Association Annual Conference 2006, UPA, Broomfield, CO.
248. Thurstone L.L. Attitudes can be measured. Am. J. Sociol. 1928, 33:529-554.
249. Travis D. Measuring satisfaction: Beyond the usability questionnaire Available at, (accessed March 24, 2011). http://www.userfocus.co.uk/articles/satisfaction.html.
250. Tullis T., Albert B. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics 2008, Morgan Kaufmann, Burlington, MA.
251. Tullis T.S., Stetson J.N. A comparison of questionnaires for assessing website usability. Paper presented at the Usability Professionals Association Annual Conference 2004, UPA, Minneapolis, MN, Available also at home.comcast.net/~tomtullis/publications/UPA2004TullisStetson.pdf (accessed October 1, 2011).
252. van de Vijver F.J.R., Leung K. Personality in cultural context: Methodological issues. J. Pers. 2001, 69:1007-1031.
253. Wang J., Senecal S. Measuring perceived website usability. J. Internet Commer. 2007, 6(4):97-112.
254. Whiteside J., Bennett J., Holtzblatt K. Usability engineering: Our experience and evolution. Handbook of Human-Computer Interaction 1988, 791-817. North-Holland, Amsterdam, Amsterdam, Netherlands. M. Helander (Ed.).
255. Wu J., Chen Y., Lin L. Empirical evaluation of the revised end user computing acceptance model. Compu. Hum. Behav. 2007, 23:162-174.
256. Zickar M.J. Modeling item-level data with item response theory. Curr. Dir. Psychol. Sci. 1998, 7:104-109.
257. Zijlstra R., van Doorn L. The Construction of a Scale to Measure Subjective Effort (Tech. Rep.) 1985, Delft University of Technology, Department of Philosophy and Social Sciences, Delft, Netherlands.
258. Zviran M., Glezer C., Avni I. User satisfaction from commercial web sites: The effect of design and use. Inf. Manag. 2006, 43:157-178.
259. Abelson R.P. Statistics as Principled Argument 1995, Lawrence Erlbaum, Hillsdale, NJ.
260. Baguley T. Understanding statistical power in the context of applied research. Appl. Ergon. 2004, 35:73-80.
261. Box G.E.P. The importance of practice in the development of statistics. Technometrics 1984, 26(1):1-8.
262. Box J.F. Fisher: The Life of a Scientist 1978, John Wiley, New York.
263. Bradley J.V. Probability; Decision; Statistics 1976, Prentice-Hall, Englewood Cliffs, NJ.
264. Bradley J.V. Robustness?. Br. J. Math. Stat. Psychol. 1978, 31:144-152.
265. Braver S.L. On splitting the tails unequally: A new perspective on one- versus two-tailed tests. Educ. Psychol. Meas. 1975, 35:283-301.
266. Cliff N. Analyzing Multivariate Data 1987, Harcourt, Brace, Jovanovich, San Diego.
267. Cohen J. Things I have learned (so far). Am. Psychol. 1990, 45(12):1304-1312.
268. Cowles M. Statistics in Psychology: An Historical Perspective 1989, Lawrence Erlbaum, Hillsdale, NJ.
269. Diaconis P. Theories of data analysis: From magical thinking through classical statistics. Exploring Data Tables, Trends, and Shapes 1985, 1-36. John Wiley, New York. D.C. Hoaglin, F. Mosteller, J.W. Tukey (Eds.).
270. Diamond W.J. Practical Experiment Designs for Engineers and Scientists 1981, Lifetime Learning Publications, Belmont, CA.
271. Fisher R.A. The statistical method in psychical research. Proc. Soc. Psych. Res. 1929, 39:189-192.
272. Gaito J. Measurement scales and statistics: Resurgence of an old misconception. Psychol. Bull. 1980, 87(3):564-567.
273. Grove J.W. In Defence of Science: Science, Technology, and Politics in Modern Society 1989, University of Toronto Press, Toronto.
274. Harris R.J. A Primer of Multivariate Statistics 1985, Academic Press, Orlando.
275. Harris R.J. Significance tests have their place. Psychol. Sci. 1997, 8(1):8-11.
276. Hornbæk K., Law E. Meta-analysis of correlations among usability measures. Proceedings of CHI 2007 2007, 617-626. ACM, San Jose, CA.
277. Lewis J.R. A rank-based method for the usability comparison of competing products. Proceedings of the Human Factors Society 35th Annual Meeting 1991, 1312-1316. HFS, Santa Monica, CA.
278. Lewis J.R. Multipoint scales: Mean and median differences and observed significance levels. Int. J. Hum. Comput. Interact. 1993, 5(4):383-392.
279. Lewis J.R., Henry S.C., Mack R.L. Integrated office software benchmarks: A case study. Proceedings of the 3rd IFIP Conference on Human-Computer Interaction, INTERACT '90 1990, 337-343. Elsevier Science, Cambridge, UK. D. Diaper (Ed.).
280. Lord F.M. On the statistical treatment of football numbers. Am. Psychol. 1953, 8:750-751.
281. Lord F.M. Further comment on "football numbers.". Am. Psychol. 1954, 9(6):264-265.
282. Mitchell J. Measurement scales and statistics: A clash of paradigms. Psychol. Bull. 1986, 100:398-407.
283. Myers J.L. Fundamentals of Experimental Design 1979, Allyn and Bacon, Boston. 3rd ed.
284. Nunnally J.C. Psychometric Theory 1978, McGraw-Hill, New York.
285. Perneger T.V. What's wrong with Bonferroni adjustments?. Br. Med. J. 1998, 316:1236-1238.
286. Rosnow R.L., Rosenthal R. Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 1989, 44:1276-1284.
287. Salsburg D. The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century 2001, W. H. Freeman, New York.
288. Sauro J., Kindlund E. A method to standardize usability metrics into a single score. Proceedings of CHI 2005 2005, 401-409. ACM, Portland.
289. Sauro J., Lewis J.R. Correlations among prototypical usability metrics: Evidence for the construct of usability. Proceedings of CHI 2009 2009, 1609-1618. ACM, Boston.
290. Scholten A.Z., Borsboom D. A reanalysis of Lord's statistical treatment of football numbers. J. Math. Psychol. 2009, 53(2):69-75.
291. Stevens S.S. On the theory of scales of measurement. Science 1946, 103(2684):677-680.
292. Stevens S.S. Measurement, psychophysics, and utility. Measurement: Definitions and Theories 1959, 18-82. John Wiley, New York. C.W. Churchman, P. Ratoosh (Eds.).
293. Stigler S.M. The History of Statistics: The Measurement of Uncertainty before 1900 1986, Harvard University Press, Cambridge, MA.
294. Stigler S.M. Statistics on the Table: The History of Statistical Concepts and Methods 1999, Harvard University Press, Cambridge, MA.
295. Townsend J.T., Ashby F.G. Measurement scales and statistics: the misconception misconceived. Psychol. Bull. 1984, 96(2):394-401.
296. Velleman P.F., Wilkinson L. Nominal, ordinal, interval, and ratio typologies are misleading. Am. Stat. 1993, 47(1):65-72.
297. Wickens C.D. Commonsense statistics. Ergon. Des. 1998, 6(4):18-22.
298. Winer B.J., Brown D.R., Michels K.M. Statistical Principles in Experimental Design 1991, McGraw-Hill, New York. 3rd ed.
299. Lewis, J.R., Sauro, J., 2012. Excel and R Companion to "Quantifying the User Experience: Practical Statistics for User Research": Rapid Answers to over 100 Examples and Exercises. Create Space Publishers, Denver.
300. Sauro J. A Practical Guide to Measuring Usability 2010, Create Space Publishers, Denver.
301. Sauro J. A Practical Guide to the System Usability Scale 2011, Create Space Publishers, Denver.
302. Searles D. PSI burn: A study of physiological deterioration in parapsychological experimentation. Omni Mag. 1978, 1(3):108-110. [A satire with amusing definitions of statistical terms.].
303. Tullis T., Albert B. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics 2008, Morgan Kaufmann, Boston.
304. Vickers A. What Is a p-Value Anyway? 2010, Addison-Wesley, Boston.