The System Usability Scale: Past, Present, and Future

International Journal of Human-Computer Interaction

Volumn 34, Issue 7, 2018, Pages 577-590

  Lewis, James R ^a  

^a IBM   (United States)

Author keywords

Perceived usability; standardized usability scale; SUS; System Usability

Indexed keywords

HUMAN COMPUTER INTERACTION; USER INTERFACES;

FUTURE PROSPECTS; PERCEIVED USABILITY; RECENT RESEARCHES; STANDARDIZED USABILITY SCALE; SYSTEM USABILITY; SYSTEM USABILITY SCALE (SUS);

USABILITY ENGINEERING;

EID: 85044620959     PISSN: 10447318     EISSN: 15327590     Source Type: Journal    
DOI: 10.1080/10447318.2018.1455307     Document Type: Article

References (81)

1. AlGhannam, B. A., Albustan, S. A., Al-Hassan, A. A., & Albustan, L. A., (2017). Towards a standard Arabic System Usability Scale (A-SUS): Psychometric evaluation using communication disorder app. International Journal of Human-Computer Interaction. doi:10.1080/10447318.2017.1388099
2. Anastasi, A., (1976). Psychological testing. New York, NY: Macmillan.
3. Bangor, A., Joseph, K., Sweeney-Dillon, M., Stettler, G., & Pratt, J., (2013). Using the SUS to help demonstrate usability’s value to business goals. In Proceedings of the Human Factors and Ergonomics Society 57th Annual Meeting (pp. 202–205). Santa Monica, CA: HFES.
4. Bangor, A., Kortum, P. T., & Miller, J. T., (2008). An empirical evaluation of the System Usability Scale. International Journal of Human–Computer Interaction, 24, 574–594.
5. Bangor, A., Kortum, P. T., & Miller, J. T., (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114–123.
6. Barnette, J. J., (2000). Effects of stem and Likert response reversals on survey internal consistency: If you feel the need, there is a better alternative to using those negatively worded stems. Educational and Psychological Measurement, 60(3), 361–370.
7. Benedek, J., & Miner, T., (2002). Measuring desirability: New methods for evaluating desirability in a usability lab setting. Paper presented at the Usability Professionals Association Annual Conference, UPA, Orlando, FL.
8. Berkman, M. I., & Karahoca, D., (2016). Re-assessing the Usability Metric for User Experience (UMUX) Scale. Journal of Usability Studies, 11(3), 89–109.
9. Blažica, B., & Lewis, J. R., (2015). A Slovene translation of the System Usability Scale (SUS). International Journal of Human-Computer Interaction, 31, 112–117.
10. Borkowska, A., & Jach, K., (2016). Pre-testing of polish translation of System Usability Scale (SUS). In J. Świątek, Z. Wilimowska, L. Borzemski, & A. Grzech (Eds.), Proceedings of 37th International Conference on Information Systems Architecture and Technology–ISAT 2016–Part I (pp. 143–153). New York, NY: Springer.
11. Borsci, S., Federici, S., Gnaldi, M., Bacci, S., & Bartolucci, F., (2015). Assessing user satisfaction in the era of user experience: An exploratory analysis of SUS, UMUX and UMUX-LITE. International Journal of Human-Computer Interaction., 31, 484–495.
12. Borsci, S., Federici, S., & Lauriola, M., (2009). On the dimensionality of the System Usability Scale: A test of alternative measurement models. Cognitive Processes, 10, 193–197.
13. Borsci, S., Federici, S., Mele, M. L., & Conti, M., (2015). Short scales of satisfaction assessment: A proxy to involve disabled users. In M. Kurosu (Ed.), Proceedings of HCII 2015 (pp. 35–42). Los Angeles, CA: Springer.
14. Brooke, J., (1996). SUS: A “quick and dirty” usability scale. In P., Jordan, B., Thomas, & B., Weerdmeester (Eds.), Usability evaluation in industry (pp. 189–194). London, UK: Taylor & Francis.
15. Brooke, J., (2013). SUS: A retrospective. Journal of Usability Studies, 8(2), 29–40.
16. Chin, J. P., Diehl, V. A., & Norman, K. L., (1988). Development of an instrument measuring user satisfaction of the human–computer interface. In E. Soloway, D. Frye, & S. B. Sheppard (Eds.), Proceedings of CHI 1988 (pp. 213–218). Washington, DC: Association for Computing Machinery.
17. Cliff, N., (1987). Analyzing multivariate data. San Diego, California: Harcourt Brace Jovanovich.
18. Coovert, M. D., & McNelis, K., (1988). Determining the number of common factors in factor analysis: A review and program. Educational and Psychological Measurement, 48, 687–693.
19. Cortina, J. M., (1993). What is coefficient alpha? An examination of theory and applications. Journal of Experimental Psychology, 78(1), 98–104.
20. Davis, D., (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 319–339.
21. Dianat, I., Ghanbari, Z., & AsghariJafarabadi, M., (2014). Psychometric properties of the Persian language version of the System Usability Scale. Health Promotion Perspectives, 4(1), 82–89.
22. Diefenbach, S., Kolb, N., & Hassenzahl, M., (2014). The “Hedonic” in human-computer interaction: history, contributions, and future research directions. In Proceedings of the 2014 Conference on Designing Interactive Systems–DIS 14 (pp. 305–314). New York, NY: Association for Computing Machinery.
23. Embretson, S. E., & Reise, S. P., (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
24. Finstad, K., (2006). The System Usability Scale and non-native English speakers. Journal of Usability Studies, 1(4), 185–188.
25. Finstad, K., (2010). The usability metric for user experience. Interacting with Computers, 22, 323–327.
26. Göransson, B., (2011). SUS Svensk: System Usability Scale in Swedish. Retrieved from http://rosenfeldmedia.com/surveys-that-work/sus-svensk-system-usability-sc/
27. Grier, R. A., (2013). The potential utility of the System Usability Scale in U.S. military acquisition. In Proceedings of the Human Factors and Ergonomics Society 57th Annual Meeting (pp. 206–209). Santa Monica, CA: Human Factors and Ergonomics Society.
28. Grier, R. A., Bangor, A., Kortum, P. T., & Peres, S. C., (2013). The System Usability Scale: Beyond standard usability testing. In Proceedings of the Human Factors and Ergonomics Society 57th Annual Meeting (pp. 187–191). Santa Monica, CA: Human Factors and Ergonomics Society.
29. ISO. (1998). Ergonomic requirements for office work with visual display terminals (VDTs), Part 11, Guidance on usability (ISO 9241-11:1998E). Geneva, Switzerland: Author.
30. Kirakowski, J., & Cierlik, B., (1998). Measuring the usability of websites. In Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting (pp. 424–428). Santa Monica, CA: HFES.
31. Kirakowski, J., & Corbett, M., (1993). SUMI: The software usability measurement inventory. British Journal of Educational Technology, 24, 210–212.
32. Kortum, P., & Acemyan, C. Z., (2013). How low can you go? Is the System Usability Scale range restricted? Journal of Usability Studies, 9(1), 14–24.
33. Kortum, P., & Acemyan, C. Z., (2018). The impact of geographic location on the subjective assessment of system usability. International Journal of Human-Computer Interaction. doi:10.1080/10447318.2018.1437865
34. Kortum, P., & Bangor, A., (2013). Usability ratings for everyday products measured with the System Usability Scale. International Journal of Human-Computer Interaction, 29, 67–76.
35. Kortum, P., & Johnson, M., (2013). The relationship between levels of user experience with a product and perceived system usability. In Proceedings of the Human Factors and Ergonomics Society 57th Annual Meeting (pp. 197–201). Santa Monica, CA: Human Factors and Ergonomics Society.
36. Kortum, P., & Oswald, F. L., (2017). The impact of personality on the subjective assessment of usability. International Journal of Human-Computer Interaction, 34(2), 177–186.
37. Kortum, P., & Sorber, M., (2015). Measuring the usability of mobile applications for phones and tablets. International Journal of Human-Computer Interaction, 31(8), 518–529.
38. Lah, U., & Lewis, J. R., (2016). How expertise affects a digital-rights-management sharing application’s usability. IEEE Software, 33(3), 76–82.
39. LaLomia, M. J., & Sidowski, J. B., (1990). Measurements of computer satisfaction, literacy, and aptitudes: A review. International Journal of Human–Computer Interaction, 2, 231–253.
40. Landauer, T. K., (1997). Behavioral research methods in human-computer interaction. In M., Helander, T. K., Landauer, & P., Prabhu (Eds.), Handbook of human-computer interaction (2nd ed., pp. 203–227). Amsterdam, Netherlands: Elsevier.
41. Lewis, J. R., (1990). Psychometric evaluation of a Post-Study System Usability Questionnaire: The PSSUQ (Tech. Report 54.535). Boca Raton, FL: International Business Machines Corp.
42. Lewis, J. R., (1992). Psychometric evaluation of the Post-Study System Usability Questionnaire: The PSSUQ. In Proceedings of the Human Factors Society 36th Annual Meeting (pp. 1259–1263). Santa Monica, CA: Human Factors Society.
43. Lewis, J. R., (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7, 57–78.
44. Lewis, J. R., (2002). Psychometric evaluation of the PSSUQ using data from five years of usability studies. International Journal of Human-Computer Interaction, 14, 463–488.
45. Lewis, J. R., (2018). Measuring perceived usability: The CSUQ, SUS, and UMUX. International Journal of Human-Computer Interaction. doi:10.1080/10447318.2017.1418805
46. Lewis, J. R., Brown, J., & Mayes, D. K., (2015). Psychometric evaluation of the EMO and the SUS in the context of a large-sample unmoderated usability study. International Journal of Human-Computer Interaction, 31(8), 545–553.
47. Lewis, J. R., & Mayes, D. K., (2014). Development and psychometric evaluation of the Emotional Metric Outcomes (EMO) questionnaire. International Journal of Human-Computer Interaction, 30(9), 685–702.
48. Lewis, J. R., & Sauro, J., (2009). The factor structure of the System Usability Scale. In M., Kurosu (Ed.), Human centered design (pp. 94–103). Heidelberg, Germany: Springer-Verlag.
49. Lewis, J. R., & Sauro, J., (2017a). Can I leave this one out? The effect of dropping an item from the SUS. Journal of Usability Studies, 13(1), 38–46.
50. Lewis, J. R., & Sauro, J., (2017b). Revisiting the factor structure of the System Usability Scale. Journal of Usability Studies, 12(4), 183–192.
51. Lewis, J. R., & Sauro, J., (In press). Item benchmarks for the System Usability Scale. To Appear in Journal of Usability Studies, 13(3).
52. Lewis, J. R., Utesch, B. S., & Maher, D. E., (2013). UMUX-LITE—When there’s no time for the SUS. In Proceedings of CHI 2013 (pp. 2099–2102). Paris, France: Association for Computing Machinery.
53. Lewis, J. R., Utesch, B. S., & Maher, D. E., (2015). Measuring perceived usability: The SUS, UMUX-LITE, and AltUsability. International Journal of Human-Computer Interaction, 31(8), 496–505.
54. Lindgaard, G., & Kirakowski, J., (2013). Introduction to the special issue: The tricky landscape of developing rating scales in HCI. Interacting with Computers, 25, 271–277.
55. Lucey, N. M., (1991). More than meets the I: User-satisfaction of computer systems (Unpublished thesis for Diploma in Applied Psychology). University College Cork, Cork, Ireland.
56. Martinsa, A. I., Rosa, A. F., Queirós, A., & Silva, A., (2015). European Portuguese validation of the System Usability Scale. Procedia Computer Science, 67, 293–300.
57. McLellan, S., Muddimer, A., & Peres, S. C., (2012). The effect of experience on System Usability Scale ratings. Journal of Usability Studies, 7(2), 56–67.
58. McSweeney, R., (1992). SUMI: A psychometric approach to software evaluation (Unpublished M.A. (Qual.) thesis in applied psychology). University College of Cork, Cork, Ireland.
59. Nunnally, J. C., (1978). Psychometric theory. New York, NY: McGraw-Hill.
60. O’Connor, B. P., (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instrumentation, and Computers, 32, 396–402.
61. Peres, S. C., Pham, T., & Phillips, R., (2013). Validation of the System Usability Scale (SUS): SUS in the wild. In Proceedings of the Human Factors and Ergonomics Society 57th Meeting (pp. 192–196). Santa Monica, CA: HFES.
62. Pilotte, W. J., & Gable, R. K., (1990). The impact of positive and negative item stems on the validity of a Computer Anxiety Scale. Educational and Psychological Measurement, 50, 603–610.
63. Reichheld, F., (2003). The one number you need to grow. Harvard Business Review, 81, 46–54.
64. Reichheld, F., (2006). The ultimate question: Driving good profits and true growth. Boston, MA: Harvard Business School Press.
65. Rummel, B., (2015). System Usability Scale–Jetzt auch auf Deutsch. Retrieved November 27, 2017, from https://experience.sap.com/skillup/system-usability-scale-jetzt-auch-auf-deutsch/
66. Sauro, J., (2010). That’s the worst website ever! Effects of extreme survey items. Retrieved March 24, 2011, from www.measuringu.com/blog/extreme-items.php
67. Sauro, J., (2011). A practical guide to the System Usability Scale (SUS): Background, benchmarks & best practices. Denver, CO: Measuring Usability LLC.
68. Sauro, J., (2015). SUPR-Q: A comprehensive measure of the quality of the website user experience. Journal of Usability Studies, 10(2), 68–86.
69. Sauro, J., & Lewis, J. R., (2009). Correlations among prototypical usability metrics: Evidence for the construct of usability. In Proceedings of CHI 2009 (pp. 1609–1618). Boston, MA: Association for Computing Machinery.
70. Sauro, J., & Lewis, J. R., (2011). When designing usability questionnaires, does it hurt to be positive? In Proceedings of CHI 2011 (pp. 2215–2223). Vancouver, Canada: ACM.
71. Sauro, J., & Lewis, J. R., (2012). Quantifying the user experience: Practical statistics for user research (1st ed.). Waltham, MA: Morgan Kaufmann.
72. Sauro, J., & Lewis, J. R., (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Cambridge, MA: Morgan Kaufmann.
73. Schmitt, N., (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8(4), 350–353.
74. Schmitt, N., & Stuits, D., (1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9(4), 367–373.
75. Schriesheim, C. A., & Hill, K. D., (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41(4), 1101–1114.
76. Sijtsma, K., (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107–120.
77. Stewart, T. J., & Frye, A. W., (2004). Investigating the use of negatively-phrased survey items in medical education settings: Common wisdom or common mistake? Academic Medicine, 79(10 Supplement), S1–S3.
78. Tossell, C. C., Kortum, P., Shepard, C., Rahmati, A., & Zhong, L., (2012). An empirical analysis of smartphone personalization: Measurement and user variability. Behaviour & Information Technology, 31(10), 995–1010.
79. Tullis, T. S., & Albert, B., (2008). Measuring the user experience: Collecting, analyzing, and presenting usability metrics. Burlington, MA: Morgan Kaufmann.
80. Tullis, T. S., & Stetson, J. N., (2004). A comparison of questionnaires for assessing website usability. Paper presented at the Usability Professionals Association Annual Conference. UPA, Minneapolis, MN. Retrieved September13, 2017, from https://www.researchgate.net/publication/228609327_A_Comparison_of_Questionnaires_for_Assessing_Website_Usability
81. Wong, N., Rindfleisch, A., & Burroughs, J., (2003). Do reverse-worded items confound measures in cross-cultural consumer research? The case of the material values scale. Journal of Consumer Research, 30, 72–91.