Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire

Assessment and Evaluation in Higher Education

Volumn 35, Issue 1, 2010, Pages 113-130

  Roszkowski, Michael J ^a   Soven, Margot ^a  

^a LA SALLE UNIVERSITY   (United States)

Author keywords

Artefactual factor; Cronbach's alpha; Likert scale; Negative phrasing; Survey

Indexed keywords

EID: 79951592168     PISSN: 02602938     EISSN: 1469297X     Source Type: Journal    
DOI: 10.1080/02602930802618344     Document Type: Article

References (84)

1. Ahlawat, K.S. 1985. On the negative valence items in self-report measures. Journal of General Psychology 112, no. 1: 89-99.
2. Barnette, J.J. 1996. Responses that may indicate nonattending behaviors in three selfadministered educational surveys. Research in the Schools 3, no. 2: 49-59.
3. Barnette, J.J. 1999. Nonattending respondent effects on internal consistency of selfadministered surveys: A Monte Carlo simulation study. Educational and Psychological Measurement 59, no. 1: 38-46.
4. Barnette, J.J. 2000. Effects of stem and Likert response option reversals on survey internal consistency: If you feel the need, there is a better alternative to using those negatively worded stems. Educational and Psychological Measurement 60, no. 3: 361-70.
5. Barnette, J.J. 2001. Likert survey primacy effect in the absence or presence of negatively worded items. Research in the Schools 8, no. 1: 77-82.
6. Bergstrom, B.A., and M.E. Lunz. 1998. Rating scale analysis: Gauging the impact of positively and negatively worded items. Paper presented at the annual meeting of the American Educational Research Association. San Diego, CA (ERIC Document Reproduction Service No. ED 423 289).
7. Bolin, B. L., and R.A. Dodder. 1990. The affect balance scale in an American college population. The Journal of Social Psychology 130, no. 6: 839-40.
8. Bors, D.A., F. Vigneau, and F. Lalande. 2006. Measuring the need for cognition: Item polarity, dimensionality, and the relation with ability. Personality and Individual Differences 40, no. 4: 819-28.
9. Burke, B.G. 1999. Item reversals and response validity in the Job Diagnostic Survey. Psychological Reports 85, no. 1: 213-19.
10. Chang, L. 1995. Connotatively consistent and reversed connotatively inconsistent items are not fully equivalent: Generalizability study. Educational and Psychological Measurement 55, no. 6: 991-7.
11. Colston, H.L. 1999. Not good is 'bad', but 'not bad' is not 'good': An analysis of three accounts of negation asymmetry. Discourse Processes 28, no. 3: 237-56.
12. Cordery, J.L., and P.P. Sevastos. 1993. Responses to the original and revised job diagnostic survey: Is education a factor in responses to negatively worded items? Journal of Applied Psychology 78, no. 1: 141-3.
13. Cortina, J.M. 1993. What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology 78, no. 1: 98-104.
14. Cronbach, L.J. 1950. Further evidence on response sets and test design. Educational and Psychological Measurement 10, no. 1: 3-31.
15. Cronbach, L.J. 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16, no. 3: 297-334.
16. Fabian, E.S., and C. Waugh. 2001. A job development efficiency scale for rehabilitation professionals. Journal of Rehabilitation 67, no. 2: 42-7.
17. Feldt, L.S., and R.A. Charter. 2006. Averaging internal consistency reliability coefficients. Educational and Psychological Measurement 66, no. 2: 215-27.
18. Finney, S.J. 2001. A comparison of the psychometric properties of negatively and positively worded questionnaire items. Dissertation Abstracts International: Section B: The Sciences and Engineering 62, no. 3-B: 1637.
19. Fisher, D.G., A. Rainof, J. Rodríguez, E. Archuleta, and J.F. Muñiz. 2002. Translation problems of the Spanish version of the readiness to change questionnaire. Alcohol and Alcoholism 37, no. 1: 100.
20. FitzGerald, J. 2004. The value of negativity. http://www.coolth.com/negitems.htm (accessed November 8, 2004).
21. Fried, Y., and G.R. Ferris. 1986. The dimensionality of job characteristics: Some neglected issues. Journal of Applied Psychology 71, no. 3: 419-26.
22. Friedman, H.H., 1988. The effects of positive and negative wording on responses to a Likert scale. Applied Marketing Research 28: 17-22.
23. Friedman, H.H., and T. Amoo. 1999. Rating the rating scales. Journal of Marketing Management 9, no. 3: 114-23.
24. Garg, R.K. 1996. The influence of positive and negative wording and issue involvement on responses to Likert scales in marketing research. Journal of the Market Research Society 38, no. 3: 235-46.
25. Gasperoni, G., and F. Giovani. 1992. Come e perché non funzionano le scale Likert con items a polarità semantica invertita. In Cosruire il dato 2, ed. A. Morradi and G. Gasperoni, 60-99. Milan: Franco Angeli.
26. Gomleksiz, M.N. 2004. Use of education technology in English classes. The Turkish Online Journal of Educational Technology - TOJET 3, no. 2, Article 11. http://www.tojet.net/articles/3211.htm (accessed November 4, 2004).
27. Green, S.B., and T.M. Stutzman. 1986. An evaluation of methods to select respondents to structured job-analysis questionnaires. Personnel Psychology 39, no. 3: 543-64.
28. Greenberger, E., C. Chen, J. Dmitrieva, and S.P. Farruggia. 2003. Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: Do they matter? Personality and Individual Differences 35, no. 6: 1241-54.
29. Guyatt, G.H., D.J. Cook, D. King, G.R. Norman, S.C. Kane, and C. van Ineveld. 1999. Effect of the framing of questionnaire items regarding satisfaction with training on residents' responses, Academic Medicine 74, no. 2: 192-4.
30. Han, H.R., M.T. Kim, and C. Weinert. 2002. The psychometric evaluation of Korean translation of the Personal Resource Questionnaire 85 - Part 2. Nursing Research 51, no. 5: 309-16.
31. Henrysson, S. 1963. Correction of item-total correlations in item analysis. Psychometrika 28, no. 2: 211-18.
32. Holden, R.R., and G.C. Fekken. 1990. Structured psychopathological test item characteristics and validity. Psychological Assessment 2, no. 1: 35-40.
33. Horan, P.M., C. Distefano, and R.W. Motl. 2003. Wording effects in self-esteem scales: Methodological artifact or response style? Structural Equation Modeling: A Multidisciplinary Journal 10, no. 3: 435-55.
34. Ibrahim, A.M. 2001. Differential responding to positive and negative items: The case of a negative item in a questionnaire for course and faculty evaluation. Psychological Reports 88, no. 2: 497-500.
35. Idaszak, J.R., and F. Drasgow. 1987. A revision of the Job Diagnostic Survey: Elimination of a measurement artefact. Journal of Applied Psychology 72, no. 1: 69-74.
36. Johnson, J.M., D.N. Bristow, and K.C. Schneider. 2004. Did you not understand the question or not? An investigation of negatively worded questions in survey research. Journal of Applied Business Research 20, no. 1: 75-86.
37. Kelloway, E.K., V.M. Catano, and R.R. Southwell. 1992. The construct validity of union commitment: Development and dimensionality of a shorter scale. Journal of Occupational and Organizational Psychology 65, no. 3: 197-211.
38. Kennedy, J.C. 2008. The effects of careless responding in a CFA framework. Paper presented at the 23rd annual conference of the Society for Industrial and Organizational Psychology (SIOP), April 27-29, in San Francisco, CA.
39. Krosnick, J.A. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5, no. 3: 213-36.
40. Jessup, K.O. 2001. The psychometric efficacy of positively versus negatively worded items on a self-report measure of middle schoolers' work ethic. Dissertation Abstracts International Section A: Humanities and Social Science 61, no. 9-A: 3465.
41. Lageson, C. 2003. Connecting caring communities Milwaukee Aging Consortium 2003: Caregiver survey final report. http://www.uwm.edu/Org/milwaging/files/caregiverretentionfinalreport.doc . (accessed November 5, 2004).
42. Lange, J.W. 2002. Testing equivalence of Spanish and English versions: The LaMonica- Oberst (Revised) patient satisfaction with nursing care scale. Research in Nursing and Health 25, no. 6: 438-51.
43. Lent, L., E. Hahn, S. Eremenco, K. Webster, and D. Cella. 1999. Using cross-cultural input to adapt the Functional Assessment of Chronic Illness Therapy (FACIT) Scales, Acta Oncologica 38, no. 6: 695-702.
44. Likert, R. 1932. A technique for the measurement of attitudes. Archives of Psychology 22, no. 140: 5.
45. Magazine, S.L., L.J. Williams, and Williams, M.L. 1996. A confirmatory factor analysis examination of reverse coding effects in Meyer and Allen's affective and continuance commitment scales. Educational and Psychological Measurement 56, no. 2: 241-50.
46. Marsh, H.W. 1984. The bias of negatively-worded items in ratings scales for preadolescent children: A cognitive-developmental phenomenon (ERIC Document Reproduction Service No. ED 242 772).
47. Marsh, H.W. 1996. Positive and negative global self-esteem: A substantively meaningful distinction or artifacts? Journal of Personality and Social Psychology 70, no. 4: 810-19.
48. McGee, G.W., C.E. Ferguson, Jr., and A. Seers. 1989. Role conflict and role ambiguity: Do the scales measure these two constructs? Journal of Applied Psychology 74, no. 5: 815-18.
49. Melnick, S.A., and R.K. Gable. 1990. The use of negative item stems: A cautionary note. Educational Research Quarterly 14, no. 3: 31-6.
50. Meloni, F., and K. Gana. 2001. Assessment wording effects in the Italian version of the Penn State Worry Questionnaire. Clinical Psychology and Psychotherapy 8, no. 4: 282-7.
51. Mook, J., W.C. Kleijn, and H.M. van der Ploeg. 1991. Symptom-positively and -negatively worded items in two popular self-report inventories of anxiety and depression. Psychological Reports 69, no. 2: 551-60.
52. Mook, J., W.C. Kleijn, and H.M. van der Ploeg. 1992. Positively and negatively worded items in a self-report measure of dispositional optimism. Psychological Reports 71, no. 1: 275-8.
53. Morrel-Samuels, P. 2002. Getting the truth into workplace surveys. Harvard Business Review 80, no. 2: 111-18.
54. Motl, R.W., D.E Conroy, and P.M. Horan. 2000. The social physique anxiety scale: An example of the potential consequence of negatively-worded items in factorial validity studies. Journal of Applied Measurement 1, no. 4: 327-45.
55. Mowday, R.T., R.M. Steers, and L.W. Porter. 1979. The measurement of organizational commitment. Journal of Vocational Behavior 14, no. 2: 224-47.
56. Narayan, S.S. 1996. Satisficing in attitude surveys: The impact of cognitive skills, motivation, and task difficulty on response effects. Dissertation Abstracts International: Section B: The Sciences and Engineering, 56, no. (9-B): 5219.
57. Nielson, E.R. 2002. Investigating the effects of combining positively and negatively oriented items on the dimensionality of Likert scales. Dissertation Abstracts International: Section B: The Sciences and Engineering 63, no. 6-B: 3062.
58. Nunnally, J.C. 1967. Psychometric theory. New York: McGraw-Hill.
59. Nunnally, J., and I.H. Bernstein. 1994. Psychometric theory, 3rd ed. New York: McGraw-Hill.
60. Parasuraman, A., V.A. Zeithaml, and L.L. Berry. 1988. SERVQUAL: A multiple-item scale for measuring consumer perceptions of service quality. Journal of Retailing 64, no. 1: 12-40.
61. Pilotte, W.J., and R.K. Gable. 1990. The impact of positive and negative item stems on the validity of a computer anxiety scale. Educational and Psychological Measurement 50, no. 3: 603-10.
62. Podsakoff, P.M., S.B. MacKenzie, J.-Y. Lee, and N.P. Podsakoff. 2003. Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology 88, no. 5: 879-903.
63. Ray, J.J. 1990. Acquiescence and problems with forced-choice scales. Journal of Social Psychology 130, no. 3: 397-9.
64. Roberts, R.E., P.M. Lewinsohn, and J.R. Seeley. 1993. A brief measure of loneliness suitable for use with adolescents. Psychological Reports 72, no. 3: 1379-91.
65. Robertson, S.I. 2004. Student perceptions of student perception of module questionnaires: Questionnaire completion as problem solving. Assessment & Evaluation in Higher Education 29, no. 6: 663-79.
66. Sandoval, J., and N.M. Lambert. 1978. Reliability and validity of teacher rating procedures in the assessment of hyperactivity as a function of rating scale format. Paper presented at the annual meeting of the American Educational Research Association, March, in Toronto, Canada (ERIC Document Reproduction Service No. ED 160 614).
67. Schmitt, N. 1996. Uses and abuses of Coefficient Alpha. Psychological Assessment 8, no. 4: 350-3.
68. Schmitt, N., and D.M. Stults. 1985. Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement 9, no. 4: 367-73.
69. Schriesheim, C.A., R.J. Eisenbach, and K.D. Hill. 1991. The effect of negation and polar opposite item reversals on questionnaire reliability and validity: An experimental investigation. Educational and Psychological Measurement 51, no. 1: 67-78.
70. Schriesheim, C.A., and R.J. Eisenbach. 1995. An exploratory and confirmatory factor-analytic investigation of item wording effects on the obtained factor structures of survey questionnaire measures. Journal of Management 21, no. 6: 1177-93.
71. Schriesheim, C.A., and K.D. Hill. 1981. Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement 41, no. 4: 1101-14.
72. Seright, O.D. 1966. Double negatives in standard modern English. American Speech 41, no. 2: 123-6.
73. Spector, P.E., P.T. van Katwyk, M.T. Brannick, and P.Y. Chen. 1997. When two factors don't reflect two constructs: How item characteristics can produce artifactual factors. Journal of Management 23, no. 5: 659-77.
74. Stevens, J., C.E. Cornell, M. Story, S.A. French, S. Levin, A. Becenti, J. Gittelsohn, S.B. Going, and R. Reid. 1999. Development of a questionnaire to assess knowledge, attitudes, and behaviors in American Indian children. American Journal of Clinical Nutrition 69, no. 4: 773-81.
75. Stone, E.F., D.L. Stone, and H.G. Guet-aletal/et-al. 1990. Influence of cognitive ability on responses to questionnaire measures: Measurement precision and missing response problems. Journal of Applied Psychology 75, no. 4: 418-27.
76. Sudman, S., N.M. Bradburn, and N. Schwarz. 1996. Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass.
77. Thumboo, J., P.-H. Feng, M.-L. Boey, C.-H. Soh, S.-T. Thio, and K.-Y. Fong. 2000. Validation of the Chinese SF-36 for quality of life assessment in patients with systemic lupus erythematosus. Lupus 9, no. 9: 708-12.
78. Weems, G.H., A.J. Onwuegbuzie, and D. Lustig. 2003. Profiles of respondents who respond inconsistently to positively- and negatively-worded items on rating scales. Evaluation and Research in Education 17, no. 1: 45-60.
79. Weems, G.H., A.J. Onwuegbuzie, J.B. Schreiber, and S.J. Eggers. 2003. Characteristics of respondents who respond differently to positively and negatively worded items on rating scales. Assessment and Evaluation in Higher Education 28, no. 6: 587-607.
80. Williams, R.L., V.J. Bush, S.H. Park, Y. Malone, and K. Jessup. 2001. Work ethic scale for middle school students. (ERIC Document Reproduction Service No. ED 452 257).
81. Wong, N., A. Rindfleisch, and J.E. Burroughs. 2003. Do reverse-worded items confound measures in cross-cultural consumer research? The case of the Material Values Scale. Journal of Consumer Research 30, no. 1: 72-91.
82. Woods, C.M. 2006. Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment 28, no. 3: 186-91.
83. Worcester, R.M., and T.R. Burns. 1975. A statistical examination of the relative precision of verbal scales. Journal of the Market Research Society 17, no. 3: 181-97.
84. Zhai, M. 2003. Discussion thread: How to handle extremely skewed results from a survey. http://www.ocair.org/files/KnowledgeBase/Statistics/skewedDistribution.a sp (accessed November 9, 2004).