Effects of semantic incompatibility on rating response

Applied Psychological Measurement

Volumn 32, Issue 3, 2008, Pages 248-260

  Lam, Tony C M ^a   Kolic, Mary ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Attitude scaling; Bias in rating; Rating scale construction; Self reporting; Survey research

Indexed keywords

EID: 41549101468     PISSN: 01466216     EISSN: 15523497     Source Type: Journal    
DOI: 10.1177/0146621607301094     Document Type: Article

References (58)

1. Bandalos, D.L., & Enders, C.K. (1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education, 9, 151-160.
2. Bendig, A.W. (1953). The reliability of self-ratings as a function of the amount of verbal anchoring and of the number of categories on the scale. Journal of Applied Psychology, 37, 38-41.
3. Bendig, A.W. (1954 a). Reliability and the number of rating scale categories. Journal of Applied Psychology, 38, 38-40.
4. Bendig, A.W. (1954 b). Reliability of short rating scales and the heterogeneity of the rated stimuli. Journal of Applied Psychology, 38, 167-170.
5. Bendig, A.W. (1955). Rater reliability and the heterogeneity of the scale anchors. Journal of Applied Psychology, 39, 37-39.
6. Chang, L. (1997). Dependability of anchoring labels of Likert-type scales. Educational & Psychological Measurement, 57, 800-808.
7. Childers, T.L., Houston, M.J., & Heckler, S.E. (1985). Measurement of individual differences in visual versus verbal information processing. Journal of Consumer Research, 12, 125-134.
8. Cox, J. (1996). Your opinion, please!: How to build the best questionnaires in the field of education. Thousand Oaks, CA: Corwin Press.
9. Desimone, L.M., & Floch, D.C.L. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26, 1-22.
10. Dixon, P.N., BoBo, M., & Stevick, R.A. (1984). Response differences and preferences for all-category-defined and end-defined Likert formats. Educational and Psychological Measurement, 44, 61-66.
11. Dunham, T.C., & Davison, M.L. (1991). Effects of scale anchors on student ratings of instructors. Applied Measurement in Education, 4, 23-35.
12. Feldt, L.S. (1969). A test of the hypothesis that Cronbach's alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363-373.
13. Fowler, F.J. (1992). How unclear terms can affect survey data. Public Opinion Quarterly, 56, 218-231.
14. Fowler, F.J. (1995). Improving survey questions: Design and evaluation (Vol. 38). Thousand Oaks, CA: Sage.
15. French-Lazovik, G., & Gibson, C.L. (1984). Effects of verbally labeled anchor points on the distributional parameters of rating measures. Applied Psychological Measurement, 2, 49-57.
16. Frey, B.F., & Edwards, L.M. (2001, April). Strong words or moderate words: A comparison of the reliability and validity of responses on attitude scales. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.
17. Furuhata, T. (2003). The influence of labels and positions in rating scales of Japanese students. Unpublished doctoral dissertation, University of Washington.
18. Gable, R.W., & Wolf, M.B. (1993). Instrument development in the affective domain (2 nd ed.). Boston: Kluwer.
19. Graesser, A.C., Bommareddy, S., Swamer, S., & Golding, J.M. (1996). Integrating questionnaire design with a cognitive computational model of human question answering. In N. Schwartz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 143-174). San Francisco: Jossey-Bass.
20. Hancock, G.R., & Klockars, A.J. (1991). The effect of scale manipulations on validity: Targeting frequency rating scales for anticipated performance levels. Applied Ergonomics, 22, 147-154.
21. Hippler, H.-J., Schwartz, N., & Sudman, S. (1988). Social information processing and survey methodology. New York: Springer-Verlag.
22. Holleman, B. (1999). Wording effects in survey research: Using meta-analysis to explain the forbid/allow asymmetry. Journal of Quantitative Linguistics, 6, 29-40.
23. Jenkins, G.D., Jr., & Taber, T.D. (1977). A Monte Carlo study of factors affecting three indices of composite scale reliability. Journal of Applied Psychology, 63, 392-398.
24. Jobe, J., & Mingay, D.J. (1991). Cognition and survey measurement: History and overview. Applied Cognitive Psychology, 54, 940-952.
25. Jobe, J.B., & Mingay, D.J. (1989). Cognitive research improved questionnaires. American Journal of Public Health, 79, 1053-1055.
26. Klockars, A.J., & Hancock, G.R. (1993). Manipulations of evaluative rating scales to increase validity. Psychological Reports, 73, 1059-1066.
27. Klockars, A.J., & Yamagishi, M. (1988). The influence of labels and positions in rating scales. Journal of Educational Measurement, 25, 85-96.
28. Krosnick, J.A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.
29. Krosnick, J.A., Narayan, S., & Smith, W.R. (1996). Satisficing in surveys: Initial evidence. New Directions for Evaluation, 70, 29-44.
30. Lam, T.C.M. (1997). Issues in rating scale construction. Unpublished manuscript. Toronto, Canada: University of Toronto.
31. Lam, T.C.M., & Klockars, A.J. (1982). Anchor point effects on the equivalence of questionnaire items. Journal of Educational Measurement, 19, 317-322.
32. Lam, T.C.M., & Stevens, J.J. (1994). Effects of content polarization, item wording, and rating scale width on rating response. Applied Measurement in Education, 7, 141-158.
33. Likert, R.A. (1932). A technique for the measurement of attitudes. Archives of Psychology, 40-52.
34. Lissitz, R.W., & Green, S.B. (1975). Effects of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60, 10-13.
35. Newstead, S.E., & Arnold, J. (1989). The effect of response format on ratings of teaching. Educational and Psychological Measurement, 49, 33-43.
36. Nunnally, J.C. (1978). Psychometric theory (2 nd ed.). New York: McGraw-Hill.
37. Ostrom, T.M., & Gannon, K.M. (1996). Exemplar generation. In N. Schwartz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 293-318). San Francisco: Jossey-Bass.
38. Peterson, R.A. (2000). Constructing effective questionnaires. Thousand Oaks, CA: Sage.
39. Rotter, G.S. (1972). Attitudinal points of agreement and disagreement. Journal of Social Psychology, 86, 211-218.
40. Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. New York: Academic.
41. Schwartz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93-105.
42. Schwarz, N., Knauper, B., Hippler, H.-J., Noelle-Neumann, E., & Clark, L. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570-582.
43. Schwartz, N., & Sudman, S. (1996). Answering questions: Methodology for determining cognitive and communicative processes in survey research. San Francisco: Jossey-Bass.
44. Sirken, M.G., Herrmann, D.J., Schechter, S., Schwarz, N., Tanur, J.M., & Tourangeau, R. (1999). Cognition and survey research. New York: John Wiley.
45. Smith, T.W. (1987). That which we call welfare by any other name would smell sweeter: An analysis of the impact of question wording on response patterns. Public Opinion Quarterly, 51, 75-83.
46. Spector, P.E. (1976). Choosing response categories for summated rating scales. Journal of Applied Psychology, 61, 374-375.
47. Spector, P.E. (1980). Ratings of equal and unequal response choice intervals. Journal of Social Psychology, 112, 115-119.
48. Spector, P.E. (1992). Summated rating scale construction: An introduction. Newbury Park, CA: Sage.
49. Tanur, J.M. (1992). Questions about questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage Foundation.
50. Thomas, S.J. (2004). Using Web and paper questionnaires for data-based decision making: From design to interpretation of the results. Thousand Oaks, CA: Corwin Press.
51. Tourangeau, R., & Rasinski, K. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299-314.
52. Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
53. Viswanathan, M. (1993). Measurement of individual differences in preference for numerical information. Journal of Applied Psychology, 78, 741-752.
54. Weng, L. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956-972.
55. Willis, G.B. (1999). Cognitive interviewing: A ''how to'' guide. Retrieved July 29, 2005, from http://www.appliedresearch.cancer.gov/areas/ cognitive/interview.pdf
56. Willis, G.B. (2004). Cognitive interviewing revisited: A useful technique, in theory? In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, et al. (Eds.), Methods for testing and evaluating survey questionnaires (pp. 23-43). New Jersey: John Wiley.
57. Willis, G.B. (2005). Cognitive interviews: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.
58. Wyatt, R.C., & Meyers, L.S. (1987). Psychometric properties of four 5-point Likert-type response scales. Educational and Psychological Measurement, 47, 27-35.