Examining question and context effects in organization survey data using item response theory

Organizational Research Methods

Volumn 12, Issue 3, 2009, Pages 529-553

  Rivers, Drew C ^a   Meade, Adam W ^a   Lou Fuller, W ^b  

^a NORTH CAROLINA STATE UNIVERSITY   (United States)

^b DUKE ENERGY   (United States)

Author keywords

Context effects; Employee surveys; Invariance testing; Item response theory; Question effects; Survey research

Indexed keywords

EID: 67649497492     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428108315864     Document Type: Article

References (63)

1. Baker, F.B. (1995). EQUATE 2.1. A Fortran-based computer program used for equate item parameters between two groups. Madison: University of Wisconsin.
2. Bless, H., Schwarz, N., & WiCurrencysignnke, M. (2003). The size of context effects in social judgment. In J. P. Forgas, K. D. Williams, & W. von Hippel (Eds.), Social judgments: Implicit and explicit processes (pp. 180-197). New York: Cambridge University Press.
3. Byrne, B.M., Shavelson, R.J., & Muthen, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105 (3). 456-466.
4. Candell, G.L., & Drasgow, F. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253-260.
5. Church, A.H. (2001). Is there a method to our madness? The impact of data collection methodology on organizational survey results. Personnel Psychology, 54, 937-969. (Pubitemid 33579331)
6. Church, A.H., & Waclawski, J. (2001). Designing and using organizational surveys: A seven step process. San Francisco: Jossey-Bass.
7. Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-254.
8. Colasanto, D., Singer, E., & Rogers, T. (1992). Context effects on responses to questions about AIDS. Public Opinion Quarterly, 56, 515-518.
9. Donoghue, J.R., & Isham, S.P. (1998). A comparison of procedures to detect item parameter drift. Applied Psychological Measurement, 22, 33-51.
10. Drasgow, F., Levine, M.V., Tsien, S., Williams, B.A., & Mead, A.D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143-165.
11. Ellis, B.B., Minsel, B., & Becker, P. (1989). Evaluation of attitude survey translations: An investigation using item response theory. International Journal of Psychology, 24, 665-684.
12. Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
13. Facteau, J.D., & Craig, S.B. (2001). Are performance appraisal ratings from different rating sources comparable ? Journal of Applied Psychology, 86, 215-227. (Pubitemid 33405775)
14. Flowers, C.P., Oshima, T.C., & Raju, N.S. (1999). A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement, 23, 309-326.
15. Gaskell, G.D., O'Muircheartaigh, C.A., & Wright, D.B. (1994). Survey questions about the frequency of vaguely defined events: The effects of response alternatives. Public Opinion Quarterly, 58, 241-254.
16. Golembiewski, R.T., Billingsley, K., & Yeager, S. (1975). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12, 133-157.
17. Groves, R.M. (1991). Measurement error across the disciplines. In P. B. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (Eds.), Measurement error in surveys (pp. 1-25). New York: John Wiley & Sons.
18. Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
19. Higgs, A.C., & Ashworth, S.D. (1996). Organizational surveys: Tools for assessment and research. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 19-40). San Francisco: Jossey-Bass.
20. Johnson, R.H. (1996). Life in the consortium: The mayflower group. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 285-309). San Francisco: Jossey-Bass.
21. Kraut, A.I. (1996). Introduction: An overview of organizational surveys. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and research (pp. 1-14). San Francisco: Jossey-Bass.
22. Kraut, A.I., & Saari, L.M. (1999). Organizational surveys: Coming of age for a new era. In A. I. Kraut & A. K. Korman (Eds.), Evolving practices in human resource management: Responses to a changing world of work (pp. 302-327). San Francisco: Jossey-Bass.
23. Lautenschlager, G.J., Flaherty, V.L., & Park, D.G. (1994). IRT differential item functioning: An examination of ability scale purifications. Educational and Psychological Measurement, 54, 21-31.
24. Maurer, T.J., Raju, N.S., & Collins, W.C. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83, 693-702.
25. McDonald, R.P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
26. Meade, A.W., & Lautenschlager, G.J. (2004). A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7, 361-388. (Pubitemid 39332694)
27. Meade, A.W., Lautenschlager, G.J., & Hecht, J.E. (2005). Establishing measurement equivalence and invariance in longitudinal data with item response theory. International Journal of Testing, 5, 279-300.
28. Meade, A.W., Lautenschlager, G.J., & Johnson, E.C. (2007). A Monte Carlo examination of the sensitivity of the differential functioning of items and tests framework for tests of measurement invariance with Likert data. Applied Psychological Measurement, 31, 430-455.
29. Park, D.G., & Lautenschlager, G.J. (1990). Improving IRT item bias detection with iterative linking and ability scale purification. Applied Psychological Measurement, 14, 163-173.
30. Raju, N. (1999). DFITPS6. A Fortran-based computer program for calculating DIF/DTF [computer software]. Chicago: Illinois Institute of Technology.
31. Raju, N.S., Laffitte, L.J., & Byrne, B.M. (2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87, 517-529. (Pubitemid 36288927)
32. Raju, N.S., van der Linden, W. J., & Fleer, P.F. (1995). IRT-based internal measure of differential functioning of items and tests. Applied Psychological Measurement, 19, 353-368.
33. Rogelberg, S.G., Church, A.H., Waclawski, J., & Stanton, J.M. (2002). Organizational survey research. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 141-160). Malden, MA: Blackwell.
34. Ryan, A.M., Horvath, M., Ployhart, R.E., Schmitt, N., & Slade, L.A. (2000). Hypothesizing differential item functioning in global employee opinion surveys. Personnel Psychology, 53 (3). 531-562.
35. Sackett, P.R., & Larson, J.R., Jr. (1990). Research strategies and tactics in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology: Vol. 1 (2 nd ed., pp. 419-489). Palo Alto, CA: Consulting Psychologists Press.
36. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monographs No. 18). Iowa City, IA: Psychometric Society.
37. Schuman, H., Kalton, G., & Ludwig, J. (1983). Context and contiguity in survey questionnaires. Public Opinion Quarterly, 47, 112-115.
38. Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. New York: Academic Press.
39. Schwarz, N. (1995). Social cognition: Information accessibility and use in social judgment. In E. E. Smith & D. N. Oshenson (Eds.), Thinking: An invitation to cognitive science (2 nd ed., pp. 345-376). Cambridge, MA: MIT Press.
40. Schwarz, N. (1999). Self-reports-How the questions shape the answers. American Psychologist, 54 (2). 93-105.
41. Schwarz, N. (in press). Attitude construction: Evaluation in context. Social Cognition. Retrieved April 25, 2007, from http://sitemaker.umich.edu/ norbert.schwarz/social-cognition
42. Schwarz, N., & Bless, H. (1992). Assimilation and contrast effects in attitude measurement: An inclusion/ exclusion model. Advances in Consumer Research, 19, 72-77.
43. Schwarz, N., Hippler, H.J., Deutsch, B., & Strack, F. (1985). Response scales-Effects of category range on reported behavior and comparative judgments. Public Opinion Quarterly, 49, 388-395.
44. Schwarz, N., Knauper, B.R., Hippler, H.J., Noelle-Neumann, E., & Clark, L. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570.
45. Schwarz, N., Strack, F., Muller, G., & Chassein, B. (1988). The range of response alternatives may determine the meaning of the question-Further evidence on informative functions of response alternatives. Social Cognition, 6, 107-117.
46. Sirken, M.G., & Schechter, S. (1999). Interdisciplinary survey methods research. In M. G. Sirken, D. J. Herrmann, S. Schechter, N. Schwarz, J. M. Tanur, & R. Tourangeau (Eds.), Cognition and survey research (pp. 1-10). New York: John Wiley & Sons.
47. Smith, T.W. (1991). Context effects in the general social survey. In P. B. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (Eds.), Measurement error in surveys (pp. 57-71). New York: John Wiley & Sons.
48. Stanton, J.M., Sinar, E.F., Balzer, W.K., & Smith, P.C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55, 167-194.
49. Stark, S. (2001). MODFIT: A computer program for model-data fit. Urbana-Champaign: University of Illinois.
50. Stark, S., Chernyshenko, O.S., & Drasgow, F. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91 (6). 1292-1306. (Pubitemid 44747822)
51. Steinberg, L. (1994). Context and serial-order effects in personality measurement-Limits on the generality of measuring changes the measure. Journal of Personality and Social Psychology, 66 (2). 341-349.
52. Steinberg, L. (2001). The consequences of pairing questions: Context effects in personality measurement. Journal of Personality and Social Psychology, 81 (2). 332-342. (Pubitemid 33594919)
53. Stocking, M.L., & Lord, F.M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210.
54. Thissen, D. (1991). MULTILOG users guide: Multiple categorical item analysis and test scoring using item response theory [computer software and manual]. Chicago: Scientific Software International.
55. Thissen, D. (2001). Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning. Chapel Hill: University of North Carolina at Chapel Hill.
56. Tourangeau, R. (1999). Context effects on answers to attitude questions. In M. G. Sirken, D. J. Herrmann, S. Schechter, N. Schwarz, J. M Tanur, & R. Tourangeau (Eds.), Cognition and survey research (pp. 111-122). New York: John Wiley and Sons.
57. Tourangeau, R., & Rasinski, K.A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299-314.
58. Tourangeau, R., Rasinski, K., Bradburn, N., & D' Andrade, R. (1989). Carryover effects in attitude surveys. Public Opinion Quarterly, 53, 495-524.
59. Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.
60. Tourangeau, R., Singer, E., & Presser, S. (2003). Context effects in attitude surveys: Effects on remote items and impact on predictive validity. Sociological Methods & Research, 31, 486-513.
61. Vandenberg, R.J., & Lance, C.E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69.
62. Wells, C.S., Subkoviak, M.J., & Serlin, R.C. (2002). The effect of item parameter drift on examinee ability estimates. Applied Psychological Measurement, 26, 77-87. (Pubitemid 36657141)
63. Wilson, T.D., & Hodges, S.D. (1992). Attitudes as temporary constructions. In L. L. Martin & A. Tesser (Eds.), The construction of social judgments (pp. 37-65). Hillsdale, NJ: Lawrence Erlbaum.