A Comparison of a Subjective and Statistical Method for Establishing Score Comparability in an Organizational Culture Survey

Journal of Business and Psychology

Volumn 27, Issue 4, 2012, Pages 451-466

  Carter, Nathan T ^a   Kotrba, Lindsey M ^b   Diab, Dalia L ^c   Lin, Bing C ^d   Pui, Shuang Y ^e   Lake, Christopher J ^f   Gillespie, Michael A ^g   Zickar, Michael J ^f   Chao, Alen ^a  

^a UNIVERSITY OF CENTRAL FLORIDA   (United States)

^b Denison Consulting   (United States)

^c XAVIER UNIVERSITY   (United States)

^d PORTLAND STATE UNIVERSITY   (United States)

^e UNIVERSITY OF ILLINOIS AT SPRINGFIELD   (United States)

^f BOWLING GREEN STATE UNIVERSITY   (United States)

^g UNIVERSITY OF SOUTH FLORIDA SARASOTA MANATEE   (United States)

Author keywords

Differential item functioning; Invariance; Language translated surveys; Multinational organizational surveys; Organizational culture surveys

Indexed keywords

EID: 84869093011     PISSN: 08893268     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10869-011-9254-1     Document Type: Article

References (61)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2. Baker, M., Olohan, M., & Perez, M. C. (2010). Text and context: Essays on translation and interpreting in honour of Ian Mason. Manchester, UK: Saint Jerome Publishing.
3. Behrend, T. S., Thompson, L. F., Meade, A. W., Newton, D. A., & Grayson, M. S. (2008). Measurement invariance in careers research: Using IRT to study gender differences in medical students' specialization decisions. Journal of Career Development, 35, 60-83.
4. Berdie, F. S. (1971). What test questions are likely to offend the general public. Journal of Educational Measurement, 8, 87-93.
5. Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071.
6. Brown, R. T., Reynolds, C. R., & Whitaker, J. S. (1999). Bias in mental testing since "Bias in Mental Testing". School Psychology Quarterly, 14, 208-238.
7. Budgell, G. R., Raju, N. S., & Quartetti, D. A. (1995). Analysis of differential item functioning in translated assessment instruments. Applied Psychological Measurement, 19, 309-321.
8. Candell, G. L., & Drasgow, F. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253-260.
9. Carter, N. T., & Zickar, M. J. (2011). A comparison of the LR and DFIT frameworks of differential functioning to the generalized graded unfolding model. Applied Psychological Measurement, 35, 623-642.
10. Clark, P. C., & LaHuis, D. M. (2011). An examination of power and type I errors for two differential item functioning indices using the graded response model. Organizational Research Methods. doi: 10. 1177/1094428111403815.
11. Cramér, H. (1945). Mathematical methods of statistics. Uppsala: Almqvist & Wiksells.
12. De Ayala, R. J., Kim, S.-H., Stapleton, L. M., & Dayton, C. M. (2002). Differential item functioning: A mixture distribution conceptualization. International Journal of Testing, 2, 243-276.
13. Denison, D. R. (1984). Bringing corporate culture to the bottom line. Organizational Dynamics, 13, 4-22.
14. Denison, D. R. (1990). Corporate culture and organizational effectiveness. New York: Wiley.
15. Denison, D. R. (2000). Organizational culture: Can it be a key lever for driving organizational change? In C. L. Cooper, S. Cartwright, & P. C. Earley (Eds.), The international handbook of organizational culture and climate (pp. 347-372). New York: Wiley.
16. Denison, D. R., Janovics, J., Young, J., & Cho, H. J. (2007). Diagnosing organizational cultures: Validating a model and method. Working paper, International Institute for Management Development, Lausanne, Switzerland.
17. Denison, D. R., & Mishra, A. K. (1995). Toward a theory of organizational culture and effectiveness. Organizational Science, 6, 204-223.
18. Denison, D. R., & Neale, W. (1996). Denison organizational culture survey. Ann Arbor: Aviat.
19. Denison, D. R., & Neale, W. (2000). Denison organizational culture survey. Ann Arbor, MI: Denison Consulting.
20. Drasgow, F. (1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are central issues. Psychological Bulletin, 95, 134-135.
21. Drasgow, F., Levine, M. V., Tsien, S., Williams, B. A., & Mead, A. D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143-165.
22. Ellis, B. B., Becker, P., & Kimmel, H. D. (1993). An item response theory evaluation of an English version of the Trier Personality Inventory. Journal of Cross-Cultural Psychology, 23, 133-148.
23. Ellis, B. B., Minsel, B., & Becker, P. (1989). Evaluation of attitude survey translations: An investigation using item response theory. International Journal of Psychology, 24, 665-684.
24. Fey, C. F., & Denison, D. R. (2003). Organizational culture and effectiveness: Can American theory be applied in Russia? Organizational Science, 14, 686-706.
25. Gierl, M. J., & Khaliq, S. N. (2001). Identifying sources of differential item and bundle functioning on translated achievement tests: A confirmatory analysis. Journal of Educational Measurement, 38, 164-187.
26. Guion, R. M. (1977). Content validity: The source of my discontent. Applied Psychological Measurement, 1, 1-10.
27. Guion, R. M. (1980). On Trinitarian doctrines of validity. Professional Psychology, 11, 385-398.
28. Hambleton, R. K., & de Jong, J. H. A. L. (2003). Advances in translating and adapting educational and psychological tests. Language Testing, 20, 127-134.
29. Hambleton, R. K., & Kanjee, A. (1995). Increasing the validity of cross-cultural assessments: Use of improved methods for test adaptation. European Journal of Psychological Assessment, 11, 147-157.
30. Hambleton, R. K., & Patsula, L. (1999). Increasing the validity of adapted tests: Myths to be avoided and guidelines for improving test adaptation practices. Journal of Applied Testing Technology, 1, 1-12.
31. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
32. Highhouse, S. (2008). Stubborn reliance on intuition and subjectivity in employee selection. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 333-342.
33. Hofstede, G. (1991). Empirical models of cultural differences. In N. Bleichrodt & P. J. D. Drenth (Eds.), Contemporary issues in cross-cultural psychology (pp. 4-20). Lisse: Swets & Zeitlinger.
34. Huang, C. D., Church, A. T., & Katigbak, M. S. (1997). Identifying cultural differences in items and traits: Differential item functioning in the NEO personality inventory. Journal of Cross-Cultural Psychology, 28, 192-218.
35. Hulin, C. L., Drasgow, F., & Komocar, J. (1982). Applications of item response theory to analysis of attitude scale translations. Journal of Applied Psychology, 67, 818-825.
36. Hulin, C. L., & Mayer, L. J. (1986). Psychometric equivalence of a translation of the job descriptive index into Hebrew. Journal of Applied Psychology, 71, 83-94.
37. Jensen, A. R. (1980). Bias in mental testing. New York: Free Press.
38. Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527-535.
39. Kim, S.-H., & Cohen, A. S. (1995). A comparison of Lord's Chi-square, Raju's area measures, and the likelihood ratio test on detection of differential item functioning. Applied Measurement in Education, 8, 291-312.
40. Kim, S.-H., & Cohen, A. S. (1998). Detection of differential item functioning under the graded response model with the likelihood ratio test. Applied Psychological Measurement, 22, 345-355.
41. Landy, F. J. (2005). Employment discrimination litigation: Behavioral, quantitative, and legal perspectives. San Francisco: Jossey-Bass.
42. McKay, R., Breslow, M., Sangster, R., Gabbard, S., Reynolds, R., Nakamoto, J., et al. (1996). Translating survey questionnaires: Lessons learned. New Directions for Evaluation, 70, 93-104.
43. Mellenbergh, G. J. (1989). Item bias and item response theory. International Journal of Educational Research, 13, 127-143.
44. Oshima, T. C., & Morris, S. B. (2008). Raju's differential functioning of items and tests. Educational Measurement: Issues and Practice, 27, 43-50.
45. Pérez, E. O. (2009). Lost in translation? Item validity in bilingual political surveys. The Journal of Politics, 71, 1530-1548.
46. Plake, B. S. (1980). A comparison of a statistical and subjective procedure to ascertain item validity: One step in the test validation process. Educational and Psychological Measurement, 40, 397-404.
47. Ramirez, M., Ford, M. E., Stewart, A. L., & Teresi, J. A. (2005). Measurement issues in health disparities research. Health Services Research, 40, 1640-1657.
48. Ramsey, P. (1993). Sensitivity review: The ETS experience as a case study. In P. Holland & H. Wainer (Eds.), Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum.
49. Rogers, T. B. (1992). Antecedents of operationalism: A case history in radical positivism. In C. W. Tolman (Ed.), Positivism in psychology: Historical and contemporary problems. London: Springer-Verlag.
50. Ross, S. J., & Okabe, J. (2006). The subjective and objective interface of bias detection on languages. International Journal of Testing, 6, 229-253.
51. Ryan, A. M., Chan, D., Ployhart, R. E., & Slade, A. L. (1999). Employee attitude surveys in a multinational organization: Considering language and culture in assessing measurement equivalence. Personnel Psychology, 52, 37-58.
52. Ryan, A. M., Horvath, M., Ployhart, R. E., Schmitt, N., & Slade, L. A. (2000). Hypothesizing differential item functioning in global employee opinion surveys. Personnel Psychology, 53, 531-562.
53. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34, 100-114.
54. Sandoval, J., & Miille, M. W. (1980). Accuracy of judgments of WISC-R item difficulty for minority groups. Journal of Consulting and Clinical Psychology, 48, 249-253.
55. Searcy, C. A., & Lautenschlager, G. J. (1999). A monte carlo investigation of DIF assessment forpolytomously scored items. Paper presented at the 14th annual meeting of the Society for Industrial and Organizational Psychology, Atlanta, GA.
56. Thissen, D. (2001). IRTLRDIFv2. 0b: Software for the computation of the statistics involved in item response theory-based likelihood ratio tests for differential item functioning. Chapel Hill, NC: University of North Carolina.
57. Thissen, D. (2003). MULTILOG 7. 0: Multiple, categorical item analysis and test scoring using item response theory [Computer Program]. Chicago, IL: Scientific Software, Inc.
58. van de Vijver, F., & Hambleton, R. K. (1996). Translating tests: Some practical guidelines. European Psychologist, 1, 89-99.
59. Willgerodt, M. A. (2003). Using focus groups to develop culturally relevant instruments. Western Journal of Nursing Research, 25, 798-814.
60. Woods, C. M. (2011). DIF testing for ordinal items with poly-SIBTEST, the Mantel and GMH tests, and IRT-LR-DIF when the latent distribution is nonnormal for both groups. Applied Psychological Measurement, 35, 145-164.
61. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning: Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, ON: Directorate of Human Resources Research and Evaluation, Department of National Defense.