How irt can solve problems of ipsative data in forced-choice questionnaires

Psychological Methods

Volumn 18, Issue 1, 2013, Pages 36-52

  Brown, Anna ^a   Maydeu Olivares, Alberto ^b  

^a UNIVERSITY OF KENT   (United Kingdom)

^b UNIVERSITY OF BARCELONA   (Spain)

Author keywords

Forced choice format; Ipsative data; Multidimensional IRT; Thurstonian factor model; Thurstonian IRT model

Indexed keywords

ARTICLE; HUMAN; PSYCHOLOGICAL MODEL; PSYCHOMETRY; QUESTIONNAIRE; REPRODUCIBILITY; STANDARD;

HUMANS; MODELS, PSYCHOLOGICAL; PSYCHOMETRICS; QUESTIONNAIRES; REPRODUCIBILITY OF RESULTS;

EID: 84882970412     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0030641     Document Type: Article

References (51)

1. Ackerman, T. A. (2005). Multidimensional item response theory modeling. In A. Maydeu-Olivares & J. J. McArdle. (Eds.), Contemporary psychometrics (pp. 3-26). Mahwah, NJ: Erlbaum.
2. Asparouhov, T., & Muthén, B. (2010). Multiple imputation with Mplus, Version 2. Retrieved from http://www.statmodel.com
3. Baron, H. (1996). Strengths and limitations of ipsative measurement. Journal of Occupational and Organizational Psychology, 69, 49-56. doi:10.1111/j.2044-8325.1996.tb00599.x
4. Bartram, D. (2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment, 15, 263-272. doi:10.1111/j.1468-2389.2007.00386.x
5. Brown, A. (2009, April). Doing less but getting more: Improving forcedchoice measures with IRT. Paper presented at the conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
6. Brown, A., Ford, T., Deighton, J., & Wolpert, M. (2012). Satisfaction in child and adolescent mental health services: Translating users' feedback into measurement. Administration and Policy in Mental Health and Mental Health Services Research. Advance online publication. doi: 10.1007/s10488-012-0433-9
7. Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71, 460-502. doi:10.1177/0013164410375112
8. Brown, A., & Maydeu-Olivares, A. (2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-012-0217-x
9. Chan, W. (2003). Analyzing ipsative data in psychological research. Behaviormetrika, 30, 99-121. doi:10.2333/bhmk.30.99
10. Chan, W., & Bentler, P. M. (1998). Covariance structure analysis of ordinal ipsative data. Psychometrika, 63, 369-399. doi:10.1007/BF02294861
11. Chernyshenko, O. S., Stark, S., Prewett, M. S., Gray, A. A., Stilson, F. R., & Tuttle, M. D. (2009). Normative scoring of multidimensional pairwise preference personality scales using IRT: Empirical comparisons with other formats. Human Performance, 22, 105-127. doi:10.1080/08959280902743303
12. Cheung, M. W. L., & Chan, W. (2002). Reducing uniform response bias with ipsative measurement in multiple-group confirmatory factor analysis. Structural Equation Modeling, 9, 55-77. doi:10.1207/S15328007SEM0901_4
13. Christiansen, N., Burns, G., & Montgomery, G. (2005). Reconsidering the use of forced-choice formats for applicant personality assessment. Human Performance, 18, 267-307. doi:10.1207/s15327043hup1803_4
14. Clemans, W. V. (1966). An analytical and empirical examination of some properties of ipsative measures (Psychometric Monograph No. 14). Retrieved from http://www.psychometrika.org/journal/online/MN14.pdf
15. Closs, S. J. (1996). On the factoring and interpretation of ipsative data. Journal of Occupational and Organizational Psychology, 69, 41-47. doi:10.1111/j.2044-8325.1996.tb00598.x
16. Cornwell, J. M., & Dunlap, W. P. (1994). On the questionable soundness of factoring ipsative data: A response to Saville & Willson. Journal of Occupational and Organizational Psychology, 67, 89-100. doi:10.1111/j.2044-8325.1994.tb00553.x
17. Dunlap, W. P., & Cornwell, J. M. (1994). Factor analysis of ipsative measures. Multivariate Behavioral Research, 29, 115-126. doi:10.1207/s15327906mbr2901_4
18. Du Toit, M. (Ed.). (2003). IRT from SSI: BILOG-MG, MULTILOG, PARSCALE, TESTFACT. Lincolnwood IL: Scientific Software International.
19. Embretson, S., & Reise, S. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
20. Friedman, H., & Amoo, T. (1999). Rating the rating scales. Journal of Marketing Management, 9, 114-123.
21. Gordon, L. V. (1976). Survey of Interpersonal Values: Revised manual. Chicago, IL: Science Research Associates.
22. Gordon, L. V. (1993). Gordon Personal Profile Inventory (GPP-1): Manual. San Antonio, TX: Psychological Corporation.
23. Green, B. F., Bock, R. D., Humphreys, L. G., Linn, R. L., & Reckase, M. D. (1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 21, 347-360. doi:10.1111/j.1745-3984.1984.tb01039.x
24. Hicks, L. E. (1970). Some properties of ipsative, normative, and forcedchoice normative measures. Psychological Bulletin, 74, 167-184. doi: 10.1037/h0029780
25. Jackson, D., Wroblewski, V., & Ashton, M. (2000). The impact of faking on employment tests: Does forced choice offer a solution? Human Performance, 13, 371-388. doi:10.1207/S15327043HUP1304_3
26. Johnson, C. E., Wood, R., & Blinkhorn, S. F. (1988). Spuriouser and spuriouser: The use of ipsative personality tests. Journal of Occupational Psychology, 61, 153-162. doi:10.1111/j.2044-8325.1988.tb00279.x
27. Johnson, T., Kulesa, P., Cho, Y. I., & Shavitt, S. (2005). The relation between culture and response styles: Evidence from 19 countries. Journal of Cross-Cultural Psychology, 36, 264-277. doi:10.1177/0022022104272905
28. Karpatschof, B., & Elkjaer, H. K. (2000). Yet the bumblebee flies: The reliability of ipsative scores examined by empirical data and a simulation study (Research Rep. No. 1). Copenhagen, Denmark: Department of Psychology, University of Copenhagen.
29. Kolb, A., & Kolb, D. (2005). The Kolb Learning Style Inventory-Version 3.1. Technical specifications. Boston, MA: Hay Resource Direct.
30. Maydeu-Olivares, A. (1999). Thurstonian modeling of ranking data via mean and covariance structure analysis. Psychometrika, 64, 325-340. doi:10.1007/BF02294299
31. Maydeu-Olivares, A., & Böckenholt, U. (2005). Structural equation modeling of paired-comparison and ranking data. Psychological Methods, 10, 285-304. doi:10.1037/1082-989X.10.3.285
32. Maydeu-Olivares, A., & Brown, A. (2010). Item response modeling of paired comparison and ranking data. Multivariate Behavioral Research, 45, 935-974. doi:10.1080/00273171.2010.531231
33. McCloy, R. A., Heggestad, E. D., & Reeve, C. L. (2005). A silk purse from the sow's ear: Retrieving normative information from multidimensional forced-choice items. Organizational Research Methods, 8, 222-248. doi:10.1177/1094428105275374
34. McDonald, R. P. (1999). Test theory: A unified approach. Mahwah, NJ: Erlbaum.
35. Meade, A. (2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology, 77, 531-551. doi:10.1348/0963179042596504
36. Muthén, L. K., & Muthén, B. O. (1998-2010). Mplus user's guide (6th ed.). Los Angeles, CA: Author.
37. Reckase, M. (2009). Multidimensional item response theory. Retrieved from http://springerlink.com/openurl.asp?genre=book&isbn=978-0-387-89975-6
38. Roberts, J. S., Donoghue, J. R., & Laughlin, J. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24, 3-32.
39. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph No. 17). Retrieved from http://www.psychometrika.org/journal/online/MN17.pdf
40. SHL. (1997). Customer contact: Manual and user's guide. Surrey, England: SHL Group.
41. SHL. (2006). OPQ32 technical manual. Surrey, England: SHL Group.
42. Stark, S. (2002). A new IRT approach to test construction and scoring designed to reduce the effects of faking in personality assessment. Unpublished doctoral dissertation, University of Illinois at Urbana- Champaign.
43. Stark, S., Chernyshenko, O., & Drasgow, F. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The Multi-Unidimensional Pairwise-Preference Model. Applied Psychological Measurement, 29, 184-203. doi:10.1177/0146621604273988
44. Stark, S., Chernyshenko, O. S., Drasgow, F., & White, L. A. (2012). Adaptive testing with multidimensional pairwise preference items: Improving the efficiency of personality and other noncognitive assessments. Organizational Research Methods, 15, 463-487. doi:10.1177/1094428112444611
45. Stark, S., Chernyshenko, O. S., Drasgow, F., & Williams, B. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring? Journal of Applied Psychology, 91, 25-39. doi:10.1037/0021-9010.91.1.25
46. Tenopyr, M. L. (1988). Artifactual reliability of forced-choice scales. Journal of Applied Psychology, 73, 749-751. doi:10.1037/0021-9010.73.4.749
47. Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286. doi:10.1037/h0070288
48. Thurstone, L. L. (1929). The measurement of psychological value. In T. V. Smith &W. K. Wright (Eds.), Essays in philosophy by seventeen doctors of philosophy of the University of Chicago (pp. 157-174). Chicago, IL: Open Court.
49. Thurstone, L. L. (1931). Rank order as a psychophysical method. Journal of Experimental Psychology, 14, 187-201. doi:10.1037/h0070025
50. Tsai, R. C., & Böckenholt, U. (2001). Maximum likelihood estimation of factor and ideal point models for paired comparison data. Journal of Mathematical Psychology, 45, 795-811. doi:10.1006/jmps.2000.1353
51. van Herk, H., Poortinga, Y. H., & Verhallen, T. M. M. (2004). Response styles in rating scales: Evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35, 346-360. doi: 10.1177/0022022104264126