A Dynamic Thurstonian item response theory of motive expression in the picture story exercise: Solving the internal consistency paradox of the PSE

Psychological Review

Volumn 121, Issue 3, 2014, Pages 481-500

  Lang, Jonas W B ^a,b  

^a GHENT UNIVERSITY   (Belgium)

^b MAASTRICHT UNIVERSITY   (Netherlands)

Author keywords

Choice model; Implicit motives; Psychometric; Reliability; Thematic apperception test

Indexed keywords

COMPUTER SIMULATION; DECISION MAKING; HUMAN; MOTIVATION; PHYSIOLOGY; PSYCHOLOGICAL MODEL; PSYCHOLOGICAL THEORY;

CHOICE BEHAVIOR; COMPUTER SIMULATION; HUMANS; MODELS, PSYCHOLOGICAL; MOTIVATION; PSYCHOLOGICAL THEORY;

EID: 84905492882     PISSN: 0033295X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0037011     Document Type: Article

References (92)

1. Atkinson, J. W. (1958). Thematic apperceptive measurement of motives within the context of a theory of motivation. In J. W. Atkinson (Ed.), Motives in fantasy, action, and society: A method of assessment and study (pp. 596-616). Princeton, NJ: Van Nostrand.
2. Atkinson, J. W., & Birch, D. (1970). The dynamics of action. New York, NY: Wiley.
3. Atkinson, J. W., Bongort, K., & Price, L. H. (1977). Explorations using computer simulation to comprehend thematic apperceptive measurement of motivation. Motivation and Emotion, 1, 1-27. doi:10.1007/ BF00997578
4. Bates, D. (2010). lme 4: Mixed-effects modeling with R [Computer software]. Retrieved from http://lme4.r-forge.r-project.org/lMMwR/lrgprt .pdf
5. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2013). lme4: Linear mixed-effects models using S4 classes (Version 1.0 -5) [Computer software]. Retrieved from http://lme4.r-forge.r-project.org/repos
6. Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137-143
7. Blankenship, V., Vega, C. M., Ramos, E., Romero, K., Warren, K., Keenan, K. Sullivan, A. (2006). Using the multifaceted Rasch model to improve the TAT/PSE measure of need for achievement. Journal of Personality Assessment, 86, 100-114
8. Blankenship, V., & Zoota, A. L. (1998). Comparing power imagery in TATs written by hand or on the computer. Behavior Research Methods, Instruments, & Computers, 30, 441-448
9. Bliese, P. D., & Ployhart, R. E. (2002). Growth modeling using random coefficient models: Model building, testing, and illustrations. Organizational Research Methods, 5, 362-387
10. Bock, R. D., Brennan, R. L., & Muraki, E. (2002). The information in multiple ratings. Applied Psychological Measurement, 26, 364-375
11. Böckenholt, U. (2001). Hierarchical modeling of paired comparison data. Psychological Methods, 6, 49-66
12. Böckenholt, U. (2004). Comparative judgments as an alternative to ratings: Identifying the scale origin. Psychological Methods, 9, 453-465
13. Böckenholt, U. (2006). Thurstonian-based analyses: Past, present, and future utilities. Psychometrika, 71, 615-629
14. Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J. S. (2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology & Evolution, 24, 127-135
15. Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71, 425-440
16. Borsboom, D., Mellenberg, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071
17. Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71, 460-502
18. Brown, A., & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18, 36-52
19. Brunstein, J. C., & Heckhausen, H. (2008). Achievement motivation. In J. Heckhausen & H. Heckhausen, Motivation and action (2nd ed., pp. 137-183)
20. Brunstein, J. C., & Maier, G. W. (2005). Implicit and self-attributed motives to achieve: Two separate but interacting needs. Journal of Personality and Social Psychology, 89, 205-222
21. Busch, H., & Hofer, J. (2012). A picture story exercise set in a German and a Cameroonian sample: Ipsative stability, retest reliability, and samplelevel stability. European Journal of Psychological Assessment, 28, 125- 131
22. Carver, C. S., & Scheier, M. F. (1998). On the self-regulation of behavior
23. Carver, C. S., & Scheier, M. F. (2002). Control processes and selforganization as complementary principles underlying behavior. Personality and Social Psychology Review, 6, 304 -315
24. Critchlow, D. E., & Fligner, M. A. (1991). Paired comparison, triple comparison, and ranking experiments as generalized linear models, and their implementation on GLIM. Psychometrika, 56, 517-533
25. Debeer, D., & Janssen, R. (2013). Modeling item-position effects within an IRT framework. Journal of Educational Measurement, 50, 164-185
26. De Boeck, P. (2008). Random item IRT models. Psychometrika, 73, 533-559
27. De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Hofman, A., Tuerlinckx, F., & Partchev, I. (2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39(12), 1-28. Retrieved from http://www.jstatsoft.org/v39/i12
28. Doran, H., Bates, D., Bliese, P. D., & Dowling, M. (2007). Estimating the multilevel Rasch model: With the lme4 package. Journal of Statistical Software, 20(2), 1-18. Retrieved from http://www.jstatsoft.org/v20/i02
29. Eggerth, D. E. (2004). Applying the Bradley-Terry-Luce method to P-E fit. Journal of Vocational Behavior, 64, 92-107
30. Embretson, S. E., & Reise, S. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
31. Emmons, R. A. (1989). Exploring the relations between motives and traits: The case of narcissism. In D. Buss & N. Cantor (Eds.), Personality psychology: Recent trends and emerging directions (pp. 32-44)
32. Entwisle, D. R. (1972). To dispel fantasies about fantasy-based measures of achievement motivation. Psychological Bulletin, 77, 377-391
33. Fineman, S. (1977). The achievement motive construct and its measurement: Where are we now? British Journal of Psychology, 68, 1-22. doi:10.1111/j.2044-8295.1977.tb01554.x
34. Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19, 72-91
35. Gruber, N., & Kreuzpointer, L. (2013). Measuring the reliability of picture story exercises like the TAT. PLoS ONE, 8, Article e79450
36. Gurin, G., Veroff, J., & Feld, S. (1957). Americans view their mental health, 1957 (ICPSR 3503) [Computer file]. Ann Arbor, MI: Inter- University Consortium for Political and Social Research
37. Gurin, G., Veroff, J., & Feld, S. (1960). Americans view their mental health: A nationwide interview study. New York, NY: Basic Books.
38. Hadfield, J. D. (2010). MCMC methods for multi-response generalized linear mixed models: The MCMCglmm R package. Journal of Statistical Software, 33(2), 1-22. Retrieved from http://www.jstatsoft.org/v33/i02/
39. Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. New York, NY: Routledge.
40. Heckhausen, H. (1963). Hoffnung und Furcht in der Leistungsmotivation [Hope and fear components in achievement motivation]. Meisenheim am Glan, Germany: Hain.
41. Houston, W. M., Raymond, M. R., & Svec, J. C. (1991). Adjustments for rater effects in performance assessment. Applied Psychological Measurement, 15, 409-421
42. Johnson, M. S., Sinharay, S., & Bradlow, E. T. (2006). Hierarchical item response theory models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp. 587-604). Amsterdam, the Netherlands: North-Holland.
43. Kehr, H. M. (2004). Integrating implicit motives, explicit motives, and perceived abilities: The compensatory model of work motivation and volition. Academy of Management Review, 29, 479-499
44. Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and practices
45. Kuhl, J. (1978). Situations-, reaktions- und personbezogene Konsistenz des Leistungsmotivs bei der Messung mittels des Heckhausen TAT [Situation-, reaction-, and person-related consistency of the achievement motive measured using Heckhausen's TAT]. Archiv für Psychologie, 130, 37-52.
46. Kuhl, J., & Blankenship, V. (1979). The dynamic theory of achievement motivation: From episodic to dynamic thinking. Psychological Review, 86, 141-151
47. Kuk, Y. C. A. (1995). Asymptotically unbiased estimation in generalized linear models with random effects. Journal of the Royal Statistical Society: Series B (Methodological), 57, 395-407.
48. Lang, J. W. B., Zettler, I., Ewen, C., & Hülsheger, U. R. (2012). Implicit motives, explicit traits, and task and contextual performance at work. Journal of Applied Psychology, 97, 1201-1217
49. LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815-852
50. Lilienfeld, S. O., Wood, J. M., & Garb, H. N. (2000). The scientific status of projective techniques. Psychological Science in the Public Interest, 1, 27-66
51. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
52. Mariano, L. T., & Junker, B. W. (2007). Covariates of the rating process in hierarchical models for multiple ratings of test items. Journal of Educational and Behavioral Statistics, 32, 287-314
53. McClelland, D. C., Koestner, R., & Weinberger, J. (1989). How do self-attributed and implicit motives differ? Psychological Review, 96, 690-702
54. McFadden, D. (2001). Economic choices. American Economic Review, 91, 351-378
55. Mitchell, S. K. (1979). Interobserver agreement, reliability, and generalizability of data collected in observational studies. Psychological Bulletin, 86, 376-390
56. Morgan, C. D., & Murray, H. A. (1935). A method for investigating fantasies: The Thematic Apperception Test. Archives of Neurology & Psychiatry, 34, 289 -306
57. Murray, H. A. (1938). Explorations in personality: A clinical and experimental study of fifty men of college age. New York, NY: Oxford University Press.
58. Ng, E. S. W., Carpenter, J. R., Goldstein, H., & Rabash, J. (2006). Estimation in generalized linear mixed models with binary outcomes by simulated maximum likelihood. Statistical Modelling, 6, 23-42
59. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
60. Pang, J. S. (2010). Content coding methods in implicit motive assessment: Standards of measurement and best practices for the picture story exercise. In O. C. Schultheiss & J. C. Brunstein (Eds.), Implicit motives (pp. 119-150)
61. Patz, R. J., Junker, B. W., Johnson, M. S., & Mariano, L. T. (2002). The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27, 341-384
62. R Development Core Team. (2013). R: A language and environment for statistical computing (Version 3.0.2) [Computer software]. Retrieved from http://www.r-project.org
63. Ramsay, J. E., & Pang, J. S. (2013). Set ambiguity: A key determinant of reliability and validity in the picture story exercise. Motivation and Emotion, 37, 661-674
64. Reise, S. P., & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133-144
65. Reitman, W. R., & Atkinson, J. W. (1958). Some methodological problems in the use of thematic apperceptive measures of human motives. In J. W. Atkinson (Ed.), Motives in fantasy, action, and society (pp. 664-683). Princeton, NJ: Van Nostrand.
66. Reuman, D. A. (1982). Ipsative behavioral variability and the quality of thematic apperception measurement of the achievement motive. Journal of Personality and Social Psychology, 43, 1098-1110
67. Revelle, W. (1986). Motivation and efficiency of cognitive performance. In D. R. Brown & J. Veroff (Eds.), Frontiers of motivational psychology: Essays in honor of John W. Atkinson (pp. 107-131)
68. Revelle, W. (2012). psych: Procedures for Personality and Psychological Research (Version 1.2.8) [Computer software]. Retrieved from http:// cran.r-project.org
69. Revelle, W., & Michaels, E. J. (1976). The theory of achievement motivation revisited: The implications of inertial tendencies. Psychological Review, 83, 394-404
70. Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145-154
71. Scheffer, D., Kuhl, J., & Eichstaedt, J. (2003). Der Operante Motiv-Test (OMT): Inhaltsklassen, Auswertung, psychometrische Kennwerte und Validierung [The operant motive test (OMT): Contents, scoring, psychometric values, and validation]. In J. Stiensmeier-Pelster & F. Rheinberg (Eds.), Diagnostik von Motivation und Selbstkonzept (pp. 151- 167). Göttingen, Germany: Hogrefe.
72. Schultheiss, O. C., & Brunstein, J. C. (2001). Assessment of implicit motives with a research version of the TAT: Picture profiles, gender differences, and relations to other personality measures. Journal of Personality Assessment, 77, 71-86
73. Schultheiss, O. C., Liening, S. H., & Schad, D. (2008). The reliability of a picture story exercise measure of implicit motives: Estimates of internal consistency, retest reliability, and ipsative stability. Journal of Research in Personality, 42, 1560-1571
74. Schultheiss, O. C., & Pang, J. S. (2007). Measuring implicit motives. In R. W. Robins, R. C. Fraley, & R. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 322-344). New York, NY: Guilford Press.
75. Schultheiss, O. C., & Schultheiss, M. (2014). Implicit motive profile analysis: An if-then contingency approach to the picture-story exercise. Social and Personality Psychology Compass, 8, 1-16
76. Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107-120
77. Sijtsma, K. (2012). Future psychometrics: Ask what psychometrics can do for psychology. Psychometrika, 77, 4-20
78. Smith, G. T., McCarthy, D. M., & Anderson, K. G. (2000). On the sins of short-form development. Psychological Assessment, 12, 102-111
79. Spangler, W. D. (1992). Validity of questionnaire and TAT measures of need for achievement: Two meta-analyses. Psychological Bulletin, 112, 140-154
80. Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286
81. Tuerlinckx, F., De Boeck, P., & Lens, W. (2002). Measuring needs with the Thematic Apperception Test: A psychometric study. Journal of Personality and Social Psychology, 82, 448-461
82. Verguts, T., & De Boeck, P. (2000). A Rasch model for detecting learning while solving an intelligence test. Applied Psychological Measurement, 24, 151-162
83. Verhelst, N. D., & Glas, C. A. W. (1993). A dynamic generalization of the Rasch model. Psychometrika, 58, 395-415
84. Veroff, J., Atkinson, J. W., Feld, S. C., & Gurin, G. (1960). The use of thematic apperception to assess motivation in a nationwide interview study. Psychological Monographs: General and Applied, 74, 1-32
85. Veroff, J., Feld, S., & Crockett, H. (1966). Explorations into the effects of picture cues on thematic apperceptive expression of achievement motivation. Journal of Personality and Social Psychology, 3, 171-181
86. Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive theory. Applied Psychological Measurement, 6, 473-492
87. Westen, D. (1998). The scientific legacy of Sigmund Freud: Toward a psychodynamically informed psychological science. Psychological Bulletin, 124, 333-371
88. Winter, D. G., John, O. P., Stewart, A. J., Klohnen, E. C., & Duncan, L. E. (1998). Traits and motives: Toward an integration of two traditions in personality research. Psychological Review, 105, 230-250
89. Winter, D. G., & Stewart, A. J. (1977). Power motivation as a function of retest instructions. Journal of Consulting and Clinical Psychology, 45, 436-440
90. Woike, B. A. (2001). Working with free response data: Let's not give up hope. Psychological Inquiry, 12, 157-159
91. Woike, B. A., & McAdams, D. P. (2001). A response to Lilienfeld, Woods and Garb: TAT-based personality measures have considerable validity. APS Observer, 14(5). Retrieved from http://www.psychologicalscience .org/index.php/uncategorized/tat-commentary.html
92. Woodhouse, B., & Jackson, P. H. (1977). Lower bounds for the reliability of a test composed of nonhomogeneous items II: A search procedure to locate the greatest lower bound. Psychometrika, 42, 579-591