Toward a Process-Focused Model of Test Score Validity: Improving Psychological Assessment in Science and Practice

Psychological Assessment

Volumn 23, Issue 2, 2011, Pages 532-544

  Bornstein, Robert F ^a  

^a ADELPHI UNIVERSITY   (United States)

Author keywords

Construct validity; Psychological assessment; Psychometrics; Test bias; Validity

Indexed keywords

ARTICLE; CONTROLLED STUDY; CORRELATION ANALYSIS; EXPERIMENTAL STUDY; MODEL; OUTCOME ASSESSMENT; PREDICTION; PROCESS FOCUSED MODEL; PSYCHOLOGIC ASSESSMENT; PSYCHOLOGIC TEST; PSYCHOMETRY; RESEARCH; SCORING SYSTEM; SELF REPORT; VALIDATION STUDY;

BEHAVIORAL RESEARCH; BIAS (EPIDEMIOLOGY); HUMANS; MODELS, PSYCHOLOGICAL; PSYCHOLOGICAL TESTS; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS;

EID: 79959318886     PISSN: 10403590     EISSN: 1939134X     Source Type: Journal    
DOI: 10.1037/a0022402     Document Type: Article

References (89)

1. Achenbach T.M., Howell C.T., Quay H.C., Conners C.K. National survey of problems and competencies among four- to sixteen-year olds: Parents' reports for normative and clinical samples. Monographs of the Society for Research in Child Development 1991, 56(3, Serial No. 225).
2. Allen A., Montgomery M., Tubman J., Frazier L., Escovar L. The effects of assessment feedback on rapport-building and self-enhancement process. Journal of Mental Health Counseling 2003, 25:165-182.
3. Standards for educational and psychological testing 1999, American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, Author, Washington, DC.
4. Ethical principles of psychologists and code of conduct 2002, American Psychological Association, Author, Washington, DC.
5. Arnell K.M., Killman K.V., Fijavz D. Blinded by emotion: Target misses follow attention capture by arousing distractors in RSVP. Emotion 2007, 7:465-477. doi: 10.1037/1528-3542.7.3.465.
6. Baltes M.M. The many faces of dependency in old age 1996, Cambridge University Press, Cambridge, England.
7. Bem S.L. The measurement of psychological androgeny. Journal of Consulting and Clinical Psychology 1974, 42:155-162. doi: 10.1037/h0036215.
8. Bender L. A visual-motor gestalt test and its clinical use 1938, American Orthopsychiatric Association, New York, NY.
9. Bollen K.A., Davis W.R. Causal indicator models: Identification, estimation, and testing. Structural Equation Modeling 2009, 16:498-522. doi: 10.1080/10705510903008253.
10. Bornstein R.F. Sex differences in objective and projective dependency tests: A meta-analytic review. Assessment 1995, 2:319-331. doi: 10.1177/1073191195002004003.
11. Bornstein R.F. Criterion validity of objective and projective dependency tests: A meta-analytic assessment of behavioral prediction. Psychological Assessment 1999, 11:48-57. doi: 10.1037/1040-3590.11.1.48.
12. Bornstein R.F. Has psychology become the science of questionnaires? A survey of research outcome measures at the close of the 20th century. The General Psychologist 2001, 36:36-40.
13. Bornstein R.F. A process dissociation approach to objective-projective test score interrelationships. Journal of Personality Assessment 2002, 78:47-68. doi: 10.1207/S15327752JPA7801_04.
14. Bornstein R.F. Behaviorally referenced experimentation and symptom validation: A paradigm for 21st century personality disorder research. Journal of Personality Disorder 2003, 17:1-18.
15. Bornstein R.F. Toward a process-based framework for classifying personality tests: Comment on Meyer and Kurtz (2006). Journal of Personality Assessment 2007, 89:202-207.
16. Bornstein R.F., Bowers K.S., Bonner S. Effects of induced mood states on objective and projective dependency scores. Journal of Personality Assessment 1996, 67:324-340. doi: 10.1207/s15327752jpa6702_8.
17. Bornstein R.F., Bowers K.S., Bonner S. Relationships of objective and projective dependency scores to sex role orientation in college students. Journal of Personality Assessment 1996, 66:555-568. doi: 10.1207/s15327752jpa6603_6.
18. Bornstein R.F., Rossner S.C., Hill E.L., Stepanian M.L. Face validity and fakability of objective and projective measures of dependency. Journal of Personality Assessment 1994, 63:363-386. doi: 10.1207/s15327752jpa6302_14.
19. Borsboom D., Mellenbergh G.J., van Heerden J. The concept of validity. Psychological Review 2004, 111:1061-1071. doi: 10.1037/0033-295X.111.4.1061.
20. Briccetti K.A. Emotional indicators of deaf children on the Draw-a-Person test. American Annals of the Deaf 1994, 139:500-505.
21. Butcher J.N. Clinical personality assessment: Practical approaches 2002, Oxford University Press, New York, NY. 2nd ed.
22. Campbell D.T., Fiske D. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 1959, 56:81-105. doi: 10.1037/h0046016.
23. Campbell D.T., Stanley J.C. Experimental and quasi-experimental designs for research 1963, Rand-McNally, Chicago, IL.
24. Cizek G.J., Rosenberg S.L., Koons H.H. Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement 2008, 68:397-412. doi: 10.1177/0013164407310130.
25. Costa P.T., McCrae R.R. NEO Personality Inventory manual 1985, Psychological Assessment Resources, Odessa, FL.
26. Cramer A.O.J., Waldorp L.J., van der Maas H.L.J., Borsboom D. Comorbidity: A network perspective. Behavioral and Brain Sciences 2010, 33:137-150. doi: 10.1017/S0140525X09991567.
27. Cronbach L.J. Test validation. Educational measurement 1971, 443-507. American Council on Education, Washington, DC. 2nd ed. R.L. Thorndike (Ed.).
28. Cronbach L.J., Meehl P.E. Construct validity in psychological tests. Psychological Bulletin 1955, 52:281-302. doi: 10.1037/h0040957.
29. Embretson (Whitely) S.E. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin 1983, 93:179-197. doi: 10.1037/0033-2909.93.1.179.
30. Embretson S.E. Applications of cognitive design systems to test development. Cognitive assessment: A multidisciplinary perspective 1994, 107-135. Plenum Press, New York, NY. C.R. Reynolds (Ed.).
31. Embretson S. A cognitive design system approach for generating valid tests: Approaches to abstract reasoning. Psychological Methods 1998, 3:300-396.
32. Embretson S., Gorin J. Improving construct validity with cognitive psychology principles. Journal of Educational Measurement 2001, 38:343-368.
33. Exner J.E. The Rorschach: A comprehensive system, Volume 2 1991, Wiley, New York, NY. 2nd ed.
34. Garb H.N. Studying the clinician: Judgment research and psychological assessment 1998, American Psychological Association, Washington, DC, doi: 10.1037/10299-000.
35. George G., Stopa L. Private and public self-awareness in social anxiety. Journal of Behavior Therapy and Experimental Psychiatry 2008, 39:57-72. doi: 10.1016/j.jbtep.2006.09.004.
36. Goldman B.A., Mitchel D.F. Directory of unpublished experimental mental measures 2003, 8. American Psychological Association, Washington, DC.
37. Goodwin L.D., Leech N.L. The meaning of validity in the new Standards for Educational and Psychological Testing: Implications for measurement courses. Measurement and Evaluation in Counseling and Development 2003, 36:181-192.
38. Guilford J.P. New standards for test evaluation. Educational and Psychological Measurement 1946, 6:427-438.
39. Hagemann D., Meyerhoff D. A simplified estimation of latent state-trait parameters. Structural Equation Modeling 2008, 15:627-650. doi: 10.1080/10705510802339049.
40. Hänze M., Meyer H.A. Mood influences on automatic and controlled semantic priming. American Journal of Psychology 1998, 111:265-278. doi: 10.2307/1423489.
41. Hershberger S.L. The growth of structural equation modeling: 1994-2001. Structural Equation Modeling 2003, 10:35-46. doi: 10.1207/S15328007SEM1001_2.
42. Hirschfeld R.M.A., Klerman G.L., Clayton P.J., Keller M.B. Personality and depression: Empirical findings. Archives of General Psychiatry 1983, 40:993-998.
43. Hirschfeld R.M.A., Klerman G.L., Gough H.G., Barrett J., Korchin S.J., Chodoff P. A measure of interpersonal dependency. Journal of Personality Assessment 1977, 41:610-618. doi: 10.1207/s15327752jpa4106_6.
44. Hogan T.P., Agnello J. An empirical study of reporting practices concerning measurement validity. Educational and Psychological Measurement 2004, 64:802-812. doi: 10.1177/0013164404264120.
45. Hunsley J., Bailey J.M. The clinical utility of the Rorschach: Unfulfilled promises and an uncertain future. Psychological Assessment 1999, 11:266-277. doi: 10.1037/1040-3590.11.3.266.
46. Jansen B.R.J., Van der Maas H.L.J. Statistical tests of the rule assessment methodology by latent class analysis. Developmental Review 1997, 17:321-357.
47. Jansen B.R.J., Van der Maas H.L.J. The development of children's rule use on the balance scale task. Journal of Experimental Child Psychology 2002, 81:383-416. doi: 10.1006/jecp.2002.2664.
48. Jonson J.L., Plake B.S. A historical comparison of validity standards and validity practices. Educational and Psychological Measurement 1998, 58:736-753. doi: 10.1177/0013164498058005002.
49. Kane M.T. An argument-based approach to validity. Psychological Bulletin 1992, 112:527-535. doi: 10.1037/0033-2909.112.3.527.
50. Kane M.T. Current concerns in validity theory. Journal of Educational Measurement 2001, 38:319-342. doi: 10.1111/j.1745-3984.2001.tb01130.x.
51. Kawada C.L.K., Oettingen G., Gollwitzer P.M., Bargh J.A. The projection of implicit and explicit goals. Journal of Personality and Social Psychology 2004, 86:545-559. doi: 10.1037/0022-3514.86.4.545.
52. Loevinger J. Objective tests as instruments of psychological theory. Psychological Reports 1957, 3:635-694.
53. MacCorquodale K., Meehl P.E. On a distinction between hypothetical constructs and intervening variables. Psychological Review 1948, 55:95-107.
54. Machover K. Personality projection in the drawing of the human figure 1949, Charles C Thomas, Springfield, IL, doi: 10.1037/11147-000.
55. Major B., O'Brien L.T. The social psychology of stigma. Annual Review of Psychology 2005, 56:393-421. doi: 10.1146/annurev.psych.56.091103.070137.
56. Masling J.M. Role-related behavior of the subject and psychologist and its effect upon psychological data. Nebraska symposium on motivation 1966, 67-104. University of Nebraska Press, Lincoln. D. Levine (Ed.).
57. Masling J. Speak, memory, or goodbye, Columbus. Journal of Personality Assessment 2002, 78:4-30. doi: 10.1207/S15327752JPA7801_02.
58. Masling J., Rabie L., Blondheim S.H. Obesity, level of aspiration, and Rorschach and TAT measures of oral dependence. Journal of Consulting Psychology 1967, 31:233-239. doi: 10.1037/h0020999.
59. McClelland D.C., Koestner R., Weinberger J. How do self-attributed and implicit motives differ?. Psychological Review 1989, 96:690-702. doi: 10.1037/0033-295X.96.4.690.
60. McFarland C., Buehler R. The impact of negative affect on autobiographical memory: The role of self-focused attention to moods. Journal of Personality and Social Psychology 1998, 75:1424-1440. doi: 10.1037/0022-3514.75.6.1424.
61. McIntire S.A., Miller L.A. Foundations of psychological testing 2000, McGraw-Hill, Boston, MA.
62. Meehl P.E. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology 1978, 46:806-834. doi: 10.1037/0022-006X.46.4.806.
63. Messick S. Validity. Educational measurement 1989, 13-103. Macmillan, New York, NY. 3rd ed. R.L. Linn (Ed.).
64. Messick S. The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher 1994, 23:13-23.
65. Messick S. Validity of psychological assessment: Validation of inferences from persons' responses and performances as inquiry into score meaning. American Psychologist 1995, 50:741-749. doi: 10.1037/0003-066X.50.9.741.
66. Meyer G.J. The Rorchach and MMPI: Toward a more scientific understanding of cross-method assessment. Journal of Personality Assessment 1996, 67:558-578. doi: 10.1207/s15327752jpa6703_11.
67. Meyer G.J. On the integration of personality assessment methods: The Rorschach and MMPI. Journal of Personality Assessment 1997, 68:297-330. doi: 10.1207/s15327752jpa6802_5.
68. Meyer G.J., Finn S.E., Eyde L.D., Kay G.G., Moreland K.L., Dies R.R., Read G.M. Psychological testing and psychological assessment: A review of evidence and issues. American Psychologist 2001, 56:128-165. doi: 10.1037/0003-066X.56.2.128.
69. Mischel W. Convergences and divergences in the search for consistency. American Psychologist 1984, 39:351-364. doi: 10.1037/0003-066X.39.4.351.
70. Mischel W., Shoda Y., Mendoza-Denton R. Situation-behavior profiles as a locus of consistency in personality. Current Directions in Psychological Science 2002, 11:50-54. doi: 10.1111/1467-8721.00166.
71. Mislevy R.J. Validity by design. Educational Researcher 2007, 36:463-469. doi: 10.3102/0013189X07311660.
72. Murray H.A. Thematic Appreciation Test manual 1943, Harvard University Press, Cambridge, MA.
73. Nosek B.A., Greenwald A.G., Banaji M.R. Understanding and using the Implicit Association Test: Method variables and construct validity. Personality and Social Psychology Bulletin 2005, 31:166-180. doi: 10.1177/0146167204271418.
74. Rholes W.S., Riskind J.H., Lane J.W. Emotional states and memory biases: Effects of cognitive priming and mood. Journal of Personality and Social Psychology 1987, 52:91-99. doi: 10.1037/0022-3514.52.1.91.
75. Robinson M.D., Clore G.L. Episodic and semantic knowledge in emotional self-report: Evidence for two judgment processes. Journal of Personality and Social Psychology 2002, 83:198-215. doi: 10.1037/0022-3514.83.1.198.
76. Rorschach H. Psychodiagnostik 1921, Bircher, Bern, Switzerland.
77. Rosenthal R. Meta-analytic procedures for social research 1991, Sage, Thousand Oaks, CA. 2nd ed.
78. Rosenthal R. Covert communication in laboratories, classrooms, and the truly real world. Current Directions in Psychological Science 2003, 12:151-154. doi: 10.1111/1467-8721.t01-1-01250.
79. Schumacker R.E., Lomax R.G. A beginner's guide to structural equation modeling 2004, Erlbaum, Mahwah, NJ. 2nd ed.
80. Shepard L.A. Evaluating test validity. Review of Research in Education 1993, 19:405-450.
81. Slaney K.L., Maraun M.D. A proposed framework for conducting data-based test analysis. Psychological Methods 2008, 13:376-390. doi: 10.1037/a0014269.
82. Spies R.A., Plake B.S. The sixteenth mental measurements yearbook 2005, Buros Institute of Mental Measurements, Lincoln, NE.
83. Sproull L.S. Managing education programs: A micro-behavioral analysis. Human Organization 1981, 40:113-122.
84. Steele C.M., Aronson J. Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology 1995, 69:797-811. doi: 10.1037/0022-3514.69.5.797.
85. Strauss M.E., Smith G.T. Construct validity: Advances in theory and methodology. Annual Review of Clinical Psychology 2009, 5:1-25.
86. Tomarken A.J., Waller N.G. Structural equation modeling: Strengths, limitations, and misconceptions. Annual Review of Clinical Psychology 2005, 1:31-65. doi: 10.1146/annurev.clinpsy.1.102803.144239.
87. Ullman J.B. Structural equation modeling: Reviewing the basics and moving forward. Journal of Personality Assessment 2006, 87:35-50. doi: 10.1207/s15327752jpa8701_03.
88. Young J.W. Differential validity, differential prediction, and college admissions testing: A comprehensive review and analysis 2001, The College Board, New York, NY.
89. Zemack-Rugar Y., Bettman J.R., Fitzsimons G.J. Effects of nonconsciously priming emotional concepts on behavior. Journal of Personality and Social Psychology 2007, 93:927-939. doi: 10.1037/0022-3514.93.6.927.