Are we really measuring what we say we're measuring? Using video techniques to supplement traditional construct validation procedures

Journal of Applied Psychology

Volumn 98, Issue 1, 2013, Pages 99-113

  Podsakoff, Nathan P ^a   Podsakoff, Philip M ^b   MacKenzie, Scott B ^b   Klinger, Ryan L ^c  

^a UNIVERSITY OF ARIZONA   (United States)

^b INDIANA UNIVERSITY   (United States)

^c OLD DOMINION UNIVERSITY   (United States)

Author keywords

Causality; Construct validation; Experimental methods; Scale development; Video technology

Indexed keywords

ARTICLE; HUMAN; METHODOLOGY; PSYCHOLOGY; REPRODUCIBILITY; TEACHING; VIDEORECORDING;

HUMANS; PATIENT SIMULATION; PSYCHOLOGY, APPLIED; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; VIDEOTAPE RECORDING;

EID: 84880973717     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0029570     Document Type: Article

References (85)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2. Anastasi, A., & Urbina, S. (1997). Psychological testing (7th ed.). Upper Saddle River, NJ: Prentice Hall.
3. Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411-423. doi:10.1037/0033-2909.103.3.411
4. Anderson, J. C., Gerbing, D. W., & Hunter, J. E. (1987). On the assessment of unidimensional measurement: Internal and external consistency, and overall consistency criteria. Journal of Marketing Research, 24, 432- 437. doi:10.2307/3151392
5. Angrist, J. D., & Imbens, G. W. (1995). Two-stage least squares estimation of average causal effects in models with variable treatment intensity. Journal of the American Statistical Association, 90, 431-442.
6. Antonakis, J., Fenley, M., & Liechti, S. (2011). Can charisma be taught? Tests of two interventions. Academy of Management Learning & Education, 10, 374-396. doi:10.5465/amle.2010.0012
7. Aronson, E., & Carlsmith, J. M. (1968). Experimentation in social psychology. In G. Lindzey & E. Aronson (Eds.), The handbook of social psychology (2nd ed., Vol. 2, pp. 1-79). Reading, MA: Addison-Wesley.
8. Bagozzi, R. P. (1980). Causal models in marketing. New York, NY: Wiley.
9. Bagozzi, R. P., Yi, Y., & Phillips, L. W. (1991). Assessing construct validity in organizational research. Administrative Science Quarterly, 36, 421-458. doi:10.2307/2393203
10. Barrick, M. R., Mount, M. K., & Judge, T. A. (2001). Personality and job performance at the beginning of the new millennium: What do we know and where do we go next? International Journal of Selection and Assessment, 9, 9-30. doi:10.1111/1468-2389.00160
11. Bollen, K. A. (1989). Structural equations with latent variables. New York, NY: Wiley.
12. Bollen, K. A., & Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110, 305-314. doi:10.1037/0033-2909.110.2.305
13. Borsboom, D. (2009). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge, England: Cambridge University Press.
14. Borsboom, D., Mellenburgh, G. J., & Van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071. doi:10.1037/0033- 295X.111.4.1061
15. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. doi:10.1037/h0046016
16. Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Newbury Park, CA: Sage.
17. Cash, T. F., & Kilcullen, R. N. (1985). The eye of the beholder: Susceptibility to sexism and beautyism in the evaluation of managerial applicants. Journal of Applied Social Psychology, 15, 591-605. doi:10.1111/ j.1559-1816.1985.tb00903.x
18. Churchill, G. A., Jr. (1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16, 64-73. doi: 10.2307/3150876
19. Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309-319. doi:10.1037/1040-3590.7.3.309
20. Coleman, M., & Liau, T. L. (1975). A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60, 283- 284. doi:10.1037/h0076540
21. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302. doi:10.1037/h0040957
22. DeGroot, T., & Motowidlo, S. J. (1999). Why visual and vocal interview cues can affect interviewers' judgments and predict job performance. Journal of Applied Psychology, 84, 986 -993. doi:10.1037/0021- 9010.84.6.986
23. Dehejia, R. H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and Statistics, 84, 151-161. doi:10.1162/003465302317331982
24. DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park, CA: Sage.
25. Diamantopoulos, A., & Winklhofer, H. M. (2001). Index construction with formative indicators: An alternative to scale development. Journal of Marketing Research, 38, 269-277. doi:10.1509/jmkr.38.2.269.18845
26. Edwards, J. R. (2001). Multidimensional constructs in organizational behavior research: An integrative analytic framework. Organizational Research Methods, 4, 144-192. doi:10.1177/109442810142004
27. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215-251. doi:10.1037/0033-295X.87.3.215
28. Ferris, G. R., & Judge, T. A. (1991). Personnel human-resources management: A political influence perspective. Journal of Management, 17, 447-488. doi:10.1177/014920639101700208
29. Ferris, G. R., Judge, T. A., Chachere, J. G., & Liden, R. C. (1991). The age context of performance-evaluation decisions. Psychology and Aging, 6, 616-622. doi:10.1037/0882-7974.6.4.616
30. Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51, 327-358. doi:10.1037/h0061470
31. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32, 221-233. doi:10.1037/h0057532
32. Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2011). Investigating the substantive aspect of construct validity for the Satisfaction With Life Scale Adapted for Children: A focus on cognitive processes. Social Indicators Research, 100, 37-60. doi:10.1007/s11205-010-9603-x
33. Gerbing, D. W., & Anderson, J. C. (1988). An updated paradigm for scale development incorporating unidimensionality and its assessment. Journal of Marketing Research, 25, 186-192. doi:10.2307/3172650
34. Gilbreth, F. B. (1912). The present state of the art of industrial management. Transactions of the American Society of Mechanical Engineers, 34, 1224-1226.
35. Gunning, R. (1952). The technique of clear writing. New York, NY: McGraw-Hill.
36. Highhouse, S. (2009). Designing experiments that generalize. Organizational Research Methods, 12, 554-566. doi:10.1177/1094428107300396
37. Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21, 967-988. doi:10.1177/ 014920639502100509
38. Hinkin, T. R., & Tracey, J. B. (1999). An analysis of variance approach to content validation. Organizational Research Methods, 2, 175-186. doi: 10.1177/109442819922004
39. Hogan, R., Hogan, J., & Roberts, B. W. (1996). Personality measurement and employment decisions. American Psychologist, 51, 469-477. doi: 10.1037/0003-066X.51.5.469
40. Hollandsworth, J. G., Jr., Kazelskis, R., Stevens, J., & Dressel, M. E. (1979). Relative contributions of verbal, articulative, and nonverbal communication to employment decisions in the job interview setting. Personnel Psychology, 32, 359-367. doi:10.1111/j.1744-6570.1979.tb02140.x
41. Howell, J. M., & Frost, P. J. (1989). A laboratory study of charismatic leadership. Organizational Behavior and Human Decision Processes, 43, 243-269. doi:10.1016/0749-5978(89)90052-6
42. Imada, A. S., & Hakel, M. D. (1977). Influence of nonverbal communication and rater proximity on impression and decisions in simulated employment interviews. Journal of Applied Psychology, 62, 295-300. doi:10.1037/0021-9010.62.3.295
43. Jarvis, C. B., MacKenzie, S. B., & Podsakoff, P. M. (2003). A critical review of construct indicators and measurement model misspecification in marketing and consumer research. Journal of Consumer Research, 30, 199-218. doi:10.1086/376806
44. Johnson, J. A. (1997). Units of analysis for the description and explanation of personality. In R. Hogan, J. Johnson, & S. Briggs (Eds.), Handbook of personality psychology (pp. 73-93). San Diego, CA: Academic Press.
45. Judge, T. A., & Cable, D. M. (2004). The effect of physical height on workplace success and income. Journal of Applied Psychology, 89, 428-441. doi:10.1037/0021-9010.89.3.428
46. Judge, T. A., & Cable, D. M. (2011). When it comes to pay, do the thin win? The effect of weight on pay for men and women. Journal of Applied Psychology, 96, 95-112. doi:10.1037/a0020860
47. Judge, T. A., Hurst, C., & Simon, L. S. (2009). Does it pay to be smart, attractive, or confident (or all three)? Relationships among general mental ability, physical attractiveness, core self-evaluations, and income. Journal of Applied Psychology, 94, 742-755. doi:10.1037/ a0015497
48. Kelley, T. L. (1927). Interpretation of educational measurements. New York, NY: Oxford University Press.
49. Kerlinger, F. N., & Lee, H. B. (2000). Foundations of behavioral research (4th ed.). New York, NY: Harcourt.
50. Kinicki, A. J., & Lockwood, C. A. (1985). The interview process: An examination of factors recruiters use in evaluating job applicants. Journal of Vocational Behavior, 26, 117-125. doi:10.1016/0001- 8791(85)90012-0
51. Locke, E. A. (1969). What is job satisfaction? Organizational Behavior and Human Performance, 4, 309 -336. doi:10.1016/0030- 5073(69)90013-0
52. MacKenzie, S. B. (2003). The dangers of poor construct conceptualization. Journal of the Academy of Marketing Science, 31, 323-326. doi: 10.1177/0092070303031003011
53. MacKenzie, S. B., Podsakoff, P. M., & Jarvis, C. B. (2005). The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology, 90, 710-730. doi:10.1037/0021-9010.90.4.710
54. MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavior research: Integrating new and existing techniques. MIS Quarterly, 35, 293-334.
55. McLaughlin, G. H. (1969). SMOG Grading-A new readability formula. Journal of Reading, 12, 639-646.
56. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York, NY: Macmillan.
57. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
58. Meyer, B. D. (1999). Natural and quasi-experiments in economics. Journal of Business & Economic Statistics, 13, 151-161.
59. Motowidlo, S. J., & Burnett, J. R. (1995). Aural and visual sources of validity in structured employment interviews. Organizational Behavior and Human Decision Processes, 61, 239 -249. doi:10.1006/ obhd.1995.1019
60. Nederhof, A. J. (1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology, 15, 263-280. doi: 10.1002/ejsp.2420150303
61. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw Hill.
62. Pingitore, R., Dugoni, B. L., Tindale, R. S., & Spring, B. (1994). Bias against overweight job applicants in a simulated employment interview. Journal of Applied Psychology, 79, 909 -917. doi:10.1037/0021- 9010.79.6.909
63. Podsakoff, N. P., Whiting, S. W., Podsakoff, P. M., & Mishra, P. (2011). Effects of organizational citizenship behaviors on selection decisions in employment interviews. Journal of Applied Psychology, 96, 310-326. doi:10.1037/a0020948
64. Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879-903. doi:10.1037/0021-9010.88.5.879
65. Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539-569. doi:10.1146/ annurev-psych-120710-100452
66. Porter, M. E. (1998). Competitive strategy. New York, NY: Free Press.
67. Robinson, S. L., & Bennett, R. J. (1995). A typology of deviant workplace behaviors: A multidimensional scaling study. Academy of Management Journal, 38, 555-572. doi:10.2307/256693
68. Scandura, T. A., & Williams, E. A. (2000). Research methodology in management: Current practices, trends, and implications for future research. Academy of Management Journal, 43, 1248-1264. doi:10.2307/ 1556348
69. Schoenfeldt, L. F. (1984). Psychometric properties of organizational research instruments. In T. S. Bateman & G. R. Ferris (Eds.), Method and analysis in organizational research (pp. 68-80). Reston, VA: Reston Publishing.
70. Schriesheim, C. A., Powers, K. J., Scandura, T. A., Gardiner, C. C., & Lankau, M. J. (1993). Improving construct measurement in management research: Comments and a quantitative approach for assessing the theoretical adequacy of paper-and-pencil survey type instruments. Journal of Management, 19, 385-417.
71. Schwab, D. P. (1980). Construct validity in organizational behavior. In B. M. Staw & L. L. Cummings (Eds.), Research in organizational behavior (Vol. 2, pp. 3-43). Greenwich, CT: JAI Press.
72. Schwab, D. P. (1999). Research methods for organizational studies. Mahwah, NJ: Erlbaum.
73. Schwenk, C. R. (1982). Why sacrifice rigor for relevance: A proposal for combining laboratory and field research in strategic management. Strategic Management Journal, 3, 213-225. doi:10.1002/smj.4250030304
74. Shadish, W. R., & Cook, T. D. (1999). Comment-design rules: More steps toward a complete theory of quasi-experimentation. Statistical Sciences, 14, 294-300.
75. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2000). Experimental and quasi-experimental designs for generalized causal inference. Berkeley, CA: Houghton Mifflin.
76. Spector, P. E. (1992). Summated rating scales: An introduction. Newbury Park, CA: Sage.
77. Stogdill, R. M. (1969). Validity of leader behavior descriptions. Personnel Psychology, 22, 153-158. doi:10.1111/j.1744-6570.1969.tb02298.x
78. Storms, M. D. (1973). Videotape and the attribution process: Reversing actors and observers points of view. Journal of Personality and Social Psychology, 27, 165-175. doi:10.1037/h0034782
79. Towler, A. J. (2003). Effects of charismatic influence training on attitudes, behavior, and performance. Personnel Psychology, 56, 363-381. doi: 10.1111/j.1744-6570.2003.tb00154.x
80. Venkatraman, N., & Grant, J. H. (1986). Construct measurement in organizational strategy research: A critique and proposal. The Academy of Management Review, 11, 71-87.
81. West, S. G., & Thoemmes, F. (2010). Campbell's and Rubin's perspectives on causal inference. Psychological Methods, 15, 18-37. doi:10.1037/ a0015917
82. Wilson, T. D., Aronson, E., & Carlsmith, K. (2010). The art of laboratory experimentation. In S. Fiske, D. T. Gilbert, & G. Lindzey (Eds.), Handbook of social psychology (5th ed., pp. 49-79). Hoboken, NJ: Wiley.
83. Yao, G., Wu, C.-H., & Yang, C.-T. (2008). Examining the content validity of the WHOQOL-BREF from respondents' perspective by quantitative methods. Social Indicators Research, 85, 483-498. doi:10.1007/s11205- 007-9112-8
84. Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp. 45-79). Amsterdam, the Netherlands: Elsevier Science.
85. Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In Robert W. Lissitz (Ed.), The concept of validity: Revisions, new directions, and applications (pp. 65-82). Charlotte, NC: Information Age.