Designing experiments that generalize

Organizational Research Methods

Volumn 12, Issue 3, 2009, Pages 554-566

  Highhouse, Scott ^a  

^a BOWLING GREEN STATE UNIVERSITY   (United States)

Author keywords

Construct validity; Experimental design; Generalizability; Stimulus sampling

Indexed keywords

EID: 67649550548     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428107300396     Document Type: Article

References (58)

1. Abelson, R.P. (1997). On the surprising longevity of flogged horses: Why there is a case for the significance test. Psychological Science, 8, 12-15.
2. Anderson, C.A., & Bushman, B.J. (1997). External validity of ''trivial'' experiments: The case of laboratory aggression. Review of General Psychology, 1, 19-41. (Pubitemid 127547052)
3. Austin, J., Scherbaum, C., & Mahlman (2002). History of research methods in industrial organizational psychology: Measurement, design, analysis. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 77-98). Malden, MA: Blackwell.
4. Bandura, A., & Locke, E.A. (2003). Negative self-efficacy and goal effects revisited. Journal of Applied Psychology, 88, 87-99. (Pubitemid 36385695)
5. Berkowitz, L., & Donnerstein, E. (1982). External validity is more than skin deep: Some answers to criticisms of laboratory experiments. American Psychologist, 37, 245-257.
6. Brinberg, D., & McGrath, J.E. (1985). Validity and the research process. Beverly Hills, CA: Sage.
7. Brunswik, E. (1944). Distal focusing of perception: Size constancy in a representative sample of situations. Psychological Monographs, 56, 1-49.
8. Calder, B.J., Phillips, L.W., & Tybout, A.M. (1981). Designing research for application. Journal of Consumer Research, 8, 197-207.
9. Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
10. Campbell, D.T. & Stanley, J.C. (1967). Experimental and quasi-experimental designs in research. Chicago: Rand McNally.
11. Campbell, J.P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. Handbook of industrial and organizational psychology. Palo Alto, CA: Consulting Psychologists Press.
12. Chow, S.L. (1988). Significance test or effect size ? Psychological Bulletin, 103, 105-110.
13. Chow, S.L. (1996). Statistical significance: Rationale, validity, and utility. Thousand Oaks, CA: Sage.
14. Chow, S.L. (1997). Science, ecological validity and experimentation. Journal of the Theory of Social Behaviour, 17, 181-194.
15. Clark, H.H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12, 335-359.
16. Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation design and analysis. New York: Guilford.
17. Dhami, M.K., Hertwig, R., & Hoffrage, U. (2004). The role of representative design in an ecological approach to cognition. Psychological Bulletin, 130, 959-988. (Pubitemid 39508395)
18. Dipboye, R.L. (1990). Laboratory vs. field research in industrial-organizational psychology. In C. L. Cooper & I. T. Robertson (Eds.), International review of industrial and organizational psychology (Vol. 5, pp. 1-34). New York: Wiley.
19. Dobbins, G.H., Lane, I.M., & Steiner, D.D. (1988). A note on the role of laboratory methodologies in applied behavioural research: Don't throw out the baby with the bath water. Journal of Organizational Behavior, 9, 281-286.
20. Farnsworth, P.R., & Misumi, I. (1931). Further data on suggestion in pictures. American Journal of Psychology, 43, 632.
21. Fern, E.F., & Monroe, K.B. (1996). Effect-size estimates: Issues and problems in interpretation. Journal of Consumer Research, 23, 89-105.
22. Festinger, L. (1953). Laboratory experiments. In L. Festinger and D. Katz (Eds.), Research methods in the behavioral sciences (pp. 136-172). New York: Rinehart and Windston.
23. Fontenelle, G.A., Phillips, A.P., & Lane, D.M. (1985). Generalizing across stimuli as well as subjects: A neglected aspect of external validity. Journal of Applied Psychology, 70, 101-107.
24. Fromkin, H.L., & Streufert, S. (1976). Laboratory experimentation. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 415-465). Chicago: Rand McNally.
25. Garner, W.R., Hake, H.W., & Eriksen, C.W. (1956). Operationism and the concept of perception. Psychological Review, 63, 149-159.
26. Grace, R.C. (2001). On the failure of operationism. Theory & Psychology, 11, 5-33. (Pubitemid 33736603)
27. Greenberg, J., & Tomlinson, E.C. (2004). Situated experiments in organizations: Transplanting the lab to the field. Journal of Management, 30, 703-724. (Pubitemid 38999042)
28. Griffin, R., & Kacmar, K.M. (1991). Laboratory research in management: Misconceptions and missed opportunities. Journal of Organizational Behavior, 12, 301-311.
29. Haack, S. (2003). Defending science-within reason: Between scientism and cynicism. Amherst, MA: Prometheus.
30. Hammond, K.R. (1948). Subject and object sampling: A note. Psychological Bulletin, 45, 530-533.
31. Hammond, K.R. (1996). Upon reflection. Thinking and Reasoning, 2, 239-248. (Pubitemid 126097537)
32. Highhouse, S., Luong, A., & Sarkar, S. (1999). Research design, measurement, and effects of attribute range on job choice: More than meets the eye. Organizational Research Methods, 2, 37-49.
33. Ilgen, D.R. (1986). Laboratory research: A question of when, not if. In E. A. Locke (Ed.), Generalizing from laboratory to field settings (pp. 257-267). Indianapolis, IN: D.C. Heath.
34. James, L.R., Mulaik, S.A., & Brett, J. (1982). Causal analysis. Models, assumptions and data. Beverly Hills, CA: Sage.
35. Johns, G., Xie, J.L., & Fang, Y. (1992). Mediating and moderating effects in job design. Journal of Management, 18, 657-676.
36. Kaiser, H.F. (1960). Directional statistical decisions. Psychological Review, 67, 160-167.
37. Kluger, A.N., & DeNisi, A. (1996). The effects of feedback interventions on performance: Historical review, meta-analysis and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284. (Pubitemid 126042646)
38. Labella, C., & Koehler, D.J. (2004). Dilution and confirmation of probability judgments based on nondiagnostic evidence. Memory & Cognition, 32, 1076-1086.
39. Leventhal, L., & Huynh, C.L. (1996). Directional decisions for two-tailed tests: Power, error rates, and sample size. Psychological Methods, 1, 278-292. (Pubitemid 126670225)
40. Locke, E.A. (1986). Generalizing from laboratory to field: Ecological validity or abstraction of essential elements? In E. A. Locke (Ed.), Generalizing from laboratory to field settings (pp. 257-267). Indianapolis, IN: D.C. Heath.
41. Mook, D.G. (1983). In defense of external invalidity. American Psychologist, 38, 379-387.
42. Nisbett, R.E., Zukier, H., & Lemley, R.E. (1981). The dilution effect: Nondiagnostic information weakens the implications of diagnostic information. Cognitive Psychology, 13, 248-277.
43. Ofir, C., & Mazursky, D. (1997). Does a surprising outcome reinforce or reverse the hindsight bias ? Organizational Behavior and Human Decision Processes, 69, 51-57. (Pubitemid 127340852)
44. Perdue, B.C., & Summers, J.O. (1986). Checking the success of manipulations in marketing experiments. Journal of Marketing Research, 23, 317-326.
45. Peters, E., & Rothbart, M. (2000). Typicality can create, eliminate, and reverse the dilution effect. Personality and Social Psychology Bulletin, 26, 177-187.
46. Peterson, R.A. (2001). On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research, 28, 450-461. (Pubitemid 33403469)
47. Podsakoff, P.M., & Dalton, D.R. (1987). Research methodology in organizational studies. Journal of Management, 13, 419-441.
48. Prentice, D.A., & Miller, D.T. (1992). When small effects are impressive. Psychological Bulletin, 112, 160-164.
49. Rosenthal, R., & Rosnow, R.L. (1991). Essentials of behavioral research: Methods and data analysis. New York: McGraw-Hill.
50. Scandura, T.A., & Williams, E.A. (2000). Research methodology in management: Current practices, trends, and implications for future research. Academy of Management Journal, 43, 1248-1264. (Pubitemid 33216846)
51. Schlenker, B.R., & Bonoma, T.V. (1978). ''Fun and games'': The validity of games for the study of conflict. Journal of Conflict Resolution, 22, 7-38.
52. Schwab, D.P. (2005). Research methods in organizational research (2 nd ed.). Mahwah, NJ: Lawrence Erlbaum.
53. Shapiro, M.A. (2002). Generalizability in communication research. Human Communication Research, 28, 491-500. (Pubitemid 37496215)
54. Slattery, J.P., & Ganster, D.C. (2002). Determinants of risk taking in a dynamic uncertain context. Journal of Management, 28, 89-106.
55. Stone-Romero, E.F. (2002). The relative validity and usefulness of various empirical research designs (77-98). In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology. Malden, MA: Blackwell.
56. Vancouver, J.B. (2005). The depth of history and explanation as benefit and bane for psychological control theories. Journal of Applied Psychology, 90, 38-52. (Pubitemid 40176547)
57. Weick, K.E. (1965). Laboratory experimentation with organizations. In J. G. March (Ed.), Handbook of organizations (pp. 194-260). Chicago: Rand McNally.
58. Wells, G.L., & Windschitl, P.D. (1999). Stimulus sampling and social psychological experimentation. Personality and Social Psychology Bulletin, 25, 1115-1125.