Getting Through the Gate: Statistical and Methodological Issues Raised in the Reviewing Process

Organizational Research Methods

Volumn 19, Issue 3, 2016, Pages 402-432

  Green, Jennifer P ^a   Tonidandel, Scott ^b   Cortina, Jose M ^a  

^a GEORGE MASON UNIVERSITY   (United States)

^b DAVIDSON COLLEGE   (United States)

Author keywords

construct validation procedures; factor analysis; interactions; method variance; qualitative research

Indexed keywords

EID: 84974653302     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428116631417     Document Type: Review

References (120)

1. H.Aguinis, S.O.Lawal (2012). Conducting field experiments using eLancing’s natural environment. Journal of Business Venturing, 27, 493–505.
2. H.AguinisC.A.PierceF.A.BoscoI.S.Muslin (2009). First decade of organizational research methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, 69–112. doi:10.1177/1094428108322641
3. L.S.AikenS.G.West (1991). Multiple regression: Testing and interpreting interactions. Thousand Oaks, CA: SAGE Publications, Inc.
4. J.C.AndersonD.W.Gerbing (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411–423.
5. J.AntonakisS.BendahanP.JacquartR.Lalive (2010). On making causal claims: A review and recommendations. Leadership Quarterly, 21, 1086–1120.
6. J.AntonakisJ.Dietz (2011). More on testing for validity instead of looking for it. Personality and Individual Differences, 50, 418–421.
7. ATLAS.ti. (n.d.). ATLAS.ti. (Version 6.2) [Computer software]. Berlin: ATLAS.ti Scientific Software Development GmbH.
8. J.T.AustinC.A.ScherbaumR.A.Mahlman (2002). History of research methods in industrial and organizational psychology: Measurement, design, analysis. In (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 3–33). Malden, MA: Blackwell Publishing.
9. R.AzenD.V.Budescu (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods, 8, 129–148.
10. R.P.BagozziJ.R.Edwards (1998). A general approach for representing constructs in organizational research. Organizational Research Methods, 1, 45–87.
11. R.P.BagozziY.Yi (1990). Assessing method variance in multitrait-multimethod matrices: The case of self-reported affect and perceptions at work. Journal of Applied Psychology, 75, 547–560.
12. R.P.BagozziY.YiL.W.Phillips (1991). Assessing construct validity in organizational research. Administrative Science Quarterly, 36, 421–458.
13. D.L.BandalosM.R.Boehm-Kaufman (2009). Four common misconceptions in exploratory factor analysis. In (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 61–87). New York, NY: Routledge.
14. R.M.BaronD.A.Kenny (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182. doi:10.1037/0022-3514.51.6.1173
15. H.BaumgartnerJ.B.E.Steenkamp (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38, 143–156.
16. T.E.Becker (2005). Potential problems in the statistical control of variables in organizational research: A qualitative analysis with recommendations. Organizational Research Methods, 8, 274–289. doi:10.1177/1094428105278021
17. J.BernerthH.Aguinis (2016). A critical review and best-practice recommendations for control variable usage. Personnel Psychology, 69, 229–283.
18. P.D.BlieseR.E.Ployhart (2002). Growth modeling using random coefficient models: Model building, testing, and illustrations. Organizational Research Methods, 5, 362–387.
19. K.A.BollenR.Lennox (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110, 305–314.
20. J.E.BonoG.McNamara (2011). Publishing in AMJ part 2: Research design. Academy of Management Journal, 54, 657–660. doi:10.5465/AMJ.2011.64869103.
21. M.T.BrannickD.ChanJ.M.ConwayC.E.LanceP.E.Spector (2010). What is method variance and how can we cope with it? A panel discussion. Organizational Research Methods, 13, 407–420.
22. S.BrutusH.AguinisU.Wassmer (2013). Self-reported limitations and future directions in scholarly reports analysis and recommendations. Journal of Management, 39, 48–75.
23. D.T.CampbellD.W.Fiske (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.
24. K.D.CarlsonJ.Wu (2012). The illusion of statistical control: Control variable practice in management research. Organizational Research Methods, 15, 413–435.
25. D.Chan (2009). So why ask me? Are self-report data really that bad? In (Eds.), Statistical and methodological myths and urban legends (pp. 309–336). New York, NY: Routledge.
26. G.W.CheungR.S.Lau (2008). Testing mediation and suppression effects of latent variables: Bootstrapping with structural equation models. Organizational Research Methods, 11, 296–325.
27. L.A.ClarkD.Watson (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309–319.
28. S.M.ClarkD.A.GioiaD.J.KetchenJ.B.Thomas (2010). Transitional identity as a facilitator of organizational identity change during a merger. Administrative Science Quarterly, 55, 397–438.
29. J.Cohen (1978). Partialed products are interactions; partialed powers are curve components. Psychological Bulletin, 85, 858–866.
30. J.Cohen (1992). A power primer. Psychological Bulletin, 112, 155–159.
31. J.CohenP.CohenS.G.WestL.S.Aiken (2003). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, NJ: Erlbaum.
32. D.A.ColeJ.A.CieslaJ.H.Steiger (2007). The insidious effects of failing to include design-driven correlated residuals in latent-variable covariance structure analysis. Psychological Methods, 12, 381–398.
33. A.L.Comrey (1988). Factor-analytic methods of scale development in personality and clinical psychology. Journal of Consulting and Clinical Psychology, 56, 754–761.
34. J.M.ConwayC.E.Lance (2010). What reviewers should expect from authors regarding common method bias in organizational research. Journal of Business and Psychology, 25, 325–334. doi:10.1007/s10869-010-9181-6
35. J.M.Cortina (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98–104.
36. J.M.Cortina (2002). Big things have small beginnings: An assortment of “minor” methodological understandings. Journal of Management, 28, 339–362.
37. J.M.Cortina (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6, 415–439.
38. L.CrockerJ.Algina (1986). Introduction to classical and modern test theory. New York, NY: CBS College Publishing.
39. D.K.DalalM.J.Zickar (2012). Some common myths about centering predictor variables in moderated multiple regression and polynomial regression. Organizational Research Methods, 15, 339–362.
40. J.F.Dawson (2014). Moderation in management research: What, why, when, and how. Journal of Business and Psychology, 29, 1–19. doi:10.1007/s10869-013-9308-7
41. E.I.DesrosiersK.SheronyE.BarrosG.A.BallingerS.SenolM.A.Campion (2002). Writing research articles: Update on the article review checklist. In (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 459–478). Malden, MA: Blackwell Publishing.
42. D.H.DotyW.H.Glick (1998). Common methods bias: Does common methods variance really bias results?Organizational Research Methods, 1, 374–406.
43. J.R.Edwards (2003). Construct validation in organizational behavior research. In (Ed.), Organizational behavior: The state of the science (2nd ed., pp. 327–371). Mahwah, NJ: Erlbaum.
44. J.R.Edwards (2010). Reconsidering theoretical progress in organizational and management research. Organizational Research Methods, 13, 615–619.
45. J.R.EdwardsJ.W.Berry (2010). The presence of something or the absence of nothing: Increasing theoretical precision in management research. Organizational Research Methods, 13, 668–689.
46. J.R.EdwardsL.S.Lambert (2007). Methods for integrating moderation and mediation: A general analytical framework using moderated path analysis. Psychological Methods, 12, 1–22.
47. M.G.Evans (1985). A Monte Carlo study of the effects of correlated method variance in moderated multiple regression analysis. Organizational Behavior and Human Decision Processes, 36, 305–323.
48. L.R.FabrigarD.T.WegenerR.C.MacCallumE.J.Strahan (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.
49. F.FaulE.ErdfelderA.G.LangA.Buchner (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
50. D.W.FiskeL.F.Fogg (1990). But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. American Psychologist, 45, 591–598. doi:10.1037/0003-066X.45.5.591
51. F.J.FloydK.F.Widaman (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286–299.
52. L.FoggD.W.Fiske (1993). Foretelling the judgments of reviewers and editors. American Psychologist, 48, 293–294. doi:10.1037/0003-066X.48.3.293
53. N.Frost (2011). Qualitative research methods in psychology. Berkshire, UK: Open University Press.
54. D.W.GerbingJ.C.Anderson (1984). On the meaning of within-factor correlated measurement errors. Journal of Consumer Research, 11, 572–580.
55. S.W.GillilandJ.M.Cortina (1997). Reviewer and editor decision making in the journal review process. Personnel Psychology, 50, 427–452.
56. D.A.GioiaK.G.CorleyA.L.Hamilton (2013). Seeking qualitative rigor in inductive research notes on the Gioia methodology. Organizational Research Methods, 16, 15–31. doi:10.1177/1094428112452151
57. A.F.Hayes (2009). Beyond Baron and Kenny: Statistical mediation analysis in the new millennium. Communication Monographs, 76, 408–420.
58. A.F.Hayes (2013). Introduction to mediation, moderation, and conditional process analysis. New York, NY: Guilford.
59. R.K.HensonJ.K.Roberts (2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66, 393–416.
60. S.Highhouse (2009). Designing experiments that generalize. Organizational Research Methods, 12, 554–566.
61. T.R.Hinkin (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1, 104–121.
62. D.HofmannM.GriffinM.Gavin (2000). The application of hierarchical linear modeling to organizational research. In (Eds.), Multilevel theory, research, and methods in organizations (pp. 467–511). San Francisco, CA: Jossey-Bass.
63. C.J.HubertyJ.D.Morris (1989). Multivariate analysis versus multiple univariate analyses. Psychological Bulletin, 105, 302–308.
64. L.R.JamesJ.M.Brett (1984). Mediators, moderators, and tests for mediation. Journal of Applied Psychology, 69, 307–321.
65. J.W.Johnson (2000). A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research, 35, 1–19.
66. R.E.JohnsonC.C.RosenC.H.ChangE.DjurdjevicM.U.Taing (2012). Recommendations for improving the construct clarity of higher-order multidimensional constructs. Human Resource Management Review, 22, 62–72.
67. R.E.JohnsonC.C.RosenE.Djurdjevic (2011). Assessing the impact of common method variance on higher order multidimensional constructs. Journal of Applied Psychology, 96, 744–761. doi:10.1037/a0021504
68. D.Kaplan (1990). Evaluating and modifying covariance structure models: A review and recommendation. Multivariate Behavioral Research, 25, 137–155.
69. K.KelleyS.E.Maxwell (2012). Sample size planning. In (Eds.), APA handbook of research methods in psychology: Foundations, planning, measures, and psychometrics (Vol. 1, pp. 181–202). Washington, DC: American Psychological Association.
70. D.J.KetchenR.D.IrelandL.T.Baker (2013). The use of archival proxies in strategic management studies: Castles made of sand?Organizational Research Methods, 16, 32–42.
71. K.J.KleinP.D.BlieseS.W.J.KozolowskiF.DansereauM.B.GavinM.A.Griffin…M.C.Bligh (2000). Multilevel analytical techniques: Commonalities, differences, and continuing questions. In (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 512–553). San Francisco, CA: Jossey-Bass.
72. R.B.Kline (2005). Principles and practice of structural equation modeling (2nd ed.) New York, NY: Guildford Press.
73. S.KoppmanA.Gupta (2014). Navigating the mutual knowledge problem: A comparative case study of distributed work. Information Technology & People, 27, 83–105.
74. A.K.Korman (1974). Contingency approaches to leadership: An overview. In (Eds.), Contingency approaches to leadership. Carbondale, IL: Southern Illinois University Press.
75. S.W.J.Kozlowski (2009). Editorial. Journal of Applied Psychology, 94, 1–4. doi:10.1037/a0014990
76. R.S.LandisB.D.EdwardsJ.M.Cortina (2009). On the practice of allowing correlated residuals among indicators in SEM. In (Eds.), Statistical and methodological myths and urban legends (pp. 193–218). New York, NY: Taylor & Francis Group, LLC.
77. T.M.LeeT.R.MitchellW.S.Harman (2011). Qualitative research strategies in industrial and organizational psychology. In (Ed.), APA handbook of industrial and organizational psychology, Building and developing the organization (Vol. 1, pp. 73–83). Washington, DC: American Psychological Association.
78. K.Locke (2002). The grounded theory approach to qualitative research. In (Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 17–43). San Francisco, CA: Jossey-Bass.
79. R.C.MacCallumM.RoznowskiL.B.Necowitz (1992). Model modification in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111, 490–504.
80. R.C.MacCallumD.T.WegenerB.N.UchinoL.R.Fabrigar (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological Bulletin, 114, 185–199.
81. D.P.MacKinnonS.CoxeA.N.Baraldi (2012). Guidelines for the investigation of mediating variables in business research. Journal of Business and Psychology, 27, 1–14.
82. D.P.MacKinnonC.M.LockwoodJ.M.HoffmannS.G.WestV.Sheets (2002). A comparison of methods to test the significance of mediation and other intervening variable effects. Psychological Methods, 7, 83–104.
83. J.E.MathieuH.AguinisS.A.CulpepperG.Chen (2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97, 951–966.
84. R.McClearyD.McDowall (2012). Time-series designs. In (Eds.), APA handbook of research methods in psychology, Vol. 2: Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 613–627). Washington, DC: American Psychological Association.
85. K.R.MurphyC.J.Russell (2016). Mend it or end it: Redirecting the search for interactions in the organizational sciences. Organizational Research Methods. doi:10.1177/1094428115625322
86. J.B.Nezlek (2001). Multilevel random coefficient analyses of event- and interval-contingent data in social and personality psychology research. Personality and Social Psychology Bulletin, 27, 771–785.
87. B.P.O’Connor (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments, & Computers, 32, 396–402.
88. C.OstroffA.J.KinickiM.A.Clark (2002). Substantive and operational issues of response bias across levels of analysis: an example of climate-satisfaction relationships. Journal of Applied Psychology, 87, 355–368.
89. R.E.PloyhartA.K.Ward (2011). The “quick start guide” for conducting and publishing longitudinal research. Journal of Business and Psychology, 26, 413–422.
90. P.M.PodsakoffS.B.MacKenzieJ.Y.LeeN.P.Podsakoff (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.
91. P.M.PodsakoffS.B.MacKenzieN.P.Podsakoff (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569.
92. P.M.PodsakoffS.B.MacKenzieN.P.Podsakoff (2016). Recommendations for creating better concept definitions in the organizational, behavioral, and social sciences. Organizational Research Methods. Advance online publication. doi:10.1177/1094428115624965
93. I.Pollach (2011). Taming textual data: The contribution of corpus linguistics to computer-aid text analysis. Organizational Research Methods, 15, 263–287. doi:10.1177/10944281114177451
94. M.G.Pratt (2000). The good, the bad, and the ambivalent: Managing identification among Amway distributors. Administrative Science Quarterly, 45, 456–493.
95. K.J.PreacherA.F.Hayes (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36, 717–731.
96. K.J.PreacherA.F.Hayes (2008). Contemporary approaches to assessing mediation in communication research. In (Eds.), The Sage sourcebook of advanced data analysis methods for communication research (pp. 13–54). Thousand Oaks, CA: Sage Publications.
97. K.J.PreacherR.C.MacCallum (2003). Repairing Tom Swift’s electric factor analysis machine. Understanding Statistics: Statistical Issues in Psychology, Education, and the Social Sciences, 2, 13–43.
98. K.J.PreacherD.D.RuckerA.F.Hayes (2007). Addressing moderated mediation hypotheses: Theory, methods, and prescriptions. Multivariate Behavioral Research, 42, 185–227.
99. D.Putka (2011, 831). Re: Construct measurement and validity [Electronic mailing list message]. Retrieved fromhttps://lists.unc.edu/read/?forum=rmnet
100. S.P.ReiseN.G.WallerA.L.Comrey (2000). Factor analysis and scale revision. Psychological Assessment, 12, 287–297.
101. C.RerupM.S.Feldman (2011). Routines as a source of change in organizational schemata: The role of trial-and-error learning. Academy of Management Journal, 54, 577–610. doi:10.5465/AMJ.2011.61968107
102. H.A.RichardsonM.J.SimmeringM.C.Sturman (2009). A tale of three perspectives: Examining post hoc statistical techniques for detection and correction of common method variance. Organizational Research Methods, 12, 762–800. doi:10.1177/1094428109332834
103. S.G.RogelbergM.AdelmanD.Askay (2009). Crafting a successful manuscript: Lessons from 131 reviews. Journal of Business and Psychology, 24, 117–121. doi:10.1007/s10869-009-9116-2
104. M.G.RothsteinR.D.Goffin (2000). The assessment of personality constructs in industrial-organizational psychology. In (Eds.), Problems and solutions in human assessment (pp. 215–248). Norwell, MA: Kluwer Academic Publishers
105. S.R.SardeshmukhR.J.Vandenberg (2016). Integrating moderation and mediation: A structural equation modeling approach. Organizational Research Methods. doi:10.1177/1094428115621609
106. T.A.ScanduraE.A.Williams (2000). Research methodology in management: Current practices, trends, and implications for future research. The Academy of Management Journal, 43, 1248–1264. doi:10.2307/1556348
107. W.R.ShadishT.D.CookD.T.Campbell (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
108. SIOP. (2003). Principles for the validation and use of personnel section tests. Retrieved fromhttp://www.siop.org/_principles/principles.pdf
109. P.E.Spector (2006). Method variance in organizational research truth or urban legend?Organizational Research Methods, 9, 221–232. doi:10.1177/1094428105284955
110. P.E.SpectorM.T.Brannick (2010). Common method issues: An introduction to the feature topic in organizational research methods. Organizational Research Methods, 13, 403–406.
111. P.E.SpectorM.T.Brannick (2011). Methodological urban legends: The misuse of statistical control variables. Organizational Research Methods, 14, 287–305. doi:10.1177/1094428110369842
112. I.StiglianiD.Ravasi (2012). Organizing thoughts and connecting brains: Material practices and the transition from individual to group-level prospective sensemaking. Academy of Management Journal, 55, 1232–1259.
113. E.Stone-RomeroA.E.WeaverJ.L.Glenar (1995). Trends in research design and data analytic strategies in organizational research. Journal of Management, 21, 141–157.
114. A.StraussJ.Corbin (1994). Grounded theory methodology: An overview. In (Eds.), Handbook of qualitative research (pp. 262–272). Thousand Oaks, CA: SAGE Publications, Inc.
115. S.TonidandelJ.M.LeBreton (2011). Relative importance analyses: A useful supplement to multiple regression analyses. Journal of Business and Psychology, 26, 1–9.
116. R.J.VandenbergD.M.Grelle (2009). Alternative model specifications in structural equation modeling: Facts, fictions, and truth. In (Eds.), Statistical and methodological myths and urban legends (pp. 165–191). New York, NY: Taylor & Francis Group, LLC.
117. E.A.WeitzmanM.B.Miles (1995). Computer programs for qualitative data analysis: A software sourcebook. Thousand Oaks, CA: Sage Publications, Inc.
118. L.Wilkinson (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604. doi:10.1037/0003-066X.54.8.594
119. R.E.WoodJ.S.GoodmanN.BeckmannA.Cook (2008). Mediation testing in management research: A review and proposals. Organizational Research Methods, 11, 270–295.
120. W.R.Zwick, W.F.Velicer (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432–442.