Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data approach using ex-ante and ex-post peer evaluation data from the Austrian science fund

Journal of the Association for Information Science and Technology

Volumn 66, Issue 11, 2015, Pages 2321-2339

  Mutz, Rüdiger ^a   Bornmann, Lutz ^b   Daniel, Hans Dieter ^c  

^a ETH ZURICH   (Switzerland)

^b MAX PLANCK INSTITUTE   (Germany)

^c UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

administrative records; funding; statistics

Indexed keywords

DECISION MAKING; STATISTICS;

ADMINISTRATIVE RECORDS; DECISION MAKING PROCEDURE; FUNDING; MISSING DATA IMPUTATIONS; MULTIPLE IMPUTATION; RESEARCH FUNDING; RESEARCH PROPOSALS; VERIFICATION BIAS;

FINANCE;

CONSENSUS DEVELOPMENT; FEMALE; FUNDING; GENDER; HUMAN; MAJOR CLINICAL STUDY; MALE; MULTILEVEL ANALYSIS; PEER REVIEW; PREDICTION; PREDICTIVE VALIDITY; REST; THEORETICAL MODEL;

EID: 84951753890     PISSN: 23301635     EISSN: 23301643     Source Type: Journal    
DOI: 10.1002/asi.23315     Document Type: Article

References (62)

1. Aguinis, H., Culpepper, S.A., &, Pierce, C.A., (2010). Revival of test bias research in preemployment testing. Journal of Applied Psychology, 95 (4), 648-680.
2. Aguinis, H., &, Stone-Romero, E.F., (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82 (1), 192-206.
3. Azur, M.J., Stuart, E.A., Frangakis, C., &, Leaf, P.J., (2011). Multiple imputation by chained equations: What is it and how does it works ? International Journal of Methods in Psychiatric Research, 20 (1), 40-49.
4. Bartlett, C.J., Bobko, P., Mosier, S.B., &, Hannan, R., (1978). Testing for fairness with a moderated multiple regression strategy: An alternative to differential analysis. Personnel Psychology, 31, 233-242.
5. Bauer, D.J., (2009). A note on comparing the estimates of models for cluster-correlated or longitudinal data with binary or ordinal outcomes. Psychometrika, 74 (1), 97-105.
6. Bliese, P.D., (2000). Within-group agreement, non-independence, and reliability-Implications for data aggregation and analysis. In, K.J. Klein, &, S.W. Kozolowski, (Eds.), Multilevel theory, research, and methods in organizations. San Francisco: Jossey-Bass.
7. Bliese, P.D., &, Halverson, R.R., (2002). Using random group resampling in multilevel research. An example of the buffering effects of leadership climate. The Leadership Quarterly, 13, 53-68.
8. Bliese, P.D., Halverson, R.R., &, Rothberg, J.M., (1994). Within-group agreement scores: Using resampling procedures to estimate expected variance. Academy of Management Best Paper Proceedings, 1, 303-307.
9. Bornmann, L., (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199-245.
10. Bornmann, L., &, Daniel, H.-D., (2005). Committee peer review at an international research foundation. Predictive validity and fairness of selection decisions on post-graduate fellowship applications. Research Evaluation, 14 (1), 15-20.
11. Bornmann, L., &, Daniel, H.-D., (2008). Selecting manuscripts for a high impact journal through peer review: A citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59 (11), 1841-1852.
12. Bornmann, L., Mutz, R., Marx, W., Schier, H., &, Daniel, H.-D., (2011). A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high-profile journal select manuscripts that are highly cited after publication? Journal of the Royal Statistical Society-Series A: Statistics in Society, 174 (4), 857-879.
13. Bornmann, L., Weymuth, C., &, Daniel, H.-D., (2010). A content analysis of referees' comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? Scientometrics, 83 (2), 493-506.
14. Cleary, T.A., (1968). Test bias: Prediction of grades of negros and white students in integrated colleges. Journal of Educational Measurement, 5 (2), 115-124.
15. Cole, N.S., (1973). Bias in selection. Journal of Educational Measurement, 10 (4), 237-255.
16. Culpepper, S.A., &, Davenport, E.C., (2009). Assessing differential prediction of college grades by race/ethnicity with a multilevel model. Journal of Educational Measurement, 46 (2), 220-242.
17. Daniel, H.-D., (2004). Guardians of science-Fairness and reliability of peer review. Weinheim, Germany: Wiley-VCH.
18. De Groot, J.A., Janssen, K.J.M., Zwinderman, A.H., Moons, K.G.M., &, Reitsma, J.B., (2008). Multiple imputation to correct for partial verification bias: A revision of literature. Statistics in Medicine, 28 (5), 5880-5889.
19. Demirtas, H., Freels, S.A., &, Yucel, R.M., (2008). Plausibility of multivariate normality assumption when multiply imputing non-Gaussian continuous outcomes: A simulation assessment. Journal of Statistical Computation and Simulation, 78 (1), 69-84.
20. Demirtas, H., &, Hedeker, D., (2008). Imputing continuous data under some non-Gaussian distributions. Statistica Neerlandica, 62 (2), 193-205.
21. Dinges, M., (2005). The Austrian Science Fund: Ex post evaluation and performance of FWF funded research projects InTeReg Research Report No. 42-2005. Vienna: Institute of Technology and Regional Policy.
22. Draper, N.R., &, Smith, H., (1998). Applied regression (3rd ed.). New York: Wiley.
23. Enders, C.K., (2010). Applied missing data analysis. New York: Guilford Press.
24. Fischer, C., &, Reckling, F., (2010). Factors influencing approval probability in FWF decision-making procedures. Vienna: Fonds zur Förderung der wissenschaftlichen Forschung (FWF).
25. Graham, J.W., (2012). Missing data-Analysis and design. New York: Springer.
26. Graham, J.W., Olchowski, A.E., &, Gilreath, T.D., (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory? Preventive Science, 8 (3), 206-213.
27. Harel, O., &, Zhou, X.-H., (2006). Multiple imputation for correction verification bias. Statistics in Medicine, 25 (22), 3769-3786.
28. Hox, J., (2010). Multilevel analysis-Techniques and applications (2nd ed.). New York: Routledge.
29. Hunter, J.E., &, Schmidt, F.L., (1976). Critical analysis of the statistical and ethical implications of various definitions of test bias. Psychological Bulletin, 83, 1053-1071.
30. Irvine, J., &, Martin, B.R., (1984). Foresight in science: Picking the winners. London: Pinter.
31. wg: An assessment of within-group agreement. Journal of Applied Psychology, 78 (2), 306-309.
32. Jayasinghe, U.W., Marsh, H.W., &, Bond, N., (2001). Peer review in the funding of research in higher education. Educational Evaluation and Policy Analysis, 23 (4), 343-364.
33. Jayasinghe, U.W., Marsh, H.W., &, Bond, N., (2003). A multilevel cross-classified modelling approach to peer review of grant proposals: The effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society Series A-Statistics in Society, 166 (Part 3), 279-300.
34. LaLonde, S.M., (2012). Transforming variables for normality and linearity-when, how, why and why not's SAS-Global Forum-Paper. Cary, NC: SAS Institute.
35. Lautenschlager, G.J., &, Mendoza, J.L., (1994). A step-down hierarchical multiple regression analysis for examining hypotheses about test bias in prediction. Psychological Measurement, 10, 133-139.
36. Lee, C.J., Sugimoto, C.R., Zhang, G., &, Cronin, B., (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64 (1), 2-17.
37. Linn, R.L., &, Werts, C.E., (1971). Considerations for studies of test bias. Journal of Educational Measurement, 8, 1-4.
38. Little, R.J., &, Rubin, D.B., (2002). Statistical analysis with missing data. Hoboken, NJ: John Wiley & Sons.
39. Mackinnon, A., (2010). The use and reporting of multiple imputation in medical research-A review. Journal of Internal Medicine, 268 (6), 586-593.
40. Maessen, K.M.H., (2012). Comment to the article by van Arensbergen and van den Besselaar the selection of scientific talent in the allocation of research grants. Higher Education Policy, 25 (4), 531-534.
41. Marsh, H.W., Jayasinghe, U.W., &, Bond, N.W., (2008). Improving the peer-review process for grant applications-Reliability, validity, bias, and generalizability. American Psychologist, 63, 160-168.
42. Meade, A.W., &, Fetzer, M., (2009). Test bias, differential prediction, and a revised approach for determining the suitability of a predictor in a selection context. Organizational Research Methods, 12 (4), 738-761.
43. Meade, A.W., &, Tonidal, S., (2010a). Final thoughts on measurement bias and differential prediction. Industrial and Organizational Psychology, 3, 192-205.
44. Meade, A.W., &, Tonidal, S., (2010b). Not seeing clearly with Cleary. What test bias analyses do not and do tell us. Industrial and Organizational Psychology, 3, 192-205.
45. Muenchen, R.A., (2011). R for SAS and SPSS users. New York: Springer.
46. Mutz, R., Bornmann, L., &, Daniel, H.-D., (2012a). Does gender matter in grant peer review? An empirical investigation using the example of the Austrian Science Fund. Zeitschrift für Psychologie / Journal of Psychology, 220 (2), 121-129.
47. Mutz, R., Bornmann, L., &, Daniel, H.-D., (2012b). Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: A generalized estimating equations approach. PLoS ONE, 7 (10).
48. Novick, M.R., (1966). The axioms and principal results of classical test theory. Journal of Mathematical Psychology, 3(1-18).
49. Oswald, F.L., Saad, S., &, Sackett, P.R., (2000). The homogeneity assumption in differential prediction analysis: Does it really matter? Journal of Applied Psychology, 85 (4), 536-541.
50. Reinhart, M., (2009). Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics, 81 (3), 789-809.
51. Sackett, P.R., Laczo, R.M., &, Lippe, Z., (2003). Differential prediction and the use of multiple predictors: The omitted variable problem. Journal of Applied Psychology, 88 (6), 1046-1056.
52. SAS Institute. (2011). SAS/STAT 9.3 user's guide (2nd ed.). Cary, NC: SAS Institute.
53. Seaman, S.R., Bartlett, J.W., &, White, I.R., (2012). Multiple imputation of missing covariates with non-linear effects and interactions: An evaluation of statistical methods. Medical Research Methodology, 12 (46).
54. Snijders, T.A.B., &, Bosker, R.J., (2012). Multilevel analysis (2nd ed.). London: Sage.
55. Sterne, J.A.C., White, I.R., Carlin, J.B., Spratt, M., Royston, P., Kenward, M.G.,... (2009). Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ, 338, b2393.
56. Thorndike, R.L., (1971). Concepts of cultural fairness. Journal of Educational Measurement, 8 (2), 63-70.
57. Van Buuren, S., (2011). Multiple imputation of multilevel data. In, J. Hox, &, J.K. Roberts, (Eds.), Handbook of advanced multilevel analysis (pp. 173-196). New York: Routledge Taylor & Francis.
58. Van Buuren, S., (2012). Flexible imputation of missing data. New York: Chapman & Hall.
59. Van Buuren, S., &, Grootshuis-Outdhoorn, K., (2011). Mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 15 (3), 1-67.
60. Van den Besselaar, P., &, Leydesdorff, L., (2009). Past performance, peer review and project selection: A case study in the social and behavioral sciences. Research Evaluation, 18 (4), 273-288.
61. Von Hippel, P.T., (2009). How to impute interactions, squares, and other transformed variables. Sociological Methodology, 39 (1), 265-291.
62. Yucel, R.M., (2011). State of the multiple imputation software. Journal of Statistical Software, 45 (1), 1-7.