Faculty Mentors’, Graduate Students’, and Performance-Based Assessments of Students’ Research Skill Development

American Educational Research Journal

Volumn 52, Issue 2, 2015, Pages 334-370

  Feldon, David F ^a   Maher, Michelle A ^b   Hurst, Melissa ^c   Timmerman, Briana ^b  

^a UTAH STATE UNIVERSITY   (United States)

^b UNIVERSITY OF SOUTH CAROLINA   (United States)

^c UNIVERSITY OF VIRGINIA   (United States)

Author keywords

assessment; graduate education; research skills; self assessment

Indexed keywords

EID: 84925871977     PISSN: 00028312     EISSN: 19351011     Source Type: Journal    
DOI: 10.3102/0002831214549449     Document Type: Article

References (111)

1. Ahern K.,Manathunga C.Clutch-starting stalled research students.Innovative Higher Education. 2004;28:237-254
2. Åkerlind G. S.Variation and commonality in phenomenographic research methods.Higher Education Research and Development. 2005;24:321-334
3. Anderson M. S.,Swazey J. P.Reflections on the graduate student experience: An overview.New Directions for Higher Education. 1998;101:3-13
4. Austin A. E.Preparing the next generation of faculty: Graduate school as socialization to the academic career.The Journal of Higher Education. 2002;73:94-122
5. Austin A. E.,McDaniels M.Preparing the professoriate of the future: Graduate student socialization for faculty roles.Higher Education: Handbook of Theory and Research. 2006;21:397-456
6. Barnes B. J.,Austin A. E.The role of doctoral advisors: A look at advising from the advisor’s perspective.Innovative Higher Education. 2009;33:297-315
7. Baruch Y.Self-performance appraisal vs direct-manager appraisal: A case of congruence.Journal of Managerial Psychology. 1996;11:50-65
8. Bishop G. F.,Tuchfarber A.,Oldendick R. W.Opinions on fictitious issues: The pressure to answer survey questions.Public Opinion Quarterly. 1986;50:240-250
9. Bloch D. A.,Kraemer H. C.2 × 2 kappa coefficients: Measures of agreement or association.Biometrics. 1989;45:269-287
10. Boyle P.,Boice B.Best practices for enculturation: Collegiality, mentoring, and structure.New Directions for Higher Education. 1998;101:87-94
11. Bronfenbrenner U.The experimental ecology of education.Educational Researcher. 1976;5:5-15
12. Brown J. S.,Collins A.,Duguid P.Situated cognition and the culture of learning.Educational Researcher. 1989;18:35-42
13. Caputo D.,Dunning D.What you don’t know: The role played by errors of omission in imperfect self-assessments.Journal of Experimental Social Psychology. 2005;41:488-505
14. Overview of doctoral educational studies and reports: 1990-present. Stanford, CA: The Carnegie Foundation for the Advancement of Teaching; 2001:
15. Chan T. F.Envisioning the future of doctoral education. Golde C. M.Walker G. E., ed. San Francisco, CA: Jossey-Bass; 2006:120-134.
16. Charlesworth M.,Farrall L.,Stokes T.,Turnbull D.Life among the scientists. Melbourne: Oxford University Press; 1989:
17. Cicchetti D. V.The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation.Behavioral and Brain Sciences. 1991;14:119-135
18. Clark R.,Harden S.,Johnson W.Mentor relationships in clinical psychology doctoral training: Results of a national survey.Teaching of Psychology. 2000;27:262-268
19. Collins A.,Brown J. S.,Holum A.Cognitive apprenticeship: Making thinking visible.American Educator. 1991;6:38-46
20. Conway J. M.,Huffcutt A. I.Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings.Human Performance. 1997;10:331-360
21. Cooper W. H.Ubiquitous halo.Psychological Bulletin. 1981;90:218-244
22. Cox M.,Andriot A.Mentor and undergraduate student comparisons of students’ research skills.Journal of STEM Education. 2009;10:31-39
23. Creswell J. W.,Plano Clark V. L.,Gutmann M. L.,Hanson W. E.Handbook of mixed methods in social & behavioral research. Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage Publications; 2003:209-240.
24. Cumming J.The doctoral experience in science: Challenging the current orthodoxy.British Educational Research Journal. 2009;35:877-890
25. Delamont S.,Atkinson P.Doctoring uncertainty: Mastering craft knowledge.Social Studies of Science. 2001;31:87-107
26. Delamont S.,Atkinson P.,Parry O.The doctoral experience: Success and failure in graduate school. London: Falmer Press; 2000:
27. Denicolo P.Assessing the PhD: A constructive view of criteria.Quality Assurance in Education. 2003;11:84-91
28. Dunbar K.Creative thought: An investigation of conceptual structures and processes. Ward T. B.Smith S. M., ed. Washington, DC: American Psychological Association; 1997:461-493.
29. Dunbar K.How scientists think in the real world: Implications for science education.Journal of Applied Developmental Psychology. 2000;21:49-58
30. Dunning D.,Heath C.,Suls J. M.Flawed self-assessment: Implications for health, education, and the workplace.Psychological Science in the Public Interest. 2004;5:69-106
31. Dunning D.,Johnson K.,Ehrlinger J.,Kriger J.Why people fail to recognize their own incompetence.Current Directions in Psychological Science. 2003;12:83-87
32. Ehrlinger J.,Dunning D.How chronic self-views influence (and potentially mislead) estimates of performance.Journal of Personality and Social Psychology. 2003;84:5-17
33. Falchikov N.,Boud D.Student self-assessment in higher education: A meta-analysis.Review of Educational Research. 1989;59:395-430
34. Falchikov N.,Goldfinch J.Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks.Review of Educational Research. 2000;70:287-322
35. Feldon D. F.Cognitive load and classroom teaching: The double-edged sword of automaticity.Educational Psychologist. 2007;42:123-137
36. Feldon D. F.Do psychology researchers tell it like it is? A microgenetic analysis of research strategies and self-report accuracy.Instructional Science. 2010;38:395-415
37. Feldon D. F.,Maher M.,Timmerman B.A call for performance-based data in the study of STEM graduate education.Science. 2010;329:282-283
38. Feldon D. F.,Peugh J.,Timmerman B. E.,Maher M. A.,Hurst M.,Strickland D.,Gilmore J. A.,Stiegelmeyer C.Graduate students’ teaching experiences improve their methodological research skills.Science. 2011;333:1037-1039
39. Feldon D. F.,Timmerman B. E.,Stowe K.,Showman R.Translating expertise into effective instruction: The impacts of cognitive task analysis (CTA) on lab report quality and student retention in the biological sciences.Journal of Research in Science Teaching. 2010;47:1165-1185
40. Fleiss J. L.,Cohen J.The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.Educational and Psychological Measurement. 1973;33:613-619
41. Franke A.,Arvidsson B.Research supervisors’ different ways of experiencing supervision of doctoral students.Studies in Higher Education. 2011;36:7-19
42. Gardner S. K.“I heard it through the grapevine”: Doctoral student socialization in chemistry and history.Higher Education. 2007;54:723-740
43. Gardner S. K.“What’s too much and what’s too little?”: The process of becoming an independent researcher in doctoral education.Journal of Higher Education. 2008;79:326-350
44. Gardner S. K.Conceptualizing success in doctoral education: Perspectives of faculty in seven disciplines.Review of Higher Education. 2009;32:383-406
45. Gilmore J.,Strickland D.,Timmerman B.,Maher M.,Feldon D. F.Weeds in the flower garden: An exploration of plagiarism in graduate students’ research proposals and its connection to enculturation, ESL, and contextual factors.International Journal of Educational Integrity. 2010;6:13-28
46. Glaser B. G.The constant comparative method of qualitative analysis.Social Problems. 1965;12:436-445
47. Golde C. M.Beginning graduate school: Explaining first-year doctoral attrition.New Directions for Higher Education. 1998;101:55-64
48. Golde C. M.,Conklin Bueschel A.,Jones L.,Walker G. E.Doctoral education and faculty of the future. Ehrenberg R. G.Kuh C. V., ed. Ithaca, NY: Cornell University Press; 2009:53-64.
49. Golde C. M.,Dore T.At cross purposes: What the experiences of doctoral students reveal about doctoral education. Philadelphia, PA: The Pew Charitable Trusts; 2001:
50. Envisioning the future of doctoral education: Preparing stewards of the discipline. Golde C. M.Walker G. E., ed. San Francisco, CA: Jossey-Bass; 2006:
51. Goldstein S. D.,Lindeman B.,Colbert-Getz J.,Arbella T.,Dudas R.,Lidor A.,Sacks B.Faculty and resident evaluations of medical students on a surgery clerkship correlate poorly with standardized exam scores.The American Journal of Surgery. 2014;207:231-235
52. Green S. G.,Bauer T. N.Supervisory mentoring by advisers: Relationships with doctoral student potential, productivity, and commitment.Personnel Psychology. 1995;48:537-562
53. Greene J. C.,Caracelli V. J.,Graham W. F.Toward a conceptual framework for mixed-method evaluation designs.Educational Evaluation and Policy Analysis. 1989;11:255-274
54. Greenwald A. G.,Banaji M. R.Implicit social cognition: Attitudes, self-esteem, and stereotypes.Psychological Review. 1995;102:4-27
55. Haaga D. A. F.Peer review of term papers in graduate psychology courses.Teaching of Psychology. 1993;20:28-32
56. Halonen J. S.,Bosack T.,Clay S.,McCarthy M.,Dunn D. S.,Hill G. W.,.. Whitlock K.A rubric for learning, teaching, and assessing scientific inquiry in psychology.Teaching of Psychology. 2003;30:196-208
57. Harris M. M.,Schaubroeck J.A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings.Personnel Psychology. 1988;41:43-62
58. Hawley P.Being bright is not enough: The unwritten rules of doctoral studies. Springfield, IL: Charles C. Thomas; 1993:
59. Hinds P. J.The curse of expertise: The effects of expertise and debiasing methods on predictions of novice performance.Journal of Experimental Psychology: Applied. 1999;5:205-221
60. Hinds P.,Patterson M.,Pfeffer J.Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance.Journal of Applied Psychology. 2001;86:1232-1243
61. Johnson L.,Lee A.,Green B.The Ph.D. and the autonomous self: Gender, rationality, and postgraduate pedagogy.Studies in Higher Education. 2000;25:135-147
62. Johnson R. L.,Penny J.,Gordon B.Assessing performance: Designing, scoring, and validating performance tasks. New York, NY: Guilford Press; 2009:
63. Johnson R. L.,Penny J.,Gordon B.,Shumate S. R.,Fisher S. P.Resolving score differences in the rating of writing samples: Does discussion improve the accuracy of scores?.Language Assessment Quarterly: An International Journal. 2005;2:117-146
64. Johnson W.The intentional mentor: strategies and guidelines for the practice of mentoring.Professional Psychology: Research and Practice. 2002;33:88-96
65. Kam B. H.Style and quality in research supervision: The supervisor dependency factor.Higher Education. 1997;34:81-103
66. Kardash C.Evaluation of an undergraduate research experiences: perceptions of undergraduate interns and their faculty mentors.Journal of Educational Psychology. 2000;92:191-201
67. Kelly G. J.,Takao A.Epistemic levels in argument: An analysis of university oceanography students’ use of evidence in writing.Science Education. 2002;86:314-342
68. Knorr-Cetina K.Epistemic cultures: How the sciences make knowledge. Cambridge, MA: Harvard University Press; 1999:
69. Krosnick J. A.,Holbrook A. L.,Berent M. K.,Carson R. T.,Hanneman W. M.,Kopp R. J.,.. Conaway M.The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice?.Public Opinion Quarterly. 2002;66:371-403
70. Lagowski J.,Vick J.Faculty as mentors.New Direction for Teaching and Learning. 1995;62:79-85
71. Larson J. R.The dynamic interplay between employees’ feedback-seeking strategies and supervisors’ delivery of performance feedback.Academy of Management Review. 1989;14:408-422
72. Lovitts B.Leaving the ivory tower: The causes and consequences of departure from doctoral study. Lanham, MD: Rowman & Littlefield; 2001:
73. Lovitts B. E.Making the implicit explicit: Creating performance expectations for the dissertation. Sterling, VA: Stylus Publishing; 2007:
74. Lovitts B. E.The transition to independent research: Who makes it, who doesn’t, and why.Journal of Higher Education. 2008;79:296-325
75. Macrae C. N.,Bodenhausen G. V.,Milne A. B.,Wheeler V.On resisting the temptation for simplification: Counterintentional effects of stereotype suppression on social memory.Social Cognition. 1996;14:1-20
76. Maher M.,Gilmore J. A.,Feldon D. F.,Davis T..Paper presented at the 2014 meeting of the American Educational Research Association; 2014Philadelphia, PA; 2014.
77. Marsh H. W.,Ball S.The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability.Journal of Experimental Education. 1989;57:151-169
78. Marsh H. W.,Bazeley P.Multiple evaluations of grant proposals by independent assessors: Confirmatory factor analysis evaluations of reliability, validity, and structure.Multivariate Behavioral Research. 1999;34:1-30
79. Marton F.,Pong W. Y.On the unit of description in phenomenography.Higher Education Research & Development. 2005;24:335-348
80. Merton R. K.The sociology of science. Chicago, IL: University of Chicago Press; 1973:
81. Milem J. F.,Berger J. B.,Dey E. L.Faculty time allocation: A study of change over twenty years.Journal of Higher Education. 2000;71:545-475
82. Nettles M. T.,Millett C. M.Three magic letters: Getting to Ph.D. Baltimore, MD: John Hopkins University Press; 2006:
83. Nickerson R. S.How we know—and sometimes misjudge—what others know: Imputing one’s own knowledge to others.Psychological Bulletin. 1999;125:737-159
84. Norcini J.Peer assessment of competence.Medical Education. 2003;37:539-543
85. Paglis L.,Green S.,Bauer T.Does mentor mentoring add value? A longitudinal study of mentoring and doctoral student outcomes.Research in Higher Education. 2006;47:451-476
86. Parry S.Disciplines and the doctorate. Dordrecht, the Netherlands: Springer; 2007:
87. Pearson M.,Brew A.Research training and supervision development.Studies in Higher Education. 2002;27:135-150
88. Peters M. A.,Olssen M.Reshaping the university: New relationships between research, scholarship and teaching. Barnett R., ed. Buckingham: Society for Research into Higher Education/Open University Press; 2005:37-47.
89. Petty R. E.,Fleming M. A.,Fabrigar L. R.The review process at PSPB: Correlates of interreviewer agreement and manuscript acceptance.Personality and Social Psychology Bulletin. 1999;25:188-203
90. Pole C.Technicians and scholars in pursuit of the PhD: Some reflections on doctoral study.Research Papers in Education. 2000;15:95-111
91. Pole C. J.,Sprokkereef A.,Burgess R. G.,Lakin E.Supervision of doctoral students in the natural sciences: Expectations and experiences.Assessment and Evaluation in Higher Education. 1997;22:49-63
92. Remler D. K.,Pema E.Why do institutions of higher education reward research while selling education?. Cambridge, MA: National Bureau of Economic Research; 2009:
93. Schwarz N.Self-reports: How the questions shape the answers.American Psychologist. 1999;54:95-105
94. Sloman S. A.Heuristics and biases: The psychology of intuitive judgment. Gilovich T.Griffin D.Kahneman D., ed. New York, NY: Cambridge University Press; 2002:379-396.
95. Stajkovic A. D.,Luthans F.Self-efficacy and work-related performance: A meta-analysis.Psychological Bulletin. 1998;124:240-261
96. Svensson L.Theoretical foundations of phenomenography.Higher Education Research & Development. 1997;16:159-171
97. Tariq V. N.,Stefan L. A. J.,Butcher A. C.,Heylings D. J. A.Developing a new approach to the assessment of project work.Assessment & Evaluation in Higher Education. 1998;23:221-240
98. Thornton G. C.The relationship between supervisory- and self-appraisals of executive performance.Personnel Psychology. 1968;21:441-455
99. TimmermanB. E. (2008). Peer review in an undergraduate biology curriculum: Effects on students’ scientific reasoning, writing, and attitudes (Unpublished doctoral dissertation). Curtin University of Technology, Perth, Australia.
100. Timmerman B.,Feldon D. F.,Maher M.,Strickland D.,Gilmore J. A.Performance-based assessment of graduate student research skills: Timing, trajectory, and potential thresholds.Studies in Higher Education. 2013;38:693-710
101. Timmerman B. E.,Johnson R. L.,Payne J..National Association of Research in Science Teaching 2007 Annual Meeting; 2007New Orleans LA; 2007.
102. Timmerman B. E. C.,Strickland D. C.,Johnson R. L.,Payne J. R.Development of a “universal’ rubric for assessing undergraduates’ scientific reasoning skills using scientific writing.Assessment & Evaluation in Higher Education. 2011;36:509-547
103. Topping K. J.,Smith E. F.,Swanson I.,Elliot A.Formative peer assessment of academic writing between postgraduate students.Assessment & Evaluation in Higher Education. 2000;25:149-169
104. Waldeck J. H.,Orrego V. O.,Plax T. G.,Kearney P.Graduate student/faculty mentoring relationships: Who gets mentored, how it happens, and to what end.Communication Quarterly. 1997;45:93-109
105. Walker G. E.,Golde C.,Jones L.,Conklin Bueschel A.,Hutchings P.The formation of scholars: Rethinking doctoral education for the twenty-first century. San Francisco, CA: Jossey-Bass; 2008:
106. Weidman J. C.,Twale D. J.,Stein E. L.Socialization of graduate and professional students: A perilous passage?. Washington, DC: Association of the Study of Higher Education; 2001:
107. Wexley K. N.,Alexander R. A.,Greenawalt J. P.,Couch M. A.Attitudinal congruence and similarity as related to interpersonal evaluations in manager-subordinate dyads.Academy of Management Journal. 1980;23:320-330
108. Willison J.,O’Regan K.Commonly known, commonly not known, totally unframed: A framework for students becoming researchers.Higher Education Research & Development. 2007;26:393-409
109. Wittwer J.,Nückles M.,Renkl A.Is underestimation less detrimental than overestimation? The impacts of experts’ beliefs about a layperson’s knowledge on learning and question asking.Instructional Science. 2008;36:27-52
110. Zhao C.,Golde C. M.,McCormick A. C.More than a signature: How advisor choice and advisor behavior affect doctoral student satisfaction.Journal of Further and Higher Education. 2007;31:263-281
111. Zimmerman C.The development of scientific reasoning.Developmental Review. 2000;20:99-149