Students' perceptions of characteristics of effective college teachers: A validity study of a teaching evaluation form using a mixed-methods analysis

American Educational Research Journal

Volumn 44, Issue 1, 2007, Pages 113-160

  Onwuegbuzie, Anthony J ^a   Witcher, Ann E ^b   Collins, Kathleen M T ^c   Filer, Janet D ^b   Wiedmaier, Cheryl D ^b   Moore, Chris W ^b  

^a UNIVERSITY OF SOUTH FLORIDA   (United States)

^b UNIVERSITY OF CENTRAL ARKANSAS   (United States)

^c UNIVERSITY OF ARKANSAS   (United States)

Author keywords

College teaching; Mixed methods; Teaching evaluation form; Validity

Indexed keywords

EID: 34047252361     PISSN: 00028312     EISSN: None     Source Type: Journal    
DOI: 10.3102/0002831206298169     Document Type: Article

References (122)

1. Aleamoni, L. M. (1981). The use of student evaluations in the improvement of instruction. NACTA Journal, 20, 16.
2. Altman, D. G. (1991). Practical statistics for medical research. London: Chapman and Hall.
3. Ambady, N., & Rosenthal, R. (1992). Half a minute. Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology, 64, 431-441.
4. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing (Rev. ed.). Washington, DC: American Educational Research Association.
5. The American Heritage College Dictionary (3rd ed.). (1997). Boston, MA: Houghton Mifflin.
6. Babad, E. (2001). Students' course selection: Differential considerations for first and last course. Research in Higher Education, 42, 469-492.
7. Bernstein, I. H., & Teng, G. (1989). Factoring items and factoring scales are different: Spurious evidence for multidimensionality due to item categorization. Psychological Bulletin, 105, 467-477.
8. Bickel, P. J., & Doksum, K. A. (1977). Mathematical statistics. San Francisco: Holden-Day.
9. Blackburn, R. T., & Clark, M. J. (1975). An assessment of faculty performance. Some correlates between administrators, colleagues, students, and self-ratings. Sociology of Education, 48, 242-256.
10. Braskamp, L. A., Brandenberg, D. C., & Ory, J. C. (1984). Evaluating teaching effectiveness: A practical guide. Beverly Hills, CA: Sage.
11. Caracelli, V. W., & Greene, J. C. (1993). Data analysis strategies for mixed-methods evaluation designs. Educational Evaluation and Policy Analysis, 15, 195-207.
12. Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245-276.
13. Centra, J. A. (1974). The relationship between student and alumni ratings of teachers. Educational and Psychological Measurement, 34, 321-326.
14. Centra, J. A., & Creech, F. R. (1976). The relationship between student teachers and course characteristics and student ratings of teacher effectiveness (Project Report 76-1). Princeton, NJ: Educational Testing Service.
15. Cliff, N., & Krus, D. J. (1976). Interpretation of canonical analyses: Rotated vs. unrotated solutions. Psychometrica, 41, 35-42.
16. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
17. Cohen, P. A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51, 281-309.
18. Colaizzi, P. F. (1978). Psychological research as the phenomenologist views it. In R. Vaile & M. King (Eds.), Existential phenomenological alternatives for psychology (pp. 48-71). New York: Oxford University Press.
19. Collins, K. M. T., Onwuegbuzie, A. J., & Jiao, Q. G. (2006). Prevalence of mixed methods sampling designs in social science research. Evaluation and Research in Education, 19(2), 119.
20. Collins, K. M. T., Onwuegbuzie, A. J., &Jiao, Q. G. (in press). A mixed methods investigation of mixed methods sampling designs in social and health science research. Journal of Mixed Methods Research.
21. Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L. (2006). A model incorporating the rationale and purpose for conducting mixed methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4, 67-100.
22. Constas, M. A. (1992). Qualitative data analysis as a public event: The documentation of category development procedures. American Educational Research Journal, 29, 253-266.
23. Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage.
24. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory, Orlando, FL: Holt, Rinehart & Winston.
25. Crumbley, L., Henry, B. K., & Kratchman, S. H. (2001). Students' perceptions of the evaluation of college teaching. Quality Assurance in Education, 9, 197-207.
26. Cuccaro-Alamin, S., & Choy, S. (1998). Post secondary financing strategies: How undergraduates combine work, borrowing, and attendance (NCES 98-088). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
27. Darlington, R. B., Weinberg, S. L., & Walberg, H. J. (1973). Canonical variate analysis and related techniques. Review of Educational Research, 42, 131-143.
28. Demmon-Berger, D. (1986). Effective teaching: Observations from research. Arlington, VA: American Association of School Administrators. (ERIC Document Reproduction Service No, ED274087)
29. Dommeyer, C. J., Baum, P., Chapman, K. S., & Hanna, R. W. (2002). Attitudes of business faculty towards two methods of collecting teaching evaluations: Paper vs. online. Assessment & Evaluation in Higher Education, 27, 455-462.
30. Doyle, K. O., & Crichton, L. A. (1978). Student, peer, and self-evaluations of college instruction. Journal of Educational Psychology, 70, 815-826.
31. Feldman, K. A. (1978). Course characteristics and college students' ratings of their teachers and courses: What we know and what we don't. Research in Higher Education, 9, 199-242.
32. Feldman, K. A. (1989). The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Research in Higher Education, 30, 583-645.
33. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine.
34. Glass, G. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
35. Glass, G. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351-379.
36. Glass, G., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
37. Glesne, C., & Peshkin, A. (1992). Becoming qualitative researchers: An introduction. White Plains, NY: Longman.
38. Goetz, J. P., & LeCompte, M. D. (1984). Ethnography and the qualitative design in educational research. New York: Academic Press.
39. Gray, M., & Bergmann, B. R. (2003). Student teaching evaluations: Inaccurate, demeaning, misused. Academe, 89(5), 44-46.
40. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274.
41. Greenwald, A. G., & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52, 1209-1217.
42. Greimel-Fuhrmann, B., & Geyer, A. (2003). Students' evaluation of teachers and instructional quality-Analysis of relevant factors based on empirical evaluation. Assessment and Evaluation in Higher Education, 28, 229-238.
43. Guthrie, E. R. (1954). The evaluation of teaching: A progress report. Seattle: University of Washington Press.
44. Haskell, R. E. (1997). Academic freedom, tenure, and student evaluation of faculty: Galloping polls in the 21st century. Education Policy Analysis Archives, 5(6). Retrieved July 26, 2006, from http://epaa.asu.edu/epaa/v5n6. html
45. Henson, R. K., Capraro, R. M., & Capraro, M. M. (2004). Reporting practice and use of exploratory factor analysis in educational research journals: Errors and explanation. Research in the Schools, 11(2), 61-72.
46. Henson, R. K., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research. Educational and Psychological Measurement, 66, 393-416.
47. Hetzel, R. D. (1996). A primer on factor analysis with comments on patterns of practice and reporting. In B. Thompson (Ed.), Advances in social science methodology (Vol. 4, pp. 175-206). Greenwich, CT: JAI.
48. Horn, L. J. (1994). Undergraduates who work while enrolled inpostsecondary education (NCES 94-311). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
49. Johnson, R. B., & Christensen, L. B. (2004). Educational research: Quantitative, qualitative, and mixed approaches. Boston: Allyn & Bacon.
50. Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26.
51. Johnson, R. B., & Turner, L. A. (2003). Data collection strategies in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 297-319). Thousand Oaks, CA: Sage.
52. Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23, 187-200.
53. Kane, R., Sandretto, S., & Heath, C. (2004). An investigation into excellent tertiary teaching: Emphasizing reflective practice. Higher Education, 47, 283-310.
54. Kieffer, K. M. (1999). An introductory primer on the appropriate use of exploratory and confirmatory factor analysis. Research in the Schools, 6(2), 75-92.
55. Krejecie, R. V., & Morgan, D. W. (1970). Determining sample sizes for research activities. Educational and Psychological Measurement, 30, 608.
56. Kulik, J. A. (2001, Spring). Student ratings: Validity, utility, and controversy. New Directions for Institutional Research, 109, 9-25.
57. Lambert, Z. V., & Durand, R. M. (1975). Some precautions in using canonical analysis. Journal of Market Research, 12, 468-475.
58. Lawley, D. N., & Maxwell, A. E. (1971). Factor analysis as a statistical method. New York: Macmillan.
59. Leech, N. L., & Onwuegbuzie, A. J. (2005, April). A typology of mixed methods research designs. Invited James E. McLean Outstanding Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.
60. Leech, N. L., & Onwuegbuzie, A. J. (in press-a). An array of qualitative data analysis tools: A call for qualitative data analysis triangulation. School Psychology Quarterly.
61. Leech, N. L., & Onwuegbuzie, A. J. (in press-b). A typology of mixed methods research designs. Quality & Quantity: International Journal of Methodology.
62. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
63. Marsh, H. W. (1987). Students' evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11, 253-388.
64. Marsh, H. W., & Bailey, M. (1993). Multidimensional students' evaluations of teaching effectiveness. A profile analysis. Journal of Higher Education, 64, 1-18.
65. Marsh, H. W., Overall, J. U., & Kessler, S. P. (1979). Validity of student evaluations of instructional effectiveness: A comparison of faculty self-evaluations and evaluations by their students, Journal of Educational Psychology, 71, 149-160.
66. Marsh, H. W., & Roche, L. A. (1993). The use of students' evaluations and an individually structured intervention to enhance university teaching effectiveness. American Educational Research Journal, 30, 217-251.
67. Maxwell, J. A. (2005). Qualitative research design: An interactive approach (2nd. ed.). Thousand Oaks, CA: Sage.
68. Merriam, S. (1988). Case study research in education: A qualitative approach. San Francisco: Jossey-Bass.
69. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Old Tappan, NJ: Macmillan.
70. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
71. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: A sourcebook of new methods. Thousand Oaks, CA: Sage.
72. Minor, L. C., Onwuegbuzie, A. J., Witcher, A. E., & James, T. L. (2002). Preservice teachers' educational beliefs and their perceptions of characteristics of effective teachers. Journal of Educational Research, 96, 116-127.
73. Moustakas, C. (1994). Phenomenological research methods. Thousands Oaks, CA: Sage.
74. Murray, H. G. (1983). Low-inference classroom teaching behaviors and student ratings of college teaching effectiveness. Journal of Educational Psychology, 71, 856-865.
75. Naftulin, D. H., Ware, J. E., & Donnelly, F. A. (1973). The Doctor Fox lecture: A paradigm of educational seduction. Journal of Medical Education, 48, 630-635.
76. Nelson, C. B., Rehm, J., Bedirhan, U., Grant, B., & Chatterji, S. (1999). Factor structures for DSM-IV substance disorder criteria endorsed by alcohol, cannabis, cocaine, and opiate users: Results from the WHO reliability and validity study. Addiction, 94, 843-855.
77. Newman, I., & Benz, C. R. (1998). Qualitative-quantitative research methodology: Exploring the interactive continuum. Carbondale: Southern Illinois University Press.
78. Newman, I., Ridenour, C. S., Newman, C., & DeMarco, G. M. P. (2003). A typology of research purposes and its relationship to mixed methods. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 167-188). Thousand Oaks, CA: Sage.
79. Okpala, C. O., & Ellis, R. (2005). The perceptions of college students on teacher quality: A focus on teacher qualifications. Education, 126, 374-378.
80. Onwuegbuzie, A. J. (2003a). Effect sizes in qualitative research: A prolegomenon. Quality & Quantity: International Journal of Methodology, 37, 393-409.
81. Onwuegbuzie, A. J. (2003b). Expanding the framework of internal and external validity in quantitative research. Research in the Schools, 10(1), 71-90.
82. Onwuegbuzie, A. J., & Collins, K. M. T. (in press). A typology of mixed methods sampling designs in social science research. The Qualitative Report.
83. Onwuegbuzie, A. J., & Daniel, L. G. (2002). A framework for reporting and interpreting internal consistency reliability estimates. Measurement and Evaluation in Counseling and Development, 35, 89-103.
84. Onwuegbuzie, A. J., & Daniel, L. G. (2003, February 12). Typology of analytical and interpretational errors in quantitative and qualitative educational research. Current Issues in Education [Electronic version], 5(2). Available from http://cie.ed.asu.edu/volume6/number2/
85. Onwuegbuzie, A. J., & Daniel, L. G. (2004). Reliability generalization: The importance of considering sample specificity, confidence intervals, and subgroup differences. Research in the Schools, 11(1), 61-72.
86. Onwuegbuzie, A. J., Daniel, L. G., & Collins, K. M. T. (2006). Student teaching evaluations: Psychometric, methodological, and interpretational issues. Manuscript submitted for publication.
87. Onwuegbuzie, A. J., Daniel, L. G., & Collins, K. M. T. (in press). A meta-validation model for assessing the score validity of student teacher evaluations. Quality and Quantity: International Journal of Methodology.
88. Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in the Schools, 73(1), 48-63.
89. Onwuegbuzie, A. J., & Leech, N. L. (2004a). Enhancing the interpretation of "significant" findings: The role of mixed methods research. The Qualitative Report, 9, 770-792. Retrieved July 26, 2006, from http://www.nova.edu/ssss/QR/QR9-4/onwuegbuzie.pdf
90. Onwuegbuzie, A. J., & Leech, N. L. (2004b). Post-hoc power: A concept whose time has come. Understanding Statistics, 3, 201-230.
91. Onwuegbuzie, A. J., & Leech, N. L. (2006). Linking research questions to mixed methods data analysis procedures. The Qualitative Report, 11, 474-498. Retrieved January 9, 2007, from http://www.nova.edu/ssss/QR/QR11-3/ onwuegbuzie.pdf
92. Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 351-383). Thousand Oaks, CA: Sage.
93. Ory, J. C. (2000, Fall). Teaching evaluation: Past, present, and future. New Directions for Teaching and Learning, 83, 13-18.
94. Ory, J. C., Braskamp, L. A., & Pieper, D. M. (1980). The congruency of student evaluative information collected by three methods. Journal of Educational Psychology, 72, 181-185.
95. Ory, J. C., & Ryan, K. (2001, Spring). How do student ratings measure up to a new validity framework? New Directions for Institutional Research, 109, 27-44.
96. Overall, J. U., & Marsh, H. W. (1980). Students' evaluations of instruction: A longitudinal study of their stability. Journal of Educational Psychology, 72, 321-325.
97. Patton, M. Q. (1990). Qualitative research and evaluation methods (2nd ed.). Newbury Park, CA: Sage.
98. Peterson, K., & Kauchak, D. (1982). Teacher evaluation: Perspectives, practices, and promises. Salt Lake City: Utah University Center for Educational Practice.
99. Sandelowski, M., & Barroso, J. (2003). Creating metasummaries of qualitative findings. Nursing Research, 52, 226-233.
100. Schaeffer, G., Epting, K., Zinn, T., & Buskit, W. (2003). Student and faculty perceptions of effective teaching: A successful replication. Teaching of Psychology, 30, 133-136.
101. Schmelkin, L. P., Spencer, K. J., & Gellman, E. S. (1997). Faculty perspectives on course and teacher evaluations. Research in Higher Education, 38, 575-592.
102. Seldin, P. (1984). Changing practices in faculty evaluation. San Francisco: Jossey-Bass.
103. Seldin, P. (1993). The use and abuse of student ratings of professors. Chronicle of Higher Education, 39, A40.
104. Seldin, P. (1999). Current practices-good and bad-nationally. In P. Seldin (Ed.), Current practices in evaluating teaching: A practical guide to improved faculty performance and promotion/tenure decisions (pp. 1-24). Bolton, MA: Anker.
105. Sheehan, D. S. (1999). Student evaluation of university teaching. Journal of Instructional Psychology, 26, 188-193.
106. Siegel, S., & Castellan, J. N. (1988). Nonparametric statistics for the behavioural sciences. New York: McGraw-Hill.
107. Simmons, T. L. (1996). Student evaluation of teachers: Professional practice or punitive policy? JALT Testing & Evaluation N-SIG Newsletter, 1(1), 12-16.
108. Spencer, K. J., & Schmelkin, L. P. (2002). Students' perspectives on teaching and its evaluation. Assessment & Evaluation in Higher Education, 27, 397-408.
109. Tabachnick, B. G., & Fidell, L. S. (2006). Using multivariate statistics (5th ed.). New York: Harper & Row.
110. Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches (Applied Social Research Methods Series, Vol. 46). Thousand Oaks, CA: Sage.
111. Tashakkori, A., & Teddlie, C. (2003). The past and future of mixed methods research: From data triangulation to mixed model designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 671-701). Thousand Oaks, CA: Sage.
112. Tashakkori, A., & Teddlie, C. (2006, April). Validity issues in mixed methods research: Calling for an integrative framework. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
113. Theall, M., & Franklin, J. (2001, Spring). Looking for bias in all the wrong places: A search for truth or a witch hunt in student ratings of instruction? New Directions for Institutional Research, 109, 45-56.
114. Thompson, B. (1980, April). Canonical correlation: Recent extensions for modelling educational processes. Paper presented at the annual meeting of the American Educational Research Association, Boston.
115. Thompson, B. (1984). Canonical correlation analysis: Uses and interpretations. Beverly Hills, CA: Sage. (ERIC Document Reproduction Service No. ED199269)
116. Thompson, B. (1988, April). Canonical correlation analysis: An explanation with comments on correct practice. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA. (ERIC Document Reproduction Service No. ED295957)
117. Thompson, B. (1990, April). Variable importance in multiple regression and canonical correlation. Paper presented at the annual meeting of the American Educational Research Association, Boston. (ERIC Document Reproduction Service No. ED317615)
118. Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC: American Psychological Association.
119. Washburn, K., & Thornton, J. F. (Ed.). (1996). Dumbing down: Essays on the strip mining of American culture. New York: Norton.
120. Williams, W. M., & Ceci, S. J. (1997). How'm I doing? Problems with student ratings of instructors and courses. Change, 29(5), 13-23.
121. Witcher, A. E., Onwuegbuzie, A. J., & Minor, L. C. (2001). Characteristics of effective teachers: Perceptions of preservice teachers. Research in the Schools, 8(2), 45-57.
122. Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442.