Use of student ratings to benchmark universities: Multilevel modeling of responses to the Australian course experience questionnaire (CEQ)

Journal of Educational Psychology

Volumn 103, Issue 3, 2011, Pages 733-748

  Marsh, Herbert W ^a,b   Ginns, Paul ^c   Morin, Alexandre J S ^b,d   Nagengast, Benjamin ^b   Martin, Andrew J ^c  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY OF WESTERN SYDNEY   (Australia)

^c UNIVERSITY OF SYDNEY   (Australia)

^d UNIVERSITÉ DE SHERBROOKE   (Canada)

Author keywords

Benchmarking universities; Multilevel analyses; Students' evaluations; University educational experience

Indexed keywords

EID: 80051818445     PISSN: 00220663     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0024221     Document Type: Article

References (64)

1. Abrami P.C., d'Apollonia S., Cohen P.A. Validity of student ratings of instruction: What we know and what we do not. Journal of Educational Psychology 1990, 82:219-231. 10.1037/0022-0663.82.2.219.
2. Evaluation and validation of the trial Postgraduate Research Experience Questionnaires 2000, Australian Council for Educational Research, Author, Camberwell, Victoria, Australia.
3. Badhni, S., & Aungles, P. (2002, October). The role of the Course Experience Questionnaire in quality assurance for the higher education sector. Paper presented at the Seventh Quality in Higher Education International Seminar, Melbourne, Victoria, Australia.
4. Barrie S., Ginns P. The linking of national teaching performance indicators to improvements in teaching and learning in classrooms. Quality in Higher Education 2007, 13:275-286. 10.1080/13538320701800175.
5. Brown R. Comparability of degree standards? (Higher Education Policy Institute report) 2010, Higher Education Policy Institute, Oxford, United Kingdom. http://www.hepi.ac.uk./files/47%20Comparability%20of%20degree%20standards.pdf.
6. Cashin W.E. Student ratings of teaching: A summary of the research (IDEA Paper No. 20) http://www.theideacenter.org/sites/default/files/Idea_Paper_20.pdf.
7. Centra J.A. Reflective faculty evaluation 1993, Jossey-Bass, San Francisco, CA.
8. Cheng J.H.S., Marsh H.W. National Student Survey: Are differences between universities and courses reliable and meaningful?. Oxford Review of Education 2010, 36:693-712. 10.1080/03054985.2010.491179.
9. Cohen P.A. Effectiveness of student-rating feedback for improving college instruction: A meta-analysis. Research in Higher Education 1980, 13:321-341. 10.1007/BF00976252.
10. d'Apollonia S., Abrami P.C. Navigating student ratings of instruction. American Psychologist 1997, 52:1198-1208. 10.1037/0003-066X.52.11.1198.
11. Elphinstone L. The development of the Course Experience Questionnaire 1990, University of Melbourne, Melbourne, Victoria, Australia, (Unpublished master's thesis).
12. Entwistle N.J., Ramsden P. Understanding student learning 1983, Croom Helm, London, United Kingdom.
13. Feldman K.A. Association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Research in Higher Education 1989, 30:583-645. 10.1007/BF00992392.
14. Feldman K.A. Instructional effectiveness of college teachers as judged by teachers themselves, current and former students, colleagues, administrators, and external (neutral) observers. Research in Higher Education 1989, 30:137-194. 10.1007/BF00992716.
15. Feldman K.A. Identifying exemplary teachers and teaching: Evidence from student ratings. Effective teaching in higher education: Research and practice 1997, 368-395. Agathon Pr, New York, NY. R.P. Perry, J.C. Smart (Eds.).
16. Feldman K.A. Reflections on the effective study of college teaching and student ratings: One continuing quest and two unresolved issues. Higher education: Handbook of theory and research 1998, 35-74. Agathon Pr, New York, NY. J.C. Smart (Ed.).
17. Gillmore G.M., Kane M.T., Naccarato R.W. The generalizability of student ratings of instruction: Estimates of teacher and course components. Journal of Educational Measurement 1978, 15:1-13. 10.1111/j.1745-3984.1978.tb00051.x.
18. Ginns P., Marsh H.W., Behnia M., Cheng J.H.S., Scalas L.F. Using postgraduate students' evaluations of research experience to benchmark departments and faculties: Issues and challenges. British Journal of Educational Psychology 2009, 79:577-598. 10.1348/978185408X394347.
19. Goldstein H. Multilevel statistical models 1995, Arnold, London, United Kingdom. 2nd ed.
20. Govendir M., Ginns P., Symons R., Tammen I. The student experience: Proceedings of 32nd HERDSA Annual Conference Improving the research higher degree experience at the Faculty of Veterinary Science, The University of Sydney 2009, 163-172. Higher Education Research and Development Society of Australasia, Milperra, New South Wales, Australia. http://www.herdsa.org.au/wp-content/uploads/conference/2009/papers/HERDSA2009_Govendir_M.pdf.
21. Enhancing the GCA National Surveys: An examination of critical factors leading to enhancements in the instrument, methodology and process 2006, Graduate Careers Australia, Author, Canberra, Australian Capital Territory, Australia.
22. Course Experience Questionnaire 2001 2002, Graduate Careers Council of Australia, Author, Canberra, Australian Capital Territory, Australia.
23. Course Experience Questionnaire 2002 2003, Graduate Careers Council of Australia, Author, Canberra, Australian Capital Territory, Australia.
24. Graham J.W. Missing data analysis: Making it work in the real world. Annual Review of Psychology 2009, 60:549-576. 10.1146/annurev.psych.58.110405.085530.
25. Graham J.W., Hofer S.M. Multiple imputation in multivariate research. Modeling longitudinal and multilevel data: Practical issues, applied approaches, and specific examples 2000, 201-218. Erlbaum, Mahwah, NJ. T.D. Little, K.U. Schnable, J. Baumert (Eds.).
26. Greenwald A.G., Gillmore G.M. No pain, no gain? The importance of measuring course workload in student ratings of instruction. Journal of Educational Psychology 1997, 89:743-751. 10.1037/0022-0663.89.4.743.
27. Higher Education Funding Council for England National Student Survey: Administration (Circular letter 22/2003) http://www.hefce.ac.uk/pubs/circlets/2003/cl22_03.htm.
28. Higher Education Funding Council for England, Quality Assurance Agency for Higher Education, Universities UK, and Standing Conference of Principals Quality assurance in higher education: Proposals for consultation (Consultation 01/45) http://www.hefce.ac.uk/pubs/hefce/2001/01_45/01_45.pdf.
29. Leckie G., Goldstein H. The limitations of using school league tables to inform school choice. Journal of the Royal Statistical Society: Series A 2009, 172:835-851.
30. Lizzio A., Wilson K., Simons R. University students' perceptions of the learning environment and academic outcomes: Implications for theory and practice. Studies in Higher Education 2002, 27:27-52. 10.1080/03075070120099359.
31. Marsh H.W. The influence of student, course and instructor characteristics on evaluations of university teaching. American Educational Research Journal 1980, 17:219-237.
32. Marsh H.W. SEEQ: A reliable, valid, and useful instrument for collecting students' evaluations of university teaching. British Journal of Educational Psychology 1982, 52:77-95.
33. Marsh H.W. Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology 1984, 76:707-754. 10.1037/0022-0663.76.5.707.
34. Marsh H.W. Students' evaluations of university teaching: Research findings, methodological issues, and directions for future research [Monograph]. International Journal of Educational Research 1987, 11(3). 10.1016/0883-0355(87)90001-2.
35. Marsh H.W. Multidimensional students' evaluations of teaching effectiveness: A test of alternative higher-order structures. Journal of Educational Psychology 1991, 83:285-296. 10.1037/0022-0663.83.2.285.
36. Marsh H.W. Do university teachers become more effective with experience? A multilevel growth model of students' evaluations of teaching over 13 years. Journal of Educational Psychology 2007, 99:775-790. 10.1037/0022-0663.99.4.775.
37. Marsh H.W. Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. The scholarship of teaching and learning in higher education: An evidence-based perspective 2007, 319-384. Springer, New York, NY. R.P. Perry, J.C. Smart (Eds.).
38. Marsh H.W., Bailey M. Multidimensional students' evaluations of teaching effectiveness: A profile analysis. Journal of Higher Education 1993, 64:1-18. 10.2307/2959975.
39. Marsh H.W., Cheng J.H.S. NSS: Dimensionality, multilevel structure, and differentiation at the level of university and discipline http://www.heacademy.ac.uk/assets/York/documents/ourwork/research/surveys/nss/NSS_herb_marsh-28.08.08.pdf.
40. Marsh H.W., Dunkin M. Students' evaluations of university teaching: A multidimensional perspective. Effective teaching in higher education: Research and practice 1997, 241-320. Agathon, New York, NY. R.P. Perry, J.C. Smart (Eds.).
41. Marsh H.W., Hau K.-T. Applications of latent-variable models in educational psychology: The need for methodological-substantive synergies. Contemporary Educational Psychology 2007, 32:151-170. 10.1016/j.cedpsych.2006.10.008.
42. Marsh H.W., Lüdtke O., Muthén B., Asparouhov T., Morin A.J.S., Trautwein U., Nagengast B. A new look at the Big Five factor structure through exploratory structural equation modeling. Psychological Assessment 2010, 22:471-491. 10.1037/a0019227.
43. Marsh H.W., Muthén B., Asparouhov T., Lüdtke O., Robitzsch A., Morin A.J.S., Trautwein U. Exploratory structural equation modeling, integrating CFA and EFA: Application to students' evaluations of university teaching. Structural Equation Modeling 2009, 16:439-476. 10.1080/10705510903008220.
44. Marsh H.W., Roche L.A. Making students' evaluations of teaching effectiveness effective. American Psychologist 1997, 52:1187-1197. 10.1037/0003-066X.52.11.1187.
45. Marsh H.W., Roche L.A. Effects of grading leniency and low workloads on students' evaluations of teaching: Popular myth, bias, validity, or innocent bystanders?. Journal of Educational Psychology 2000, 92:202-228. 10.1037/0022-0663.92.1.202.
46. Marsh H.W., Rowe K.J., Martin A. PhD students' evaluations of research supervision: Issues, complexities, and challenges in a nationwide Australian experiment in benchmarking universities. Journal of Higher Education 2002, 73:313-348. 10.1353/jhe.2002.0028.
47. Marsh H.W., Seaton M., Trautwein U., Lüdtke O., Hau K.T., O'Mara A.J., Craven R.G. The big-fish-little-pond-effect stands up to critical scrutiny: Implications for theory, methodology, and future research. Educational Psychology Review 2008, 20:319-350. 10.1007/s10648-008-9075-6.
48. McKeachie W.J. Student ratings of faculty: A reprise. Academe 1979, 65:384-397. 10.2307/40248725.
49. McKeachie W.J. Student ratings: The validity of use. American Psychologist 1997, 52:1218-1225. 10.1037/0003-066X.52.11.1218.
50. McKinnon K.R., Walker S.H., Davis D. Benchmarking: A manual for Australian universities 2000, Australian Department of Education, Training and Youth Affairs, Canberra, Australian Capital Territory, Australia. http://www.dest.gov.au/archive/highered/otherpub/bench.pdf.
51. Muthén L.K., Muthén B. Mplus user's guide 2010, Muthén & Muthén, Los Angeles, CA.
52. Parsons P.G. The Lancaster approaches to studying inventory and course perceptions questionnaire: A replicated study at the Cape Technikon. South African Journal of Higher Education 1988, 2:103-111.
53. Patrick, K. (2003, March). The CEQ in practice: Using the CEQ for improvement. In Graduate Careers Council of Australia symposium Graduates: Outcomes, Quality and the Future, Canberra, Australian Capital Territory, Australia.
54. Ramsden P. Course evaluation in higher education 1976, Council for National Academic Awards, Lancaster, United Kingdom, (Unpublished masters thesis).
55. Ramsden P. Student learning and perceptions of the academic environment. Higher Education 1979, 8:411-427. 10.1007/BF01680529.
56. Ramsden P. A performance indicator of teaching quality in higher education: The Course Experience Questionnaire. Studies in Higher Education, 16 1991, 2:129-150. 10.1080/03075079112331382944.
57. Ramsden P. Report on the Course Experience Questionnaire trial. Performance indicators on higher education 1991, Vol. 2:1-85. Australian Government Publishing Service, Canberra, Australian Capital Territory, Australia. R. Linke (Ed.).
58. Ramsden P. Learning to teach in higher education 1992, Routledge, London, United Kingdom. 10.4324/9780203413937.
59. Ramsden P., Entwistle N.J. Effects of academic departments on students' approaches to studying. British Journal of Educational Psychology 1981, 51:368-383.
60. Rasbash J., Steele F., Browne W., Prosser B. A user's guide to MLwiN Version 2.0 2005, University of Bristol, Bristol, United Kingdom.
61. Raudenbush S.W., Bryk A.S. Hierarchical linear models 2002, Sage, Thousand Oaks, CA. 2nd ed.
62. Richardson J.T.E. Instruments for obtaining student feedback: A review of the literature. Assessment & Evaluation in Higher Education 2005, 30:387-415. 10.1080/02602930500099193.
63. Watkins D. Correlates of approaches to learning: A cross-cultural meta-analysis. Perspectives on thinking, learning, and cognitive styles 2001, 139-166. Erlbaum, Mahwah, NJ. R. Sternberg, L.-F. Zhang (Eds.).
64. Wilson K., Lizzio A., Ramsden P. The development, validation and application of the Course Experience Questionnaire. Studies in Higher Education 1997, 22(1):33-53. 10.1080/03075079712331381121.