The international state of research on measurement of competency in higher education

Studies in Higher Education

Volumn 40, Issue 3, 2015, Pages 393-411

  Zlatkin Troitschanskaia, Olga ^a   Shavelson, Richard J ^b   Kuhn, Christiane ^a  

^a JOHANNES GUTENBERG UNIVERSITY   (Germany)

^b STANFORD UNIVERSITY   (United States)

Author keywords

assessment; competency; higher education; learning outcome; measurement; quality

Indexed keywords

EID: 84924804261     PISSN: 03075079     EISSN: 1470174X     Source Type: Journal    
DOI: 10.1080/03075079.2015.1004241     Document Type: Article

References (94)

1. AACU (Association of American Colleges and Universities). 2009. “The VALUE Project Overview.” Peer Review 11 (1): 4–7.
2. ACER (Australian Council for Educational Research). 2014. “Graduate Skills Assessment (GSA).” ACER. Accessed November 19, 2014. http://www.acer.edu.au/tests/gsa
3. ACT. 2015a. “ACT Products and Services. http://www.act.org/products.html
4. ACT. 2015b. “Collegiate Assessment of Academic Proficiency.” http://www.act.org/caap/
5. AERA, APA, and NCME (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education). 2014. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.
6. Allen, J., and R. van der Velden. 2005. “The Role of Self-Assessment of Skills.” Research Centre for Education and the Labour Market. Maastricht University (REFLEX Working paper, 2).
7. Arum, R., and J. Roksa. 2010. Academically Adrift. Limited Learning on College Campuses. Chicago, IL: The University of Chicago Press.
8. Baer, M., G. Dörr, and U. Fraefel. 2006. “Standarderreichung in der Lehrerinnen- und Lehrerausbildung. Analyse der Wirksamkeit der berufsfeldorientierten Ausbildung [Standard Attainability in Teacher Training. Analysis of the Effectiveness of Job-oriented Training]”. In Qualität durch Standards? Beiträge zum Schwerpunktthema der 67. Tagung der AEPF [Quality through Standards? Contributions to the Main Emphasis of the 67th AEPF Conference], edited by F. Eder, 237–51. Münster: Waxmann.
9. Beck, E., M. Baer, T. Guldimann, S. Bischoff, C. Brühwiler, P. Müller, R. Niedermann, M. Rogalla, and F. Vogt. 2008. Adaptive Lehrkompetenz. Analyse und Struktur, Veränderbarkeit und Wirkung handlungssteuernden Lehrerwissens [Adaptive Teaching Competency. Analysis and Structure, Variability and Effect of Teacher's Action-controlling Knowledge]. Münster: Waxmann.
10. Bender, E., N. Schaper, M. E. Caspersen, M. Margaritis, and P. Hubwieser. 2015. “Identifying and Formulating Teachers' Beliefs and Motivational Orientations for Computer Science Teacher Education”. Studies in Higher Education. doi:10.1080/03075079.2015.1004233
11. Blömeke, S., A. Bremerich-Voss, G. Kaiser, G. Nold, H. Haudeck, J.-U. Keßler, and K. Schwippert, eds. 2013. Professionelle Kompetenzen im Studienverlauf. Weiterer Ergebnisse zur Deutsch-, Englisch- und Mathematiklehrerausbildung aus TEDS-LT [Professional Competencies During the Course of Study. Further TEDS-LT Results on the Training of Teachers in the Areas German, English and Mathematics]. Münster: Waxmann.
12. Blömeke, S., U. Suhl, and G. Kaiser. 2011. “Teacher Education Effectiveness: Quality and Equity of Future Primary Teachers’ Mathematics and Mathematics Pedagogical Content Knowledge.” Journal of Teacher Education 62 (2): 154–71. doi: 10.1177/0022487110386798
13. Blömeke, S., O. Zlatkin-Troitschanskaia, C. Kuhn, and J. Fege, eds. 2013. Modeling and Measuring Competencies in Higher Education – Tasks and Challenges. Rotterdam: Sense Publishers.
14. Boyle, B., I. Lamprianou, and T. Boyle. 2005. “A Longitudinal Study of Teacher Change: What Makes Professional Development Effective? Report of the Second Year of the Study.” Journal of School Effectiveness and School Improvement 16 (1): 1–27. doi: 10.1080/09243450500114819
15. Brennan, J., and D. Jary. 2005. “What is Learned at University? The Social and Organisational Mediation of University Learning: A Research Project.” London (SOMUL Working Paper, 1).
16. Brennan, J., K. Patel, K., and W. Tang. 2009. “Diversity in the Student Learning Experience and Time Devoted to Study: A Comparative Analysis of UK and European Evidence.” Report to HEFCE by Centre for Higher Education Research and Information (CHERI), The Open University. http://www.hefce.ac.uk/media/hefce/content/pubs/2009/rd0609/rd06_09.pdf
17. Briggs, D. C. 2011. “Cause or Effect? Validating the Use of Tests for High-stakes Inferences in Education.” In Looking Back: Proceedings of a Conference in Honor of Paul W. Holland, edited by N. J. Dorans and S. Sinharay, 131–47. New York: Springer.
18. Brownell, S., and M. Kloser. 2015. “Toward a Conceptual Framework for Measuring the Effectiveness of Eourse-based Undergraduate Research Experiences in Undergraduate Biology.” Studies in Higher Education. doi:10.1080/03075079.2015.1004234
19. Brückner, S., M. Förster, O. Zlatkin-Troitschanskaia, and W. B. Walstad. 2015. “Effects of Prior Economic Education, Native Language, and Gender on Economic Knowledge of First-Year Students in Higher Education. A Comparative Study between Germany and the United States.” Studies in Higher Education. doi:10.1080/03075079.2015.1004235
20. CAE (Council for Aid to Education). 2014. “Collegiate Learning Assessment (CLA).” http://www.collegiatelearningassessment.org/
21. Carrillo-de-la-Pena, M. T., E. Bailles, X. Caseras, A. Martinez, G. Ortet, and J. Perez. 2009. “Formative Assessment and Academic Achievement in Pre-Graduate Students of Health Sciences.” Advances in Health Sciences Education 14 (1): 61–7. doi: 10.1007/s10459-007-9086-y
22. CENEVAL (Centro Nacional de Evaluación para la Educación Superior). 2014a. “Examen General Para el Egreso de la Licenciatura (EGEL).” http://www.ceneval.edu.mx/ceneval-web/content.do?page=1676
23. CENEVAL (Centro Nacional de Evaluación para la Educación Superior). 2014b. “Examen Nacional de Ingreso al Posgrado (EXANI-III).” http://www.ceneval.edu.mx/ceneval-web/content.do?page=1675
24. CENEVAL (Centro Nacional de Evaluaciòn para la Educación Superior). 2014c. “Exámenes Generales para el Egreso del Técnico Superior Universitario (EGETSU).” http://www.ceneval.edu.mx/ceneval-web/content.do?page=2098
25. Clauser, J. C., and R. K. Hambleton. 2012. “Item Analysis Procedures for Classroom Assessments in Higher Education.” In Handbook on Measurement, Assessment, and Evaluation in Higher Education, edited by C. Secolsky and B. D. Denison, 296–309. New York: Routledge.
26. Coates, H., ed. 2014. Higher Education Learning Outcomes Assessment – International Perspectives. Frankfurt/Main: Peter Lang.
27. Coates, H., and D. Edwards. 2008. Work Readiness Assessment Package (WRAP). Camberwell: Australian Council for Educational Research (ACER).
28. Coates, H., and M. Mahat. 2014. “Assessing Student Engagement and Outcomes: Modelling Insights from Australia and Around the World.” International Journal of Chinese Education 2 (2): 241–64. doi: 10.1163/22125868-12340023
29. Coates, H., and A. Radloff. 2008. “Tertiary Engineering Capability Assessment. Concept Design.” https://minerva-access.unimelb.edu.au/bitstream/handle/11343/28875/264246_pub_2008_coates_tertiary_report.pdf?sequence=1
30. College Board. 2014. “SAT.” http://www.collegeboard.com/
31. Council of Chief State School Officers. 2010. “The Model Core Teaching Standards: A Ressource for State Dialogue.” http://www.ccsso.org/Documents/2010/InTASC_FAQ_2010.pdf
32. Creech, A., I. Papageorgi, C. Duffy, F. Morton, E. Hadden, J. Potter, C. De Bezenac, T. Whyton, E. Himonides, and G. Welch. 2008. “Investigating Musical Performance: Commonality and Diversity among Classical and Non-classical musicians.” Music Education Research 10 (2): 215–34. doi: 10.1080/14613800802079080
33. Domingue, B. 2014. “Higher education VA: Results from Colombia.” Paper presented at the annual meeting of the American Educational Research Association, Philadelphia, April 3–7.
34. Dresel, M., B. Schmitz, B. Schober, C. Spiel, A. Ziegler, T. Engelschalk, G. Jöstl, J. Klug, A. Roth, B. Wimmer, and G. Steuer. 2015. “Competencies for Successful Self-regulated Learning in Higher Education: Structural Model and Indications Drawn from Expert Interviews.” Studies in Higher Education. doi:10.1080/03075079.2015.1004236
35. ETS (Educational Testing Service). 2014a. “Graduate Record Examination (GRE) General Test.” http://www.ets.org/portal/site/ets/menuitem.fab2360b1645a1de9b3a0779f1751509/
36. ETS (Educational Testing Service). 2014b. “Major Field Tests.” http://www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=f119af5e44df4010VgnVCM10000022f95190RCRD&vgnextchannel=86f346f1674f4010VgnVCM10000022f95190RCRD
37. ETS (Educational Testing Service). 2014c. “Measure of Academic Proficiency and Progress (MAPP).” http://www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=ff3aaf5e44df4010VgnVCM10000022f95190RCRD&vgnextchannel=f98546f1674f4010VgnVCM10000022f95190RCRD
38. Ewell, P. T. 2005. “Can Assessment Serve Accountability? It Depends on the Question.” In Achieving Accountability in Higher Education, edited by J. C. Burke and Associates, 1–24. San Francisco, CA: Jossey-Bass.
39. Frey, A. 2008. Kompetenzstrukturen von Studierenden in der ersten und zweiten Phase der Lehrerbildung. Eine nationale und internationale Standortbestimmung [Competency Structures of Students in the First and Second Phase of Teacher Training. A National and International Determination of Location]. Landau: Empirische Pädagogik.
40. FUNDAÇÃO CESGRANRIO. 2014. “Exame Nacional de Cursos (ENC or Provão).” http://www.inep.gov.br/superior/provao/default.asp
41. FUNDAÇÃO CESGRANRIO and CESPE. 2014. “Exame Nacional de Desempenho dos Estudantes (ENADE).” http://www.inep.gov.br/superior/enade/default.asp
42. GCA (Graduate Careers Australia). 2014a. “Australian Graduate Survey.” http://www.graduatecareers.com.au/research/surveys/australiangraduatesurvey/
43. GCA (Graduate Careers Australia). 2014b. “CEQuery.” http://www.graduatecareers.com.au/research/cequery/
44. Haertel, G., E. Wentland, L. Yarnall, and R. J. Mislevy. 2012. “Evidence-Centered Design in Assessment Development.” In Handbook on Measurement, Assessment, and Evaluation in Higher Education, edited by C. Secolsky and B. D. Denison, 257–76. New York: Routledge.
45. Hambleton, R. K., P. F. Merenda, and C. D. Spielberger, eds. 2005. Adapting Educational and Psychological Tests for Cross-cultural Assessment. Hillsdale, NJ: Lawrence Erlbaum Publishers.
46. Henze, I., and J. H. van Driel. 2006. “The Development of Experienced Science Teachers. Pedagogical Content Knowledge in the Context of Educational Innovation.” In Towards Research-based Science Teacher Education. Proceedings of the 18th Symposium on Chemical and Science Education held at the University of Bremen, 15–17 June 2006, edited by I. Eilks and B. Ralle, 99–112. Aachen: Shaker.
47. Humburg, M., and R. van der Velden. 2015. “Self-assessments or Tests? Comparing Cross-National Differences in Patterns and Outcomes of Graduates' Skills Based on International Large-Scale Surveys.” Studies in Higher Education. doi:10.1080/03075079.2015.1004237
48. IES National Center for Educational Statistics. 2014. “Highlights from the 2003 International Adult Literacy and Lifeskills Survey (ALL).” http://nces.ed.gov/Surveys/ALL/issuebrief.asp
49. Jones, R. 2008. “Evaluating a Cross-Continent EU Simulation.” Journal of Political Science Education 4 (4): 404–34. doi: 10.1080/15512160802413790
50. Joumady, O., and C. Ris. 2005. “Performance in European Higher Education: A Non-Parametric Production Frontier Approach.” Education Economics 13 (2): 189–205. doi: 10.1080/09645290500031215
51. Kane, M. T. 2013. “Validating the Interpretations and Uses of Test Scores.” Journal of Educational Measurement 50 (1): 1–73. doi: 10.1111/jedm.12000
52. Klein, S. P., R. Benjamin, R. Shavelson, and R. Bolus. 2007. “The Collegiate Learning Assessment. Facts and Fantasies.” Evaluation Review 31 (5): 415–39. doi: 10.1177/0193841X07303318
53. Kuhn, C., M. Toepper, and O. Zlatkin-Troitschanskaia. 2014. “Current International State and Future Perspectives on Competence Assessment in Higher Education – Report from the KoKoHs Affiliated Group Meeting at the AERA Conference on April 4, 2014 in Philadelphia (USA) (KoKoHs Working Papers, 6).” Berlin: Humboldt University & Johannes Gutenberg University.
54. Kuhn, C., and O. Zlatkin-Troitschanskaia. 2011. Assessment of Competencies among University Students and Graduates – Analyzing the State of Research and Perspectives. Mainz: Johannes Gutenberg University Mainz. http://www.wipaed.uni-mainz.de/ls/ArbeitspapiereWP/gr_Nr.59.pdf
55. Kulikowich, J. M., and P. A. Alexander. 2003. “Cognitive Assessment.” In The Encyclopedia of Cognitive Science, edited by L. Nadel, Vol. 1, 526–32. London: Nature Publishing Group.
56. Mallinger, R., C. Holzbaur, M. Dierich, M. Heidegger, K. D. Hänsgen, and B. Spicher. 2008. “EMS Eignungstest für das Medizinstudium in Österreich.” Zentrum für Testentwicklung Universität Fribourg/Schweiz, Medizinische Universität Innsbruck, Medizinische Universität Wien. http://www.eignungstest-medizin.at/Bericht_EMSAT08.pdf
57. Mamlok-Naaman, R., D. Taitelbaum, M. Carmeli, and A. Hofstein. 2006. “A Model of Professional Development of Chemistry Teachers.” In Towards Research-based Science Teacher Education. Proceedings of the 18th Symposium on Chemical and Science Education held at the University of Bremen, 15–17 June 2006, edited by I. Eilks and B. Ralle, 113–24. Aachen: Shaker.
58. Marino, J. P. 2014. “Colombia's Assessment System.” Paper presented at the annual meeting of the American Educational Research Association, Philadelphia, April 3–7.
59. Ministry of Advanced Education (British Columbia). 2014. “2007 ESL Survey: 2007 English as a Second Language Student Outcomes Survey Report.” http://www.aved.gov.bc.ca/esl/docs/2007_esl_survey_report.pdf
60. Mislevy, R. J., and G. D. Haertel. 2006. “Implications of Evidence-centered Design for Educational Testing.” Educational Measurement: Issues and Practice 25: 6–20. doi: 10.1111/j.1745-3992.2006.00075.x
61. Müller-Hartmann, A. 2005. “Interkulturelles Lernen in internationalen Telekollaborationsprojekten. Die Entwicklung von Erfahrungswissen angehender Fremdsprachenlehrer/innen [Intercultural Learning in International Telecollaboration Projects. The Development of Experience-based Knowledge of Prospective Foreign Language Teachers].” Fremdsprachen lehren und lernen 34: 192–206.
62. Musekamp, F., and J. Pearce. 2015. “Assessing Engineering Competencies: The Conditions for Educational Improvement.” Studies in Higher Education. doi:10.1080/03075079.2015.1004238
63. NCHEMS (National Center for Higher Education Systems). 2014. “National Survey of Student Engagement (NSSE).” http://nsse.iub.edu/index.cfm
64. Nilsson, P. 2006. “Student Teachers. Reflection.” In Towards Research-based Science Teacher Education. Proceedings of the 18th Symposium on Chemical and Science Education held at the University of Bremen, 15–17 June 2006, edited by I. Eilks and B. Ralle, 53–64. Aachen: Shaker.
65. OECD (Organisation for Economic Co-operation and Development). 2004. “PIAAC Draft Strategy Paper. Policy Objects, Strategic Options and Cost Implications.” http://www.oecd.org/dataoecd/3/3/34463133.pdf
66. OECD (Organisation for Economic Co-operation and Development). 2010. “PIAAC Draft Strategy Paper. Policy Objects, Strategic Options and Cost Implications.” http://www.oecd.org/dataoecd/3/3/34463133.pdf
67. OECD (Organisation for Economic Co-operation and Development). 2012. “AHELO Feasibility Study Report. Volume 1. Design and Implementation.” http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf
68. OECD (Organisation for Economic Co-operation and Development). 2013a. “AHELO Feasibility Study Report. Volume 2. Data Analysis and National Experiences.” http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume2.pdf
69. OECD (Organisation for Economic Co-operation and Development). 2013b. “AHELO Feasibility Study Report. Volume 3. Further Insights.” http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume3.pdf
70. Osterlind, S. J., and Z. Wang. 2012a. “Item Response Theory in Measurement, Assessment, and Evaluation for Higher Education.” In Handbook on Measurement, Assessment, and Evaluation in Higher Education, edited by C. Secolsky and B. D. Denison, 150–60. New York: Routledge.
71. Osterlind, S. J., and Z. Wang. 2012b. “Statistical Modeling in Measurement, Assessment, and Evaluation for Higher Education.” In Handbook on Measurement, Assessment, and Evaluation in Higher Education, edited by C. Secolsky and B. D. Denison, 208–15. New York: Routledge.
72. Pascarella, E., and P. Terenzini. 2005. How College Affects Students: Findings and Insights from Twenty Years of Research. Volume 2: A Third Decade of Research. San Francisco, CA: Jossey-Bass.
73. Pellegrino, J. W., N. Chudowsky, and R. Glaser, eds. 2001. Knowing What Students Know: The Science and Design of Educational Assessment. Washington, DC: The National Academies Press.
74. Pitton, A. 2005. “Relevanz fachdidaktischer Forschungsergebnisse für die Lehrerbildung.” Jahrestagung der GDCP in Heidelberg 2004. Münster: LIT.
75. Pokorny, M., and H. Pokorny. 2005. “Widening Participation in Higher Education. Student Quantitative Skills and Independent Learning as Impediments to Progression.” International Journal of Mathematical Education in Science and Technology 36 (5): 445–67. doi: 10.1080/00207390500062621
76. Rychen, D. S. 2004. “Key Competencies for All: An Overarching Conceptual Frame of Reference.” In Developing Key Competencies in Education: Some Lessons from International and National Experience, edited by D. S. Rychen and A. Tiana, 5–34. Paris: UNESCO.
77. Schmeiser, C. B., and C. J. Welch. 2006. “Test Development.” In Educational Measurement, edited by R. L. Brennan, 4th ed., 307–53. Westport, CT: Praeger.
78. Secolsky, C., and B. D. Denison, eds. 2012. Handbook on Measurement, Assessment, and Evaluation in Higher Education. New York: Routledge.
79. Shavelson, R. J. 2010. Measuring College Learning Responsibly: Accountability in a New Era. Stanford, CA: Stanford University Press.
80. Shavelson, R. J. 2013. “An Approach to Testing & Modeling Competence.” In Modeling and Measuring Competencies in Higher Education. Tasks and Challenges, edited by S. Blömeke, O. Zlatkin-Troitschanskaia, C. Kuhn, and J. Fege, 29–43. Rotterdam: Sense Publishers.
81. Shavelson, R. J., S. Klein, and R. Benjamin. 2014. “The Limitations of Portfolios.” Inside Higher Education. http://www.insidehighered.com/views/2009/10/16/shavelson
82. Shulman, L. S. 1987. “Knowledge and Teaching: Foundations of the New Reform.” Harvard Educational Review 57 (1): 1–22.
83. Smith, A., A. McCull, A. Moran, and L. Clarke. 2008. A Value-based Approach to Teacher Education: Non-Technical Summary. Swindon: Economic and Social Research Council (ESRC).
84. Spiel, C., B. Schober, and M. Litzenberger. 2008. Evaluation der Eignungstests für das Medizinstudium in Österreich [Evaluation of Aptitude Tests for Austrian Students in the Field of Medicine]. Wien (project report).
85. Swedish National Agency for Higher Education. 2014. “New Entrants to Higher Education.” http://www.hsv.se/highereducationinsweden/studyingatheis/newentrantstohighereducation.4.28afa2dc11bdcdc557480002489.html
86. Tremblay, K. 2013. “OECD Assessment of Higher Education Learning Outcomes (AHELO): Rationale, Challenges and Initial Insights from the Feasibility Study.” In Modeling and Measuring Competencies in Higher Education, edited by S. Blömeke, O. Zlatkin-Troitschanskaia, C. Kuhn, and J. Fege, 113–26. Rotterdam: Sense Publishers.
87. Walstad, W., and K. Rebeck. 2008. “The Test of Understanding of College Economics.” American Economic Review 98 (2): 547–51. doi: 10.1257/aer.98.2.547
88. Webb, N. M., R. J. Shavelson, and J. T. Steedle. 2012. “Generalizability Theory in Assessment Contexts.” In Handbook on Measurement, Assessment, and Evaluation in Higher Education, edited by C. Secolsky and B. D. Denison, 132–49. New York: Routledge.
89. Weinert, F. E. 2001. “Concept of Competence: A Conceptual Clarification.” In Defining and Selecting Key Competencies, edited by D. S. Rychen and L. H. Salganik, 45–65. Seattle, WA: Hogrefe and Huber Publishers.
90. Wiley, E., and R. Shavelson. 2014. “Conceptual and Statistical Considerations in Estimating College Value Added to Learning.” Paper presented at the annual meeting of the American Educational Research Association, Philadelphia, April 3–7.
91. Winch, C., and L. Foreman-Peck. 2004. “Editorial”. Policy Futures in Education 2 (1): 1–4. doi: 10.2304/pfie.2004.2.1.1
92. Wolf, R., D. Zahner, and R. Benjamin. 2015. “Methodological Challenges in International Comparative Post-Secondary Assessment Programs: Lessons Learned and the Road Ahead.” Studies in Higher Education. doi:10.1080/03075079.2015.1004239
93. Yamaoka, M., and T. Asano. 2015. “How to Reason Economic Concepts: A Cognitive Process of Japanese Undergraduate Students.” Studies in Higher Education. doi:10.1080/03075079.2015.1004240
94. Zlatkin-Troitschanskaia, O., C. Kuhn, and M. Toepper. 2014. “Modelling and Assessing Higher Education Learning Outcomes in Germany.” In Higher Education Learning Outcomes Assessment – International Perspectives, edited by H. Coates, 213–35. Frankfurt/Main: Peter Lang.