Methodological challenges in international comparative post-secondary assessment programs: lessons learned and the road ahead

Studies in Higher Education

Volumn 40, Issue 3, 2015, Pages 471-481

  Wolf, Raffaela ^a   Zahner, Doris ^a   Benjamin, Roger ^a  

^a CAE ^*  (United States)

Author keywords

AHELO; Collegiate Learning Assessment; international assessment programs

Indexed keywords

EID: 84924902609     PISSN: 03075079     EISSN: 1470174X     Source Type: Journal    
DOI: 10.1080/03075079.2015.1004239     Document Type: Article

References (31)

1. AHELO. 2012a. AHELO Feasibility Study Interim Report. Paris: OECD.
2. AHELO. 2012b. Testing Students and University Performance Globally: OECD's AHELO. Accessed January 3, 2014. http://www.oecd.org/edu/skills-beyond-school/testingstudentanduniversityperformancegloballyoecdsahelo.htm.
3. Angrist, J. D., and A. B. Krueger. 2001. Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments (Working Paper 8456). Accessed July 7, 2014. http://www.nber.org/papers/w8456.pdf.
4. Benjamin, R. 2014. Leveling the Playing Field from College to Career. New York: Council for Aid to Education.
5. Brand, J. E., and C. N. Halaby. 2006. “Regression and Matching Estimates of the Effects of Elite College Attendance on Educational and Career Achievement.” Social Science Research 35 (3): 749–70. doi: 10.1016/j.ssresearch.2005.06.006
6. Byun, S.-y. 2010. “Does Policy Matter in Shadow Education Spending? Revisiting the Effects of the High School Equalization Policy in South Korea.” Asia Pacific Education Review 11 (1): 83–96. doi: 10.1007/s12564-009-9061-9
7. Cole, J. S., D. A. Bergin, and T. A. Whittaker. 2008. “Predicting Student Achievement for Low Stakes Tests with Effort and Task Value.” Contemporary Educational Psychology 33: 609–24. doi: 10.1016/j.cedpsych.2007.10.002
8. Cole, J. S., and S. J. Osterlind. 2008. “Investigating Differences Between Low- and High-stakes Test Performance on a General Education Exam.” The Journal of General Education 57 (2): 119–30. doi: 10.1353/jge.0.0018
9. Cook, T. D., P. M. Steiner, and S. Pohl. 2009. “How Bias Reduction is Affected by Covariate Choice, Unreliability, and Mode of Data Analysis: Results from Two Types of Within-study Comparisons.” Multivariate Behavioral Research 44 (6): 828–47. doi: 10.1080/00273170903333673
10. Frisco, M. L., C. Muller, and K. Frank. 2007. “Parents’ Union Dissolution and Adolescents’ School Performance: Comparing Methodological Approaches.” Journal of Marriage and Family 69 (3): 721–41. doi: 10.1111/j.1741-3737.2007.00402.x
11. Haberman, S. J. 2008. “When Can Subscores Have Value?” Journal of Educational and Behavioral Statistics 33 (2): 204–29. doi: 10.3102/1076998607302636
12. Haberman, S., S. Sinharay, and G. Puhan. 2009. “Reporting Subscores for Institutions.” British Journal of Mathematical and Statistical Psychology 62 (1): 79–95. doi: 10.1348/000711007X248875
13. Hartig, J., and J. Höhler. 2008. “Representation of Competencies in Multidimensional IRT Models with Within-Item and Between-Item Multidimensionality.” Zeitschrift für Psychologie [Journal of Psychology] 216 (2): 89–101. doi: 10.1027/0044-3409.216.2.89
14. Hart Research Associates. 2009. Learning and Assessment: Trends in Undergraduate Education – A Survey among Members of the Association of American Colleges and Universities. Washington, DC: Hart Research Associates.
15. Hong, G., and B. Yu. 2008. “Effects of Kindergarten Retention on Children's Social-emotional Development: An Application of Propensity Score Method to Multivariate, Multilevel Data.” Developmental Psychology 44 (2): 407–21. doi: 10.1037/0012-1649.44.2.407
16. Hoyle, R. H. 1995. Structural Equation Modeling: Concepts, Issues, and Applications. Thousand Oaks, CA: Sage.
17. Klein, S., O. L. Liu, J. Sconing, R. Bolus, B. Bridgeman, H. Kugelmass, A. Nemeth, S. Robbins, J. Steedle. 2009. Test Validity Study (TVS) Report. Supported by the Fund for the Improvement of Postsecondary Education. Accessed June 2, 2013. http://www.cae.org/content/pdf/TVS_Report.pdf.
18. Klein, S., D. Zahner, R. Benjamin, R. Bolus, and J. Steedle. 2013. Observations on AHELO's Generic Skills Strand Methodology and Findings. New York: Council for Aid to Education.
19. Lunz, M. E., and J. M. Linacre. 1998. “Measurement Designs Using Multifacet Rasch Modeling.” In Modern methods for business research, edited by G. A. Marcoulides, 47–57. Mahwah, NJ: Lawrence Erlbaum Associates.
20. OECD. 2008a. Measuring Improvements in Learning Outcomes: Best Practices to Assess the Value-Added of Schools. Paris: Organisation for Economic Co-Operation and Development.
21. OECD. 2008b. Tertiary Education for the Knowledge Society. Paris: Organisation for Economic Co-operation and Development.
22. OECD. 2012. Assessment of Higher Education Learning Outcomes (Feasibility Study Report, Volume 1 – Design and Implementation). Paris: Organisation for Economic Co-Operation and Development.
23. OECD. 2013. Assessment of Higher Education Learning Outcomes (Feasibility Study Report, Volume 3 – Further Insights). Paris: Organisation for Economic Co-Operation and Development.
24. Ou, S.-R., and A. J. Reynolds. 2010. “Grade Retention, Postsecondary Education, and Public Aid Receipt.” Educational Evaluation and Policy Analysis 32 (1): 118–39. doi: 10.3102/0162373709354334
25. Puhan, G., S. Sinharay, S. Haberman, and K. Larkin. 2010. “The Utility of Augmented Subscores in a Licensure Exam: An Evaluation of Methods Using Empirical Data.” Applied Measurement in Education 23 (3): 266–85. doi: 10.1080/08957347.2010.486287
26. Raudenbush, S. W. 2005. “Learning from Attempts to Improve Schooling: The Contribution of Methodological Diversity.” Educational Researcher 34 (5): 25–31. doi: 10.3102/0013189X034005025
27. Rosenbaum, P. R., and D. B. Rubin. 1983. “The Central Role of the Propensity Score in Observational Studies for Causal Effects.” Biometrika 70 (1): 41–55. doi: 10.1093/biomet/70.1.41
28. Rosenbaum, P. R., and D. B. Rubin. 1984. “Reducing Bias in Observational Studies Using Subclassification on the Propensity Score.” Journal of the American Statistical Association 79 (387): 516–24. doi: 10.1080/01621459.1984.10478078
29. Wise, S. L., and C. E. DeMars. 2005. “Low Examinee Effort in Low-stakes Assessment: Problems and Potential Solutions.” Educational Assessment 10 (1): 1–17. doi: 10.1207/s15326977ea1001_1
30. Wolf, R., D. Zahner, F. Kostoris, and R. Benjamin. 2014. “A Case Study of An International Performance-based Assessment of Critical Thinking Skills.” Paper presented at the American Educational Research Association, Philadelphia, PA.
31. Zahner, D., and J. Steedle. 2014. “Evaluating Performance Task Scoring Comparability in an International Testing Program.” Paper presented at the American Educational Research Association, Philadelphia, PA.