Updating the duplex design for test-based accountability in the twenty-first century

Measurement

Volumn 8, Issue 2-3, 2010, Pages 110-129

  Bejar, Isaac I ^a   Graf, E Aurora ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 79960492496     PISSN: 15366367     EISSN: 15366359     Source Type: Journal    
DOI: 10.1080/15366367.2010.511976     Document Type: Article

References (93)

1. Argyris, C. (1997). Learning and Teaching: A Theory of Action Perspective. Journal of Management Education, 21(1), 9-26.
2. Atkinson, R. C., & Geiser, S. (2009). Reflections on a century of college admissions tests. Educational Researcher, 38(9), 665-676.
3. Attali, Y., & Powers, D. (2008). Effect of immediate feedback and revision on psychometric properties of open-ended GRE® subject test items (RR-08-21). Princeton, NJ: Educational Testing Service.
4. Baker, E. L., & Linn, R. L. (2004). Validity issues for accountability systems. In S. Fuhrman & R. Elmore (Eds.), Redesigning Accountability (pp. 47-72). New York: Teachers College Press.
5. Bejar, I. I., Braun, H., & Tannenbaum, R. (2007). A prospective, predictive and progressive approach to standard setting. In R. W. Lissitz (Ed.), Assessing and modeling cognitive development in school: Intellectual growth and standard setting (pp. 1-30). Maple Grove: MN: JAM Press.
6. Bejar, I. I., Graf, E. A., Oranje, A., & Veldkamp, B. P. (2008). Application of optimization methods to assessment design. Paper presented in April at the National Council of Measurement in Education, New York, NY.
7. Bejar, I. I., Lawless, R. R., Morley, M. E., Wagner, M. E., Bennett, R. E., & Revuelta, J. (2003). A feasibility study of on-the-fly item generation in adaptive testing. Journal of Technology, Learning, and Assessment, 2(3), Retrieved March 1, 2010, from http://www.jtla.org
8. Bennett, R. E. (2010). Cognitively based assessment of, for, and as learning (CBAL): A preliminary theory of action for summative and for mative assessment. Measurement: Interdisciplinary Research and Perspectives, 8(2-3), 70-91.
9. Bennett, R. E., & Gitomer, D. H. (2009). Transforming K-12 assessment: Integrating accountability testing, formative assessment and professional support. In C. Wyatt-Smith & J. Cumming (Eds.), Educational assessment in the 21st century (pp. 43-61). New York: Springer.
10. Betebenner, D. (2005). The importance of performance standards in measures of student growth. Paper presented at the CCSSO Retrieved March 1, 2010, from http://216.250.255.51/content/pdfs/Betebenner2005NCLSA.pdf
11. Betebenner, D. (2009). Norm-and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4), 42-51.
12. Betebenner, D. W., Shang, Y., Xiang, Y., Zhao, Y., & Yue, X. (2008). The impact of performance level misclassification on the accuracy and precision of percent at performance level measures. Journal of Educational Measurement, 45(2), 119-137.
13. Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80(2), 139-148.
14. Bock, R. D., & Mislevy, R. J. (1981). An item response curve model for matrix-sampling data: The California grade-three assessment. New Directions for Testing and Measurement, 10, 65-90.
15. Bock, R. D., & Mislevy, R. J. (1988). Comprehensive educational assessment for the states: The Duplex Design. Educational Evaluation and Policy Analysis, 10(2), 89-105.
16. Bransford, J. D., Brown, A. I., & Cocking, R. R. (1999). How people learn. Washington DC: National Academy Press.
17. Braun, H. (2009). Discussion: With choices come consequences. Educational Measurement: Issues and Practice, 28(4), 52-55.
18. Brennan, R. L., Ping, Y., & Kane, M. T. (2003). Methodology for examining the reliability of group mean difference scores. Journal of Educational Measurement, 40(3), 207-230.
19. Briggs, D. C., & Weeks, J. P. (2009). The impact of vertical scaling decisions on growth interpretations. Educational Measurement: Issues and Practice, 28(4), 3-14.
20. Carlson, D. (undated). Focusing state educational accountability systems: Four methods of judging school quality and progress [Electronic version]. Council of Chief State School Officers. Retrieved August 28, 2007, from http://www.ccsso.org/content/pdfs/Dale020402.doc
21. Clauser, B. E., Kane, M. T., & Swanson, D. B. (2002). Validity issues for performance-based tests scored with computerautomated scoring systems. Applied Measurement in Education, 15(4), 413-432.
22. Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. (1995). Generalizability analysis for educational assessments. Educational and Psychological Measurement, 57(3), 373-399.
23. Darling-Hammond, L. (2010). Performance counts: Assessment systems that support high-quality learning. Washington, DC, Council of Chief State School Officers. Retrieved March 1, 2010, from http://www.ccsso.org/publications/details.cfm?PublicationID=381
24. Darling-Hammond, L., & McCloskey, L. (2008). Assessment for learning around the world: What would It mean to be internationally competitive? Phi Delta Kappan, 90(4), 263-272.
25. Deane, P. (2010). Rethinking K-12 writing assessment. Unpublished manuscript.
26. Dorans, N., Holland, P., & Pomerich, M. (Eds.). (2007). Linking and aligning scores and scales New York: Springer-Verlag.
27. Drasgow, F., Luecht, R., & Bennett, R. E. (2006). Technology and testing. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 471-515). Westport, CT: Praeger.
28. Dunn, J. L., & Allen, J. (2009). Holding schools accountable for the growth of nonproficient students: Coordinating measurement and accountability. Educational Measurement: Issues and Practice, 28(4), 27-41.
29. Embretson, S. (1993). Psychometric models for learning and cognitive processes. In N. Frederiksen, R. J. Mislevy, & I. I. Bejar (Eds.), Test theory for a new generation of tests (pp. 125-150). Hillsdale, NJ: Erlbaum.
30. Embretson, S. E. (1983). Construct validity: Construct representation versus nomothetic span. Psychological bulletin, 93, 179-197.
31. Ferrara, S., Phillips, G.W., Williams, P. L., Leinwand, S., Mahoney, S., & Ahadi, S. (2007). Vertically articulated performance standards: An exploratory of inferences about achievement and growth. In R. W. Lissitz (Ed.), Assessing and modeling cognitive development in school: Intellectual growth and standard setting. Maple Grove: MN: Jam Press.
32. Feuer, M. J. (2008). Future directions for educational accountability: Notes for a political economy of measurement. In K. E. Ryan & L. A. Shepard (Eds.), The future of test-based educational accountability (pp. 115-137). New York: Routledge.
33. Feuer, M. J. (2010). Externalities of testing: Lessons from the Blizzard of 2010. Measurement: Interdisciplinary Research and Perspectives, 8, 59-69.
34. Forsyth, R. A. (1998). NAEP frameworks and achievement levels. In M. L. Bourque (ed.), Proceedings of achievement levels workshop: Boulder, CO: National Assessment Governing Board (Available from ERIC, Document ED458220).
35. Gambell, T., & Hunter, D. (2004). Teacher scoring of large-scale assessment: Professional development or debilitation? Journal of Curriculum Studies, 36(6), 697-724.
36. Goldberg, G. L., & Roswell, B. S. (2000). From perception to practice: The impact of teachers' scoring experience on performance-based instruction and classroom assessment. Educational Assessment, 6(4), 257-290.
37. Haberman, S. (2008). When can subscores have value? Journal of Educational and Behavioral Statistics, 33(2), 204.
38. Haertel, E. H. (1999). Validity arguments for high-stakes testing: In search of the evidence. Educational Measurement: Issues and Practice, 18(4), 5-9.
39. Haertel, E. H., & Herman, J. l. (2005). A historical perspective on validity arguments for accountability testing. Yearbook of the National Society for the Study of Education, 104(2), 1-34.
40. Hambleton, R. K., Brennan, R. L., Brown, W., Dodd, B., Forsyth, R. A., Mehrens, W. A., et al. (2000). A response to "Setting Reasonable and Useful Performance Standards" in the National Academy of Science' Grading the Nations Report Card. Educational Measurement: Issues and Practice, 19(2), 5-14.
41. Harris, D. J. (2007). Practical issues in vertical scaling. In N. J. Dorans, M. Pommerich, & P. W. Holland (Eds.), Linking and aligning scores and scales (pp. 231-252). New York: Springer-Verlag.
42. Hill, R. (2001). Issues related to the reliability of school accountability scores. The National Center for the Improvement of Educational Assessment. Retrieved March 1, 2010, from http://www.nciea.org/publication/RILS2000Paper_Hill01.pdf
43. Kane, M.T. (2006). Validation. In R. L. Brennan (Ed.), Educationalmeasurement (4th ed., pp. 17-64). Westport, CT:Praeger.
44. Kingsbury, G. G. (2007). CAT in the K-12 schools: 50 million CATs and counting. In D. J. Weiss (Ed.), Proceedings of the 2007 GMAC Conference on Computerized Adaptive Testing. Retrieved from http://www.psych.umn.edu/psylabs/catcentral/cat07schedule.htm
45. Kiplinger, V. L. (2008). Reliability of large-scale assessment and accountability systems. In K. E. Ryan & L. A. Shepard (Eds.), Reliability of large-scale assessment and accountability systems (pp. 93-114). New York: Routledge.
46. Kiplinger, V. L., & Hamilton, L. S. (2008). Equating and linking of educational assessments in high stakes accountability systems. In K. E. Ryan & L. A. Shepard (Eds.), The future of test-based educational accountability (pp. 115-137). New York: Routledge.
47. Kirsch, I. S., Jungeblut, A., Jenkins, L., & Kolstad, A. (1993). Adult literacy in America: A first look at the findings of the National Adult Literacy Survey. Washington, DC: National Center for Education Statistics. Retrieved March 1, 2010, from http://nces.ed.gov/pubs93/93275.pdf
48. Koretz, D. (2005). Alignment, high stakes, and the inflation of test scores. In J. Herman & E. Haertel (Eds.), Uses and misuses of data in accountability testing. Yearbook of the National Society for the Study of Education, vol. 104, Part 2 (pp. 99-118). Malden, MA: Blackwell.
49. Koretz, D. (2006). Testing for accountability in K-12. In R. Brennan (Ed.), Educational measurement (4th ed., pp. 531-578). Westport, CT: Praeger.
50. Koretz, D. (2008). Further steps toward the development of an accountability-oriented science of measurement. In K. E. Ryan & L. A. Shepard (Eds.), The future of test-based educational accountability (pp. 71-91). New York: Routledge.
51. Koretz, D. M., & Béguin, A. (2010). Self-monitoring assessments for educational accountability systems. Measurement: Interdisciplinary Research and Perspectives, 8(2-3), 92-109.
52. Lane, S., & Stone, C. A. (2002). Strategies for examining the consequences of assessment and accountability programs. Educational Measurement: Issues and Practice, 21(1), 23-30.
53. Leighton, J. P., & Gierl, M. J., (Eds.). (2007). Cognitive diagnostic assessment in education: Theory and applications. New York: Cambridge University Press.
54. Levin, H. M. (1974). A conceptual framework for accountability in education. School Review, 82(3), 363-391.
55. Linn, R. L. (2004). Accountability models. In S. Fuhrman & R. Elmore (Eds.), Redesigning accountability (pp. 73-93). New York: Teachers College Press.
56. Linn, R. L. (2005a). Conflicting demands of No Child Left Behind and state systems: Mixed messages about school performance. Education Policy Analysis Archives, 13(33). Retrieved March 16, 2010, from http://epaa.asu.edu/ojs/article/view/2138
57. Linn, R. L. (2005b). Fixing the NCLB accountability system (CRESST Policy Brief 8) Los Angeles: CA: CRESST. Retrieved December 15, 2007, from http://www.cse.ucla.edu/products/policy/cresst_policy8.pdf
58. Linn, R. L. (2008). Educational accountability systems. In K. E. Ryan & L. A. Shepard (Eds.), The future of test-based educational accountability (pp. 3-24). New York: Routledge.
59. Linn, R. L., & Burton, E. (1994). Performance-based assessment: Implications of task specificity. Educational measurement: Issues and practice, 13(1), 5-8.
60. Lissitz, R.W., & Huynh, H. (2003). Vertical equating for state assessments: Issues and solutions in determining adequate yearly progress and school accountability. Practical Assessment, Research, & Evaluation, 8(10). Retrieved March 1, 2010, from http://PAREonline.net.getvn.asp?v=8&n=10
61. Lord, F. M. (1980). Applications of item-response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
62. Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.
63. Mislevy, R. J. (2007). Validity by design. Educational Researcher, 36(8), 463-469.
64. Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25(4), 6-20.
65. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1(1), 3-62.
66. Myers, M. (2003). What can computers and AES contribute to a K-12 writing program? In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Lawrence Erlbaum.
67. National Council of Teachers of Mathematics. (2006). Curriculum focal points for prekindergarten through grade 8 mathematics. Retrieved September 12, 2006, from http://www.nctm.org/focalpoints/downloads.asp
68. Oregon Department of Education. (2008). 2007-2008 Technical Report: Oregon Statewide Assessment System. Retrieved March 26, 2010, from http://www.ode.state.or.us/search/page/?=1305
69. Patz, R. J., & Yao, L. (2007). Methods and models of vertical scaling. In N. J. Dorans, M. Pommerich, & P. W. Holland (Eds.), Linking and aligning scores and scales (pp. 253-272). New York: Springer-Verlag.
70. Pellegrino, J. W. (2000). A response to ACT's technical advisers on NAEP standard setting. Educational Measurement: Issues and Practice, 19(2), 14-15.
71. Pellegrino, J. W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
72. Perie, M. (2008). A guide to understanding and developing performance-level descriptors. Educational Measurement: Issues and Practice, 27(4), 15-29.
73. Plake, B. S., & Huff, K. (2009, April). Evidence-centered assessment design as a foundation for achievement level descriptor development and for standard setting. Paper presented at the National Council of Measurement in Education, San Diego, CA.
74. Puhan, G. (2009, April). Detecting and correcting scale drift in test equating: An illustration from a large scale testing program. Applied Measurement in Education, 22(1), 79-103.
75. Rawls, A., Yu, L., & Liu, Y. (2009). An overview of online assessments in statewide tests. Paper presented at the National Council of measurement in Education, San Diego, CA.
76. Ryan, K. E., & Shepard, L. A. (2008). The future of test-based educational accountability. New York: Routledge.
77. Scalise, K., & Gifford, B. (2006). Computer-based assessment in e-learning: A framework for constructing "intermediate constraint" questions and tasks for technology platforms. Journal of Technology, Learning, and Assessment, 4(6), Retrieved March 1, 2010, from http://www.jtla.org.
78. Sheingold, K., Heller, J. I., & Paulukonis, S. T. (1994). Actively seeking evidence: Teacher change through assessment development (Center for Performance Assessment Report No. 2 MS # 94-04). Princeton, NJ: Educational Testing Service.
79. Shepard, L. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14.
80. Shepard, L. A. (2006). Classroom assessment. In R. L. Brennan (Ed.), Educational Measurement (4th ed., pp. 623-646). Westport, CT: Praeger.
81. Shepard, L. A. (2008). A brief history of accountability testing, 1965-2007. In K. E. Ryan & L. A. Shepard (Eds.), The future of test-based educational accountability (pp. 25-46). New York: Routledge.
82. Stecher, B. M., & Hanser, L. M. (1992). Local accountability in vocational education: A theoretical model and its limitations in practice. Palo Alto, CA: Rand Corporation.
83. Tate, R. L., & King, F. J. (1994). Factors which influence precision of school-level IRT ability estimates. Journal of Educational Measurement, 31(1), 1-15.
84. United States Congress. (2001). No Child Left Behind Act of 2001: Conference report to accompany H.R. 1 (Report 107-334). Washington, DC: Government Printing Office.
85. V-model. (2004, July 22). Wikipedia, the free encyclopedia. Retrieved April 6, 2009, from http://en.wikipedia.org/wiki/Sudoku
86. Wainer, H., Hambleton, R. K., & Meara, K. (1999). Alternative displays for communicating NAEP results: A redesign and validity study. Journal of Educational Measurement, 36(4), 301-335.
87. Wang, S., & Jiao, H. (2009). Construct equivalence across grades in a vertical scale for a K-12 large-scale reading assessment. Educational and Psychological Measurement, 69(5), 760-777.
88. Weiss, D. J. (Ed.). (1983). New horizons in testing: Latent trait test theory and computerized adaptive testing. NewYork: Academic Press.
89. Wholey, J. S. (1979). Evaluation-promise and performance. Washington: Urban Institute.
90. Williamson, D. M., Mislevy, R. J., & Bejar, I. I. (2006). Automated scoring of complex tasks in computer-based testing. Mahwah, NJ: Lawrence Erlbaum Associates.
91. Wilson, M. (2009). Measuring progressions: Assessment structures underlying a learning progression. Journal of Research in Science Teaching, 46(6), 716-730.
92. Wilson, M. (Ed.). (2004). Towards coherence between classroom assessment and accountability. Chicago: Chicago University Press.
93. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In N. J. Dorans, M. Pommerich, & P. W. Holland (Eds.), Linking and aligning scores and scales (pp. 273-284). New York: Springer-Verlag.