Developing and evaluating instructionally sensitive assessments in science

Journal of Research in Science Teaching

Volumn 49, Issue 6, 2012, Pages 691-712

  Ruiz Primo, Maria Araceli ^a   Li, Min ^b   Wills, Kellie ^b   Giamellaro, Michael ^a   Lan, Ming Chih ^b   Mason, Hillary ^a   Sands, Deanna ^a  

^a UNIVERSITY OF COLORADO   (United States)

^b UNIVERSITY OF WASHINGTON   (United States)

Author keywords

assessment development; instructional sensitivity; science assessments

Indexed keywords

ASSESSMENT DEVELOPMENT; DEVELOPMENT APPROACH; EFFECT SIZE; EMPIRICAL EVIDENCE; INSTRUCTIONAL SENSITIVITY; LIFE-SCIENCES; RESEARCH QUESTIONS; SCIENCE ASSESSMENTS; SCIENCE MODULES; THREE DIMENSIONS;

EDUCATION; RESEARCH;

CURRICULA;

EID: 84863841351     PISSN: 00224308     EISSN: 10982736     Source Type: Journal    
DOI: 10.1002/tea.21030     Document Type: Article

References (61)

1. Airasian, P. W., &, Madaus, G. F., (1983). Linking testing and instruction. Journal of Educational Measurement, 20, 103-118.
2. Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Wittrock, M. C., (2001). A taxonomy for learning, teaching, and assessing. A revision of Bloom's taxonomy of educational objectives. New York, NY: Longman.
3. Barnett, S. M., &, Ceci, S. J., (2002). When and where do we apply what we learned? A taxonomy of far transfer. Psychological Bulletin, 128 (4), 612-637.
4. Bransford, J. D., Brown, A. L., &, Cocking, R. R., (1999). How people learn: Brain, mind, and school. National Research Council. Washington, D.C.: National Academies Press.
5. Briddle, B. J., (1997). Foolishness, dangerous nonsense, and real correlates of state differences in achievement. Phi Delta Kappan, 79 (1), 8-13. Retrieved January 4, 2007 from.
6. Brown, A. L., &, Kane, M. J., (1988). Preschool children can learn to transfer: Learning to learn and learning from example. Cognitive Psychology, 20, 493-523.
7. Brown, A. L., Kane, M. J., &, Echols, C. H., (1986). Young children's mental models determine analogical transfer across problems with common goal structure. Cognitive Development, 1, 103-121.
8. Burstein, L., (1989). Conceptual considerations in instructionally sensitive assessments. CSE Technical Report 333. Center for Research on Evaluation, Standards, and Student Testing. Los Angeles, CA: University of California, Los Angeles.
9. Burstein, L., Aschbacher, P., Chen, Z., &, Lin, L., (1990). Establishing the content validity of tests designed to serve multiple purposes: Bridging secondary-postsecondary mathematics. CSE Technical Report 313. Center for Research on Evaluation, Standards, and Student Testing. Los Angeles, CA: University of California, Los Angeles.
10. Cohen, J., (1992). A power primer. Psychological Bulletin, 112 (1), 155-159.
11. Commission on Instructionally Supportive Assessment. (2001). Building tests to support instruction and accountability: A guide for policymakers (James Popham, Commission Chair). Washington, D.C.: American Association of School Administrators. Retrieved December 23, 2007, from.
12. Cooley, W. W., &, Leinhardt, G., (1980). The instructional dimension study. Educational Evaluation and Policy Analysis, 2 (1), 7-25.
13. Cox, R. C., &, Vargas, J. S., (1966). A comparison of item-selection techniques for norm referenced and criterion referenced tests. Paper presented at the Annual Conference of the National Council on Measurement in Education, Chicago, IL.
14. Cronbach, L. J., (1988). Five perspectives on validity arguments. In, H. Wainer, &, H. I. Braun, (Eds.), Test validity (pp. 3-17). Hillsdale, NJ: Lawrence Earlbaum.
15. D'Agostino, J. V., Welsh, M. E., &, Corson, N. M., (2007). Instructional sensitivity of a state's standards-based assessment. Educational Assessment, 12 (1), 1-22.
16. Darling-Hammond, L., (1999). Teacher quality and student achievement: A review of state policy evidence (Document R-99-1). Retrieved January 4, 2008, from the Center for the Study of teaching and Policy Web site
17. Darling-Hammond, L., (2007). Race, inequality, and educational accountability: The irony of 'No Child Left Behind'. Race Ethnicity and Education, 10 (3), 245-260. (Pubitemid 47393856)
18. Giamellaro, M., Lan, M., Ruiz-Primo, M. A., Li, M., (2011). Addressing elementary teachers' misconceptions in science and supporting peer learning through curriculum mapping. Paper presented at National Association for Research in Science Teaching annual meeting, Orlando, FL.
19. Giamellaro, M., Ruiz-Primo, M. A., Li, M., (2012a). Quality elementary science teaching as reflected in productive failure. Paper presented at the National Association for Research in Science Teaching annual meeting, Indianapolis.
20. Giamellaro, M., Sands, D., Wills, K., Ruiz-Primo, M. A., Li, M., (2012b). Is this testing what was taught? Teachers' and students' perceptions of instructionally sensitive assessments. Paper presented at the American Educational Research Association annual meeting, Vancouver, B.C.
21. Haertel, E. H., (1999). Validity arguments for high-stakes testing: In search of the evidence. Educational Measurement: Issues and Practice, 18 (4), 5-9.
22. Haertel, G., Lash, A., Javitz, H., &, Quellmalz, E., (2006). An instructional sensitivity study of science inquiry items from three large-scale science examinations. Paper presented at the Annual Meeting of the American Educational Research Association. San Francisco, CA.
23. Haladyna, T., &, Roid, G., (1981). The role of instructional sensitivity in the empirical review of criterion-referenced test items. Journal of Educational Measurement, 18 (1), 39-53.
24. Hickey, D. T., &, Pellegrino, J. W., (2005). Theory, level, and function. Three dimensions for understanding transfer and student assessment. In, J. P. Mestre, (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 251-293). Greenwich, CN: Information Age Publishing.
25. Judd, C. H., (1908). The relation of special training to general intelligence. Educational Review, 36, 28-42.
26. Kane, M. T., (1992). An argument based approach to validity. Psychological Bulletin, 112, 527-535.
27. Kane, M. T., (2006). Validation. In, R. L. Brennan, (Ed.), Educational measurement (4th ed., pp. 17-64). Westpot, CN: Praeger Publishers.
28. Lan, M.-C., Li, M., Ruiz-Primo, M. A., Wang, T., Giamellaro, M., Mason, H., (2012). Linking quality of instruction to instructionally sensitivity assessments. Paper presented at the American Educational Research Association annual meeting, Vancouver, B.C.
29. Leinhardt, G., (1983). Overlap: Testing whether it is taught. In, G. F. Madaus, (Ed.), The courts, validity, and minimum competency testing (pp. 153-170). Boston, MA: Kluwer-Nijhoff Publishing.
30. Leinhardt, G., &, Seewald, A. M., (1981). Overlap: What's tested, what's taught ? Journal of Educational Measurement, 18 (2), 85-96.
31. Li, M., (2001). A framework for science achievement and its link to test items. Unpublished dissertation. Stanford, CA: Stanford University.
32. Li, M., Lan, M.-C., Ruiz-Primo, M. A., Giamellaro, M., &, Wang, T., (2012). Supporting students to make conceptual connections. Paper presented at the NARST annual meeting, Minneapolis, IN.
33. Li, M., Ruiz-Primo, M. A., &, Shavelson, R. J., (2006). Towards a science achievement framework: The case of TIMSS 1999. In, S. Howie, &, T. Plomp, (Eds.), Contexts of learning mathematics and science: Lessons learned from TIMSS (pp. 291-311). London: Routledge.
34. Li, M., &, Shavelson, R. J., (2001). Examining the linkage between science achievement and assessment. Paper presented at the AERA Annual Meeting, Seattle, WA.
35. Linn, R. L., (1983). Curricular validity: Convincing the courts that it was taught without precluding the possibility of measuring it. In, G. F. Madaus, (Ed.), The courts, validity, and minimum competency testing (pp. 115-132). Boston, MA: Kluwer-Nijhoff Publishing.
36. Madaus, G. F., Airasian, P. W., &, Kellaghan, T., (1980). School effectiveness: a reassessment of the evidence. New York: McGraw-Hill.
37. Messick, S., (1989). Validity. In, R. Linn, (Ed.), Educational measurement (pp. 201-219). New York, NY: Macmillan Publishing Company.
38. Mestre, J., (2002). Transfer of learning: Issues and research agenda (No. NSF03-212). Arlington, VA: National Science Foundation.
39. Muthén, B, (1988). Instructionally sensitive psychometrics: Applications to the Second International Mathematics Study. Office of Educational Research and Improvement. Technical Report 143 (OERI-G-86-003).
40. Muthén, B., Kao, C.-F., &, Burnstein, L., (1988). Instructional sensitivity in mathematics achievement test items: Application of a new IRT-based detection. CSE Technical Report 292. Center for Research on Evaluation, Standards, and Student Testing. Los Angeles, CA: University of California, Los Angeles.
41. Mullis, I. V. S., Martin, M. O., Rudock, G. J., O'Sullivan, C. Y., Arora, A., &, Erberber, E., (2005). TIMSS 2007 Assessment Framewroks. Chesnut Hill, MA: IEA, International Association for the Evaluation of Educational Achievements.
42. Organisation for Economic Co-Operation and Development. (2006). Assessing scientific, reading and mathematical literacy. A framework for PISA 2006. Paris, France: OECD.
43. Polikoff, M. S., (2010). Instructional sensitivity as a psychometric property of assessments. Educational Measurement: Issues and Practices, 29 (4), 3-14.
44. Popham, W. J., (2006). Diagnostic assessment a measurement mirage ? Educational Leadership, 64 (2), 90-91.
45. Popham, W. J., (2007a). Conducting instructional-sensitivity reviews of educational accountability tests. Unpublished paper. Los Angeles, CA: University of California, Los Angeles.
46. Popham, W. J., (2007b). Instructional sensitivity of tests: Accountability's dire drawback. Phi Delta Kappan, 89 (2), 146-150, 155.
47. Popham, W. J., &, Ryan, J. M., (2012). Determining a high-stake test's instructional sensitivity. Paper presented in the Annual Meeting of the National Council on Measurement in Education: Vancouver, British Columbia, Canada.
48. Porter, A. C., Smithson, J., Blank, R., &, Zeider, T., (2007). Alignment as a teacher variable. Applied Measurement in Education, 20 (1), 27-51.
49. Ruiz-Primo, M. A., (1997). Toward a framework of subject-matter achievement assessment. Unpublished manuscript. Stanford, CA: Stanford University.
50. Ruiz-Primo, M. A., (1998, September). Models for measuring science achievement. Invited talk, National CRESST Conference, University of California, Los Angeles.
51. Ruiz-Primo, M. A., (2002, February). On a seamless assessment system. Paper presented at the Seamless Science Education Symposium. Paper presented at the annual meeting of the American Association of the Advancement of Science, Boston, MA.
52. Ruiz-Primo, M. A., &, Li, M., (2008). Building a methodology for developing and evaluating instructionally sensitive assessments. Proposal submitted to National Science Foundation. Award ID: DRL-0816123. Washington, DC: National Science Foundation.
53. Ruiz-Primo, M. A., Li, M., Tasker, T., O'Brian, J., &, Sands, D., (2009). Developing and Evaluating Instructionally Sensitive Assessments (DEISA): Mapping the Intended Curriculum. Unpublished manuscript. Laboratory of Educational Assessment, Research and InnovatioN (LEARN). Denver, CO: School of Education and Human Resources, University of Colorado Denver.
54. Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., &, Klein, S., (2002). On the evaluation of systemic education reform: Searching for instructional sensitivity. Journal of Research in Science Teaching, 39 (5), 369-393.
55. Schmidt, W. H., Porter, A. C., Schwille, J. R., Floden, R. E., &, Freeman, D. J., (1983). Validity as a variable: Can the same certification test be valid for all students? In, G. F. Madaus, (Ed.), The courts, validity, and minimum competency testing (pp. 133-151). Boston, MA: Kluwer-Nijhoff Publishing.
56. Shavelson, R. J., &, Ruiz-Primo, M. A., (1999). On the psychometrics of assessing science understanding. In, J. Mintzes, J. H. Wandersee, &, J. D. Novak, (Eds.), Assessing science understanding (pp. 304-341). San Diego: Academic Press.
57. Toutkoushian, R. K., &, Curtis, T., (2005). Effects of socioeconomic factors on public high school outcomes and rankings. The Journal of Educational Research, 98 (5), 259-270.
58. Webb, N. L., (2005). "Webb Alignment Tool" Wisconsin Center of Educational Research. University of Wisconsin-Madison. Retrieved April, 20, 2012
59. Wiliam, D., (2007). Sensitivity to instruction: The missing ingredient in large-scale assessment systems ? Paper presented at the Annual Meeting of the International Association for Educational Assessment. Baku, Azerbaijan.
60. Wills, K., Li, M., Ruiz-Primo, M. A., Giamellaro, M., Mason, H., (2012). Comparing two experimental designs to evaluate item sensitivity. Paper presented at the American Educational Research Association annual meeting, Vancouver, B.C.
61. Yoon, B., &, Resnick, L. B., (1998). Instructional validity, opportunity to learn and equity: New standards examinations for California Mathematics Renaissance. CSE Technical Report 484. National Center for Research on Evaluation, Standards, and Student Testing (CRESST), University of California, Los Angeles.