An Analytic Framework for Evaluating the Validity of Concept Inventory Claims

Journal of Engineering Education

Volumn 104, Issue 4, 2015, Pages 454-496

  Jorion, Natalie ^a   Gane, Brian D ^a   James, Katie ^a   Schroeder, Lianne ^a   Dibello, Louis V ^a   Pellegrino, James W ^a  

^a UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

Author keywords

Assessment; Concept inventories; Validity

Indexed keywords

ASSESSMENT; ASSESSMENT TOOL; CONCEPT INVENTORIES; CONCEPTUAL UNDERSTANDING; DYNAMICS CONCEPT INVENTORIES; ENGINEERING DISCIPLINES; INVENTORY ANALYSIS; STRUCTURED APPROACH; USE CONCEPT; VALIDITY;

STUDENTS;

EID: 84944737077     PISSN: 10694730     EISSN: 21689830     Source Type: Journal    
DOI: 10.1002/jee.20104     Document Type: Article

References (57)

1. Allen, K. (2006). The Statistics Concept Inventory: Development and analysis of a cognitive assessment instrument in statistics (Doctoral dissertation). SSRN Electronic Journal. doi:10.2139/ssrn.2130143
2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA/APA/NCME). (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3. Baker, F. (2001). The basics of item response theory. ERIC Clearinghouse on Assessment and Evaluation, University of Maryland, College Park, MD. Retrieved from http://echo.edres.org:8080/irt/baker
4. Bartholomew, D. J., Steele, F., & Moustaki, I. (2008). Analysis of multivariate social science data. Boca Raton, FL: CRC Press.
5. Battisti, B. T., Hanegan, N., Sudweeks, R., & Cates, R. (2010). Using item response theory to conduct a distracter analysis on conceptual inventory of natural selection. International Journal of Science and Mathematics Education, 8, 845-868. doi:10.1007/s10763-009-9189-4
6. Bond, T. G., & Fox, C. M. (2006). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
7. Bonett, D. G., & Price, R. M. (2005). Inferential methods for the tetrachoric correlation coefficient. Journal of Educational and Behavioral Statistics, 30, 213-225. doi: 10.3102/10769986030002213
8. Briggs, D. C., Alonzo, A. C., Schwab, C., & Wilson, M. (2006). Diagnostic assessment with ordered multiple-choice items. Educational Assessment, 11, 33-63. doi: 10.1207/s15326977ea1101_2
9. Brophy, S., Klein, S., Portsmore, M., & Rogers, C. (2008). Advancing engineering education in P-12 classrooms. Journal of Engineering Education, 97, 369-387. doi: 10.1002/j.2168-9830.2008.tb00985.x
10. Crocker, L., & Algina, J. (2006). Introduction to classical and modern test theory. New York, NY: Wadsworth.
11. DiBello, L. V., Henson, R. A., & Stout, W. F. (2015). A family of generalized diagnostic classification models for multiple choice option-based scoring. Applied Psychological Measurement, 39, 62-79. doi: 10.1177/0146621614561315
12. DiBello, L. V., Roussos, L., & Stout, W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. V. Rao & S. Sinharay (Eds.), Handbook of statistics, Vol. 26 (pp. 979-1030). Amsterdam, the Netherlands: Elsevier. doi: 10.1016/s0169-7161(06)26031-0
13. DiBello, L. V., & Stout, W. (2003). Student profile scoring for formative assessment. In H. Yanai, A. Okada, K. Shigemasu, Y. Kano, & J. Meluma (Eds.), New developments in psychometrics (pp. 369-380). New York, NY: Springer.
14. Douglas, K. A., & Purzer, Ş. (2015). Validity: Meaning and relevancy in assessment for engineering education research. Journal of Engineering Education, 104(2), 108-118. doi: 10.1002/jee.20070
15. Gane, B. D., Denick, D., Jorion, N., DiBello, L. V., Streveler, R. A., & Pellegrino, J. W. (In preparation). Validating the diagnostic capacity of a STEM concept inventory.
16. Gray, G. L., Costanzo, F., Evans, D., Cornwell, P., Self, B., & Lane, J. L. (2005). The Dynamics Concept Inventory Assessment Test: A progress report and some results. In Proceedings of the 2005 ASEE Annual Conference and Exposition. Retrieved from https://peer.asee.org/the-dynamics-concept-inventory-assessment-test-a-progress-report-and-some-results
17. Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66, 64-74. doi: 10.1119/1.18809
18. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory: Measurement methods for the social science. Newbury Park, CA: Sage.
19. Haertel, E. H., & Lorié, W. A. (2004). Validating standards-based test score interpretations. Measurement, 2(2), 61-103. doi: 10.1207/s15366359mea0202_1
20. Herman, G. L., Zilles, C., & Loui, M. C. (2014). A psychometric evaluation of the digital logic concept inventory. Computer Science Education, 24, 277-303. doi: 10.1080/08993408.2014.970781
21. Herrmann-Abell, C. F., Flanagan, J. C., & Roseman, J. E. (2012). Results from a pilot study of a curriculum unit designed to help middle school students understand chemical reactions in living systems. Paper presented at the annual international conference of the National Association for Research in Science Teaching, Indianapolis, IN.
22. Herrmann-Abell, C. F., Flanagan, J. C., & Roseman, J. E. (2013). Developing and evaluating an eighth grade curriculum unit that links foundational chemistry to biological growth: Using student measures to evaluate the promise of the intervention. Paper presented at the annual international conference of the National Association for Research in Science Teaching, Rio Grande, Puerto Rico.
23. Hestenes, D., & Halloun, I. (1995). Interpreting the Force Concept Inventory. The Physics Teacher, 33, 502-506. doi: 10.1119/1.2344278
24. Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force Concept Inventory. The Physics Teacher, 30, 141-158. doi: 10.1119/1.2343497
25. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185. doi: 10.1007/bf02289447
26. Huffman, D., & Heller, P. (1995). What does the Force Concept Inventory actually measure? Physics Teacher, 33, 138-143. doi: 10.1119/1.2344171
27. Jorion, N., James, K., Schroeder, L., DiBello, L. V., & Pellegrino, J. W. (2013). Statistical and diagnostic analyses of student performance on concept inventories. In J. Pellegrino (Chair), Evaluating the validity of concept inventories as aids for STEM teaching and learning. Symposium conducted at the meeting of the American Educational Research Association, San Francisco, CA.
28. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272. doi: 10.1177/01466210122032064
29. Kane, M. T. (1992). An argument-based approach to validation. Psychological Bulletin, 112, 527-535. doi: 10.1037/0033-2909.112.3.527
30. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: American Council on Education and Praeger.
31. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1-73. doi: 10.1111/jedm.12000
32. Ledesma, R. D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out parallel analysis. Practical Assessment, Research & Evaluation, 12(2), 1-11.
33. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York, NY: Macmillan.
34. Miller, R. L., Streveler, R. A., Olds, B. M., & Nelson, M. A. (2003). What conceptual models do engineering students use to describe momentum transfer and heat conduction? Paper presented at the American Educational Research Association annual meeting, Chicago, IL.
35. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-67. doi: 10.1207/s15366359mea0101_02
36. Muraki, E., & Bock, R. D. (2003). PARSCALE 4: IRT item analysis and test scoring for rating-scale data (Computer program). Chicago, IL: Scientific Software.
37. National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Committee on a Conceptual Framework for New K-12 Science Education Standards, Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academies Press.
38. Oosterhof, A. (1994). Classroom applications of educational measurement (2nd ed.). New York, NY: Macmillan.
39. Pellegrino, J. W., Chudowsky, N, & Glaser, R. (Eds). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academies Press.
40. Pellegrino, J. W., DiBello, L. V., & Brophy, S. (2014). The science and design of assessment in engineering education. In A. Johri & B. Olds (Eds.), Cambridge handbook of engineering education research (pp. 571-598). New York, NY: Cambridge University Press. doi: 10.1017/cbo9781139013451.036
41. Reed-Rhoads, T., & Imbrie, P. K. (2008). Concept inventories in engineering education. Paper presented at the National Research Council's Workshop on Linking Evidence to Promising Practices in STEM Undergraduate Education, Washington, DC. Retrieved from http://sites.nationalacademies.org/dbasse/bose/dbasse_080106
42. Rencher, A. C. (2002). Methods of multivariate analysis (2nd ed.). New York, NY: Wiley-Interscience.
43. Roussos, L. A., DiBello, L. V., Stout, W. F., Hartz, S. M., Henson, R. A., & Templin, J. H. (2007). The fusion model skills diagnostic system. In J. Leighton & M. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications. New York, NY: Cambridge University Press.
44. Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford.
45. Sadler, P. M. (1998). Psychometric models of student conceptions in science: Reconciling qualitative studies and distractor-driven assessment instruments. Journal of Research in Science Teaching, 35(3), 265-296. doi: 10.1002/(sici)1098-2736(199803)35:3<265::aid-tea3>3.0.co;2-p
46. Santiago-Román, A. I. (2009). Fitting cognitive diagnostic assessment to the Concept Assessment Tool for Statics (CATS) (Unpublished doctoral dissertation). Purdue University, West Lafayette, Indiana.
47. Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. Journal of Educational Research, 99, 323-338. doi: 10.3200/joer.99.6.323-338
48. Singer, S. R., Nielsen, N. R., & Schweingruber, H. A. (Eds.). (2012). Discipline-based education research: Understanding and improving learning in undergraduate science and engineering. Committee on the Status, Contributions, and Future Directions of Discipline-Based Education Research. Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. doi: 10.1002/jee.20030
49. Steif, P. S., & Dantzler, J. A. (2005). A statics concept inventory: Development and psychometric analysis. Journal of Engineering Education, 94, 363-371. doi: 10.1002/j.2168-9830.2005.tb00864.x
50. Steif, P. S., & Hansen, M. A. (2006). Comparisons between performances in a statics concept inventory and course examinations. International Journal of Engineering Education, 22, 1070-1076.
51. Steif, P. S., & Hansen, M. A. (2007). New practices for administering and analyzing the results of concept inventories. Journal of Engineering Education, 96, 205-212. doi: 10.1002/j.2168-9830.2007.tb00930.x
52. Stone, A., Allen, K., Rhoads, T. R., Murphy, T. J., Shehab, R. L., & Saha, C. (2003). The Statistics Concept Inventory: A pilot study. In Proceedings of the 33rd ASEE/IEEE Frontiers in Education Conference (Vol. 1, pp. T3D-1-T3D-6). doi: 10.1109/fie.2003.1263336
53. Streveler, R. A., Olds, B. M., Miller, R. L., & Nelson, M. A. (2003). Using a delphi study to identify the most difficult concepts for students to master in thermal and transport science. In Proceedings of the 2003 ASEE Annual Conference and Exposition. Retrieved from https://peer.asee.org/using-a-delphi-study-to-identify-the-most-difficult-concepts-for-students-to-master-in-thermal-and-transport-sciences
54. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). New York, NY: Pearson.
55. Thissen, D. (2003). MULTILOG 7: Multiple categorical item analysis and test scoring using item response theory (Computer program). Chicago, IL: Scientific Software.
56. Villafañe, S. M., Loertscher, J., Minderhout, V., & Lewis, J. E. (2011). Uncovering students' incorrect ideas about foundational concepts for biochemistry. Chemistry Education Research and Practice, 12, 210-218. doi: 10.1039/c1rp90026a
57. Zimowski, M. F., Muraki, E., Mislevy, R. J., & Bock, R. D. (2003). BILOG-MG 3 (Computer program). Chicago, IL: Scientific Software.