A psychometric evaluation of the digital logic concept inventory

Computer Science Education

Volumn 24, Issue 4, 2014, Pages 277-303

  Herman, Geoffrey L ^a   Zilles, Craig ^b   Loui, Michael C ^c  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b NONE   (United States)

^c University of Illinois at Urbana Champaign   (United States)

Author keywords

concept inventory; digital logic; item response theory; misconceptions; reliability; validity

Indexed keywords

EID: 84908144731     PISSN: 08993408     EISSN: 17445175     Source Type: Journal    
DOI: 10.1080/08993408.2014.970781     Document Type: Article

References (60)

1. Almstrum, V. L., Henderson, P. B., Harvey, V., Marion, C. H. H., Riedesel, C., & Tew, A. E. (2006). Concept inventories in computer science for the topic discrete mathematics. ACM SIGCSE Bulletin, 38, 132–145.
2. Barbera, J. (2013). A psychometric analysis of the chemical concepts inventory. Journal of Chemical Education, 90, 546–553.
3. Bardar, E. M., Prather, E. E., Brecher, K., & Slater, T. F. (2007). Development and validation of the Light and Spectroscopy Concept Inventory. Astronomy, 5, 103–113.
4. Bechger, T. M., Maris, G., Verstralen, H., & Beguin, A. A. (2003). Using classical test theory in combination with item response theory. Applied Psychological Measurement, 27, 319–334.
5. Bock, R., & Lieberman, M. (1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35, 179–197.
6. Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. New York, NY: Cambridge University Press.
7. Chi, M. T. H. (2006). Methods to assess the representations of experts’ and novices’ knowledge. In K. Ericsson, N. Charness, P. Feltovich, & R. Hoffman (Eds.), Cambridge handbook of expertise and expert performance (pp. 167–184). Cambridge: Cambridge University Press.
8. Chi, M. T. H., & Slotta, J. D. (1993). The ontological coherence of intuitive physics. Commentary on A. diSessas “Toward an epistemology of physics”. Cognition and Instruction, 10, 249–260.
9. Clayton, M. J. (1997). Delphi: A technique to harness expert opinion for critical decision-making tasks in education. Educational Psychology, 17, 373–386.
10. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Harcourt Brace Jovanovitch.
11. DiBello, L. V., Pellegrino, J., Miller, R., Streveler, R., Jorion, N., James, K.,.. Stout, W. (2013). An analytical framework for investigating cis. In Proceedings of the 2013 Annual Meeting of the American Educational Research Association, San Francisco, CA.
12. Ding, L., & Beichner, R. (2009). Approaches to data analysis of multiple-choice questions. Physics Re, 5, 020103.
13. Ding, L., Chabay, R., Sherwood, B., & Beichner, R. (2006). Evaluating an electricity and magnetism assessment tool: Brief Electricity and Magnetism Assessment. Physics Review Special Topics: Physics Education Research, 2, 010105.
14. diSessa, A., Gillespie, N., & Esterly, J. (2004). Coherence versus fragmentation in the development of the concept of force. Cognitive Science, 28, 843–900.
15. Evans, D. L., Gray, G. L., Krause, S., Martin, J., Midkiff, C., Notaros, B. M.,.. Wage, K. (2003). Progress on concept inventory assessment tools. In Proceedings of the 33rd ASEE/IEEE Frontiers in Education Conference (pp. T4G-1–T4G-8), Boulder, CO.
16. George, D., & Mallery, P. (2009). SPSS for windows step by step: A simple guide and reference. Boston, MA: Pearson Education.
17. Goldman, K., Gross, P., Heeren, C., Herman, G. L., Kaczmarczyk, L., Loui, M. C., & Zilles, C. (2010). Setting the scope of concept inventories for introductory computing subjects. ACM Transactions on Computing Education, 10, 5–29.
18. Hake, R. (1998). Interactive-engagement vs. traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66, 64–74.
19. Hake, R. (2002). Lessons from the physics education reform effort. Conservation Ecology, 5, 28.
20. Hambleton, R. K., & Jones, R. J. (1993). Comparison of classical test theory and item response theory and their applications to test deveolpment. Educational Measurement: Issues & Practice, 12, 253–262.
21. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Thousand Oaks, CA: Sage.
22. Herman, G. L. & Loui, M. (2011, June 26–29). Administering the digital logic concept inventory at multiple institutions. In Proceedings of the 2011 American Society for Engineering Education Annual Conference and Exposition (pp. AC2011-1800), Vancouver, Canada.
23. Herman, G. L. & Loui, M. C. (2012, June 10–13). Identifying the core conceptual frame- work of digital logic. In Proceedings of the 2012 American Society for Engineering Education Annual Conference and Exposition (pp. AC2012-4637), San Antonio, TX.
24. Herman, G. L., Loui, M. C., Kaczmarczyk, L., & Zilles, C. (2012). Discovering the what and why of students’ difficulties in Boolean logic. ACM Transactions on Computing Education, 12, 3:1–28.
25. Herman, G. L., Loui, M. C. & Zilles, C. (2010, March 10–13). Creating the digital logic concept inventory. In Proceedings of the forty-first ACM technical symposium on computer science education (pp. 102–106), Milwaukee, WI.
26. Herman, G. L., Loui, M. C., & Zilles, C. (2011a). Students’ misconceptions about medium-scale integrated circuits. IEEE Transactions on Education, 54, 637–645.
27. Herman, G. L., Zilles, C., & Loui, M. C. (2011b). How do students misunderstand number representations? Computer Science Education, 23, 289–312.
28. Herman, G. L., Zilles, C., & Loui, M. C. (2012). Flipops in students’ conceptions of state. IEEE Transactions on Education, 55, 88–98.
29. Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30, 141–166.
30. Jorion, N., Gane, B. D., James, K., Schroeder, L., DiBello, L. V., & Pellegrino, J. W. (2014a). Conceptual and analytical frameworks for examining validity and utility of concept inventories. In Proceedings of the 2014 Annual Meeting of the American Educational Research Association, Philadelphia, PA.
31. Jorion, N., Gane, B. D., James, K., Schroeder, L., DiBello, L. V., & Pellegrino, J. W. (2014b). Quantitative analyses of student performance on concept inventory. In Proceedings of the 2014 Annual Meeting of the American Educational Research Association, Philadelphia, PA.
32. Jorion, N., James, K., Schroeder, L. & DiBello, L. V. (2013). Statistical and diagnostic analyses of student performance on concept inventories. In Proceedings of the 2014 Annual Meeting of the American Educational Research Association, San Francisco, CA.
33. Kline, T. J. B. (2005). Psychological testing: A practical approach to design and evaluation. Thousand Oaks, CA: Sage.
34. Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151–160.
35. Leech, N., & Onwuegbuzie, A. (2007). An array of qualitative data analysis tools: A call for data analysis triangulation. School Phsychology Quarterly, 22, 557–584.
36. Libarkin, J. C., & Anderson, S. W. (2006). The geoscience concept inventory: Application of Rasch analysis to concept inventory development in higher education. In X. Liu & W. Boone (Eds.), Applications of Rasch measurement in science education (pp. 45–73). Maple Grove, MN: JAM Publishers.
37. Litzinger, T., Vanmeter, P., Firetto, C., Passmore, L., Masters, C., Turns, S.,.. Zappe, S. (2010). A cognitive study of problem solving in statics. Journal of Engineering Education, 99, 337–353.
38. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
39. Manktelow, K. (2000). Reasoning and thinking. Hove: Psychology Press.
40. Mestre, J. P., Dufresne, R. J., Gerace, W. J., Hardiman, P. T., & Touger, J. S. (1993). Promoting skilled problem solving behavior among beginning physics students. Journal of Research in Science Teaching, 30, 303–317.
41. Montfort, D., Herman, G. L., Streveler, R. & Brown, S. (2012, October 3–6). Assessing the application of three theories of conceptual change to interdisciplinary data sets. In Proceedings of the Forty-Second ASEE/IEEE Frontiers in Education Conference (p. S1B-1–S1B-6), Seattle, WA.
42. Montfort, D. B., Herman, G. L., Brown, S. A., Matusovich, H. M. & Streveler, R. A. (in press). Trans-disciplinary patterns in student conceptual understanding of engineering.
43. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). Columbus, OH: McGraw-Hill.
44. Pellegrino, J., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
45. Pellegrino, J. W., DiBello, L. V., & Brophy, S. P. (2014). 29 the science and design of assessment in engineering education. In A. Johri & B. M. Olds (Eds.), Cambridge handbook on engineering education research (pp. 571–600). Cambridge: Cambridge University Press.
46. Pellegrino, J. W., DiBello, L. V., James, K., Jorion, N., & Schroeder, L. (2011). Concept inventories as aids for instruction: Example applications of a validity framework. In Proceedings of the Research in Engineering Education Symposium, Madrid.
47. Pellegrino, J. W., DiBello, L. V., Miller, R. L., & Streveler, R. A. (2013). Components of a comprehensive approach to validity. In Proceedings of the 2013 Annual Meeting of the American Educational Research Association, San Francisco, CA.
48. Pill, J. (1971). The Delphi method: Substance, context, a critique and an annotated bibliography. Socio-Economic Planning Sciences, 5, 57–71.
49. Porter, L., Garcia, S., Tseng, H. W. & Zingaro, D. (2013). Evaluating student understand- ing of core concepts in computer architecture. In Proceedings of the 18th Annual Conference on Innovation and Technology in Computer Science Education. Canterbury.
50. Reiner, M., Slotta, J. D., Chi, M. T. H., & Resnick, L. B. (2000). Naive physics reasoning: A commitment to substance-based conceptions. Cognition and Instruction, 18, 1–34.
51. Rizopoulos, D. (2006). Itm: An R package for latent variable modeling and item response theory analysis. Journal of Statistical Software, 17, 1–25.
52. Steif, P. S., & Dantzler, J. A. (2005). A statics concept inventory: Development and psychometric analysis. Journal of Engineering Education, 33, 363–371.
53. Stone, A. (2006). A psychometric analysis of the statistics concept inventory (Unpublished doctoral dissertation). University of Oklahoma, Norman, OK.
54. Streveler, R. A., Miller, R. L., Santiago-Roman, A. I., Nelson, M. A., Geist, M. R., & Olds, B. M. (2011). Rigorous method for concept inventory development: Using the “assessment triangle” to develop and test the thermal and transport science concept inventory (TTCI). International Journal of Engineering Education, 27, 968–984.
55. Taylor, C., Zingaro, D., Porter, L., Webb, K. C., Lee, C. B., & Clancy, M. (in press). Computer science concept inventories: Past and future. Computer Science Education.
56. Tew, A. E. (2010). Assessing fundamental introductory computing concept knowledge in a language independent manner (Unpublished doctoral dissertation). Georgia Institute of Technology, Atlanta, GA.
57. Vosniadou, S., & Brewer, W. F. (1992). Mental models of the earth: A study of conceptual change in childhood. Cognitive Psychology, 24, 535–585.
58. Wallace, C. S., & Bailey, J. M. (2010). Do concept inventories actually measure anything? Astronomy Education Review, 9, 010116-1.
59. Webb, K., & Taylor, C. (2014). Developing a pre- and post-course concept inventory to gauge operating systems learning. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education, Atlanta, GA.
60. Wren, D., & Barbera, J. (2014). Psychometric analysis of the thermochemistry concept inventory. Chemistry & Biodiversity Education Research and Practice, 15, 380–390.