Rigorous methodology for concept inventory development: Using the 'assessment triangle' to develop and test the thermal and transport science concept inventory (TTCI)

International Journal of Engineering Education

Volumn 27, Issue 5, 2011, Pages 968-984

  Streveler, Ruth A ^a   Miller, Ronald L ^b   Santiago Román, Aidsa I ^c   Nelson, Mary A ^d   Geist, Monica R ^e   Olds, Barbara M ^f  

^a PURDUE UNIVERSITY   (United States)

^b Department of Geology and Geological Engineering   (United States)

^c UNIVERSITY OF PUERTO RICO   (Puerto Rico)

^d UNIVERSITY OF COLORADO   (United States)

^e Front Range Community College   (United States)

^f NATIONAL SCIENCE FOUNDATION   (United States)

Author keywords

Assessment triangle; Concept inventory; Misconceptions

Indexed keywords

EID: 80052993637     PISSN: 0949149X     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (54)

1. R. Duit, Students' and Teachers' Conceptions and Science Education, Kiel, Germany: Institute for Science Education, http://www.ipn.uni-kiel.de/aktuell/ stcse/, accessed May 2009.
2. I. A. Halloun and D. Hestenes, The initial knowledge state of college physics students, American Journal of Physics, 53(11), 1985, pp. 1043-1055.
3. D. Hestenes, M. Wells and G. Swackhamer, Force concept inventory, The Physics Teacher, 30(3), 1992, pp. 159-166.
4. I. A. Halloun and D. Hestenes, Common sense concepts about motion, American Journal of Physics, 53(11), 1985, pp. 1056-1065.
5. R. R. Hake, Interactive-engagement versus traditional methods: a six-thousand-student survey of mechanics test data for introductory physics courses, American Journal of Physics, 66(1), 1998, pp. 64-74.
6. E. Mazur, Qualitative vs. quantitative thinking: are we teaching the right thing?, Optics and Photonics News, 3, 1992, pp. 38.
7. D. L. Evans, G. L. Gray, S. Krause, J. Martin, C. Midkiff and B. M. Notaros, Progress on concept inventory assessment tools, Proceedings of the 33rd Annual Frontiers in Education (electronic), Boulder, CO, 2003.
8. T. R. Rhoads and R. J. Roedel, The wave concept inventory-a cognitive instrument based on Bloom's taxonomy, Proceedings of the 29th Annual Frontiers in Education (electronic), San Juan, PR, 1999.
9. J. K. Martin, J. Mitchell and T. Newell, Work in progress: analysis of reliability of the fluid mechanics concept inventory, Proceedings of the 34th Annual Frontiers in Education (electronic), Savannah, GA, 2004.
10. A. Jacobi, J. Martin, J. Mitchell and T. Newell, A concept inventory for heat transfer, Proceedings of the 33rd Annual Frontiers in Education (electronic), Boulder, CO, 2003.
11. S. Krause, J. C Decker and R. Griffin, Using a materials concept inventory to assess conceptual gain in introductory materials engineering courses, Proceedings of the 33rd Annual Frontiers in Education (electronic), Boulder, CO, 2003.
12. K. E. Wage, J. R. Buck, C. H. G. Wright and T. B. Welch, The signals and systems concept inventory, IEEE Transactions on Education, 48(3), 2005, pp. 448-461. (Pubitemid 41430913)
13. P. S. Steif and J. A. Dantzler, A statics concept inventory: development and psychometric analysis, Journal of Engineering Education, 94(4), 2005, pp. 363-371. (Pubitemid 41606903)
14. A. D. Stone,Apsychometric analysis of the statistics concept inventory, Ph.D. dissertation, University of Oklahoma, Norman, OK, 2006.
15. J. Richardson, P. Steif, J. Morgan and J. Dantzler, Development of a concept inventory for strength of materials, Proceedings of the 33rd Annual Frontiers in Education (electronic), Boulder, CO, 2003.
16. K. C. Midkiff, T. A. Litzinger and D. L. Evans, Development of engineering thermodynamics concept inventory instruments, Proceedings of the 31st Annual Frontiers in Education Conference (electronic), Reno, NV, 2001.
17. A. I. Santiago Román, Fitting cognitive diagnostic assessment to the content assessment tool for statics, Ph.D. dissertation, Purdue University, West Lafayette, IN, 2009.
18. R. A. Streveler, T. A. Litzinger, R. L. Miller and P. S. Steif, Learning conceptual knowledge in engineering: overview and future research directions, Journal of Engineering Education, 97(3), 2008, pp. 279-294.
19. J. Pellegrino, N. Chudowsky and R. Glaser, Knowing What Students Know: The Science and Design of Educational Assessment, National Academy Press, Washington, DC, 2001.
20. M. Adler and E. Ziglio, Gazing Into the Oracle: The Delphi Method and its Application to Social Policy and Public Health, Jessica Kingsley Publishers, London, 1996.
21. R. A. Streveler, B. M. Olds, R. L. Miller and M. A. Nelson, Using a Delphi study to identify the most difficult concepts for students to master in thermal and transport science, Proceedings of the 110th Annual Conference of the American Society for Engineering Education (electronic), Nashville, TN, June 2003.
22. M. J. Clayton, Delphi:Atechnique to harness expert opinion for critical decision-making tasks in education, Educational Psychology, 17, 1997, pp. 373-386. (Pubitemid 127155424)
23. H. A. Linstone and M. Turoff, The Delphi Method: Techniques and Applications, Addison-Wesley Publishing Company, Reading, MA, 1996.
24. L. S. Fish and D. M. Busby, The Delphi method. In D. H. Sprenkle and S. M. Moon (eds) Research Methods in Family Therapy, Guilford Press, New York, 1996, pp. 469-482.
25. M. A. Nelson, M. R. Geist, R. A. Streveler, R.L. Miller, B. M. Olds, C. S. Ammerman and R.F. Ammerman, From practice to research: using professional expertise to inform research about engineering students' conceptual understanding, paper presented at the Annual Conference of the American Educational Research Association, Montreal, Quebec, Canada, April 2005.
26. B. M. Olds, R. A. Streveler, R. L. Miller and M. A. Nelson, Preliminary results from the development of a concept inventory in thermal and transport science, Proceedings of the 111th American Society for Engineering Education Annual Conference and Exposition (electronic), Salt Lake City, UT, June, 2004.
27. K. Carlton, Teaching about heat and temperature, Physics Education, 35(2), 2000, pp. 101-105.
28. P. L. Thomas and R.W.Schwenz, College physical chemistry students' conceptions of equilibrium and fundamental thermodynamics, Journal of Research in Science Teaching, 35(10), 1998, pp. 1151-1160. (Pubitemid 128459006)
29. M. F. Thomaz, I. M. Malaquias, M. C. Vanente and M. J. Antunes, An attempt to overcome alternative conceptions related to heat and temperature, Physics Education, 30, 1995, pp. 19-26.
30. G. J. Posner, K. A Strike, P. W. Hewson and W. A. Gertzog, Accommodation of a scientific conception: toward a theory of conceptual change, Science Education, 66(2), 1982, pp. 211-227.
31. R.L. Miller, R.A. Streveler, B. M. Olds, M. T. H. Chi, M. A. Nelson and M. R. Geist, Misconceptions about rate processes: preliminary evidence for the importance of emergent schemas in thermal and transport sciences, Proceedings of the 113th American Society for Engineering Education Annual Conference and Exposition (electronic), Chicago, IL, 2006.
32. S. Vosniadou, Conceptual change research: an introduction. In International Handbook of Research on Conceptual Change, Routledge, New York and London, 2008.
33. R. A. Streveler, R. L. Miller and B. M. Olds, Threshold concepts and troublesome knowledge in engineering: evidence from misconceptions research, Proceedings of the Threshold Concepts Conference (electronic), Kingston, Ontario, Canada, June 2008.
34. R. A. Streveler, R. L. Miller and J. D. Slotta, Using ontology training to investigate why some engineering science concepts are so difficult to learn, Proceedings of on Being an Engineer: Cognitive Underpinnings of Engineering Education Conference (electronic), Lubbock, TX, February 2008.
35. A. A. diSessa, Knowledge in pieces. In G. Forman and P. Pufall (eds.) Constructivism in the Computer Age, Erlbaum, Hillsdale, NJ, 1988, pp. 49-70.
36. M. T. H. Chi, Commonsense Conceptions of emergent processes: why some misconceptions are robust. Journal of the Learning Sciences, 14(2), 2005, pp. 161-199. (Pubitemid 40551444)
37. M. A. Nelson, M. R. Geist, R. L. Miller, R. A. Streveler and B. M. Olds, How to create a concept inventory: the thermal and transport concept inventory, paper presented at the Annual Conference of the American Educational Research Association, Chicago, IL, 2007.
38. R. L. Miller, R. A. Streveler, B. M. Olds, M. A. Nelson and M. R. Geist, Concept inventories meet cognitive psychology: using beta testing as a mechanism for identifying engineering student misconceptions, Proceedings of the American Society for Engineering Education Annual Conference (electronic), Portland, OR, June, 2005.
39. S. M. Downing, Twelve Steps for Effective Test Development, in Handbook of Test Development, edited by S.M. Downing and T. M. Haladyna, Erlbaum, Mahwah, NJ, 2006, pp. 3-26.
40. D. R. Mulford and W. R. Robinson, An inventory for alternate conceptions among first-semester general chemistry students, Journal of Chemical Education, 79(6), 2002, pp. 739-744.
41. M. Reiner, J. D. Slotta, M. T. H. Chi and L. B. Resnick, Naive physics reasoning: a commitment to substance-based conceptions, Cognition and Instruction, 18(1), 2000, pp. 1-35.
42. R. J. Mislevy, M. R. Wilson, K. Ercikan and N. Chudowsky, Psychometric Principles in Student Assessment, Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies, University of California, Los Angeles, 2002.
43. T. M. Bechger, G. Maris, H. Verstralen and A. A. Beguin, Using classical test theory in combination with item response theory, Applied Psychological Measurement, 27(5), 2003, pp. 319.
44. X. Fan, Item Response theory and classical test theory: an empirical comparison of their item/person statistics, Educational and Psychological Measurement, 58(3), 1998, pp. 357.
45. G. F. Kuder and M. W. Richardson, The theory of the estimation of test reliability, Psychometrika, 2(3), 1937, pp. 151-160.
46. T. J. B. Kline, Psychological Testing:APractical Approach to Design and evaluation, Sage, Thousand Oaks, CA, 2005.
47. C. Stage, Classical Test Theory or Item Response Theory: The Swedish Experience, Umeå University, Department of Educational Measurement, 2003.
48. J. L. Ndalichako and W. T. Rogers, Comparison of finite state score theory, classical test theory, and item response theory in scoring multiple-choice items, Educational and Psychological Measurement, 57(4), 1997, pp. 580.
49. F. B. Baker, Item Response Theory: Parameter Estimation Techniques, CRC Press, Cleveland, OH, 2004.
50. R. M. Thorndike, IRT and intelligence testing: past, present, and future. In S. E. Embretson and S. L. Hershberger (eds.) The New Rules of Measurement: What Every Psychologist and Educator Should Know, Erlbaum, Mahwah, NJ, 1999.
51. J. Minstrell, R. Duit, F. Goldberg and H. Niedderer, Facets of students' knowledge and relevant instruction. In Proceedings, International Workshop-Research in Physics Learning: Theoretical Issues and Empirical Studies (electronic), The Institute for Science Education at the University of Kiel, Kiel, Germany, 1992
52. J. P. Leighton and M. J. Gierl, Why cognitive diagnostic assessment? In Cognitive Diagnostic Assessment for Education: Theory and Applications, Cambridge University Press, Cambridge UK, 2007, pp. 3-18.
53. D. Yang, R. A. Streveler, R. L. Miller and A. I. Santiago Román, Repairing misconceptions: a pilot study with advanced engineering students on their use of schema training modules, Proceedings of the 116th ASEE Annual Conference and Exposition (electronic), Austin, TX, 2009.
54. National Academy of Engineering, Grand challenges for engineering, http://www.grandchallenges.org, accessed July 2009.