Evaluation of diagnostic tools that tertiary teachers can apply to profile their students’ conceptions

International Journal of Science Education

Volumn 39, Issue 5, 2017, Pages 565-586

  Schultz, Madeleine ^a   Lawrie, Gwendolyn A ^b   Bailey, Chantal H ^b   Bedford, Simon B ^c   Dargaville, Tim R ^a   O'Brien, Glennys ^c   Tasker, Roy ^d   Thompson, Christopher D ^e   Williams, Mark ^f   Wright, Anthony H ^b  

^a QUEENSLAND UNIVERSITY OF TECHNOLOGY   (Australia)

^b UNIVERSITY OF QUEENSLAND   (Australia)

^c UNIVERSITY OF WOLLONGONG   (Australia)

^d PURDUE UNIVERSITY   (United States)

^e MONASH UNIVERSITY   (Australia)

^f UNIVERSITY OF WESTERN SYDNEY   (Australia)

Author keywords

Concept diagnostic instrument; general chemistry; ordered multiple choice; psychometric analysis

Indexed keywords

EID: 85015720658     PISSN: 09500693     EISSN: 14645289     Source Type: Journal    
DOI: 10.1080/09500693.2017.1296980     Document Type: Article

References (90)

1. Adams, W. K., & Wieman, C. E., (2011). Development and validation of instruments to measure learning of expert-like thinking. International Journal of Science Education, 33, 1289–1312. doi:10.1080/09500693.2010.512369
2. Arjoon, J. A., Xu, X., & Lewis, J. E., (2013). Understanding the state of the art for measurement in chemistry education research:Examining the psychometric evidence. Journal of Chemical Education, 90, 536–545. doi:10.1021/ed3002013
3. Ausubel, D. P., (1968). Educational psychology:A cognitive view. New York, NY:Holt, Rinehart & Winston.
4. Barbera, J., (2013). A psychometric analysis of the chemical concepts inventory. Journal of Chemical Education, 90, 546–553. doi:10.1021/ed3004353
5. de Berg, K., (2011). A study of first-year chemistry students’ understanding of solution concentration at the tertiary level. Chemistry Education Research and Practice, 13, 8–16. doi:10.1039/c1rp90056k
6. Bergquist, W., & Heikkinen, H., (1990). Student ideas regarding chemical equilibrium:What written test answers do not reveal. Journal of Chemical Education, 67, 1000–1003. doi:10.1021/ed067p1000
7. Biggs, J., (1993). What do inventories of students’s learning processes really measure? A theoretical review and clarification. British Journal of Educational Psychology, 63, 3–19. doi:10.1111/j.2044-8279.1993.tb01038.x
8. Bodner, G. M., (1991). I have found you an argument:The conceptual knowledge of beginning chemistry graduate students. Journal of Chemical Education, 68, 385–388. doi:10.1021/ed068p385
9. Brandriet, A. R., & Bretz, S. L., (2014). The development of the redox concept inventory as a measure of students’ symbolic and particulate redox understandings and confidence. Journal of Chemical Education, 91, 1132–1144. doi:10.1021/ed500051n
10. Bridle, C. A., & Yezierski, E. J., (2012). Evidence for the effectiveness of inquiry-based, particulate-level instruction on conceptions of the particulate nature of matter. Journal of Chemical Education, 89, 192–198. doi:10.1021/ed100735u
11. Briggs, D. C., Alonzo, A. C., Schwab, C., & Wilson, M., (2006). Diagnostic assessment with ordered multiple-choice items. Educational Assessment, 11, 33–63. doi:10.1207/s15326977ea1101_2
12. Chu, H.-E., Treagust, D. F., Yeo, S., & Zadnik, M., (2012). Evaluation of students’ understanding of thermal concepts in everyday contexts. International Journal of Science Education, 34, 1509–1534. doi:10.1080/09500693.2012.657714
13. Cook, M., Wiebe, E. N., & Carter, G., (2008). The influence of prior knowledge on viewing and interpreting graphics with macroscopic and molecular representations. Science Education, 92, 848–867. doi:10.1002/sce.20262
14. Coştu, B., Ayas, A., & Niaz, M., (2010). Promoting conceptual change in first year students’ understanding of evaporation. Chemistry Education Research and Practice, 11, 5–16. doi:10.1039/C001041N
15. Crisp, V., & Sweiry, E., (2006). Can a picture ruin a thousand words? The effects of visual resources in exam questions. Educational Research, 48, 139–154. doi:10.1080/00131880600732249
16. Devetak, I., Vogrinc, J., & Glažar, S. A., (2009). Assessing 16-year-old students’ understanding of aqueous solution at submicroscopic level. Research in Science Education, 39, 157–179. doi:10.1007/s11165-007-9077-2
17. Dickson, H., Thompson, C. D., & O'Toole, P., (2016). A picture is worth a thousand words:Investigating first year chemistry students’ ability to visually express their understanding of chemistry concepts. International Journal of Innovation in Science and Mathematics Education, 24, 12–23.
18. Driver, R., & Erickson, G., (1983). Theories-in-action:Some theoretical and empirical issues in the study of students’ conceptual frameworks in science. Studies in Science Education, 10, 37–60. doi:10.1080/03057268308559904
19. Duit, R., & Treagust, D. F., (2003). Conceptual change:A powerful framework for improving science teaching and learning. International Journal of Science Education, 25, 671–688. doi:10.1080/09500690305016
20. Duit, R., & Treagust, D. F., (2012). How can conceptual change contribute to theory and practice in science education?. In B. J., Fraser, K. G., Tobin, & C. J., McRobbie (Eds.), Second international handbook of science education (pp. 107–118). Dordrecht:Springer Science+Business Media.
21. Gess-Newsome, J., (2015). A model of teacher professional knowledge and skill including PCK:Results of the thinking from the PCK summit. In A., Berry, P., Friedrichsen, & J., Loughran (Eds.), Re-examining pedagogical content knowledge in science education (pp. 28–42). New York, NY:Routledge.
22. Gliem, J. S., & Gliem, R. R., (2003). Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for likert-type scales. Paper presented at the 2003 Midwest Research to Practice Conference in Adult, Continuing, and Community Education, Columbus, Ohio.
23. Hackling, M. W., & Garnett, P. J., (1985). Misconceptions of chemical equilibrium. European Journal of Science Education, 7, 205–214. doi:10.1080/0140528850070211
24. Hadenfeldt, J. C., Bernholt, S., Liu, X., Neumann, K., & Parchmann, I., (2013). Using ordered multiple-choice items to assess students’ understanding of the structure and composition of matter. Journal of Chemical Education, 90, 1602–1608. doi:10.1021/ed3006192
25. Haladyna, T. M., & Downing, S. M., (1989). Validity of a taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2, 51–78. doi:10.1207/s15324818ame0201_4
26. Haladyna, T. M., Downing, S. M., & Rodriguez, M. C., (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15, 309–333. doi:10.1207/S15324818AME1503_5
27. Hand, B. M., & Treagust, D. F., (1988). Application of a conceptual conflict teaching strategy to enhance student learning of acids and bases. Research in Science Education, 18, 53–63. doi:10.1007/BF02356580
28. Jansoon, N., Coll, R. K., & Somsook, E., (2009). Understanding mental models of dilution in Thai students. International Journal of Environmental & Science Education, 4, 147–168.
29. Johnson, P., (1998). Children’s understanding of changes of state involving the gas state, part 1:Boiling water and the particle theory. International Journal of Science Education, 20, 567–583. doi:10.1080/0950069980200505
30. Johnstone, A. H., (1983). Chemical education research:Facts, findings, consequences. Journal of Chemical Education, 60, 968–971. doi:10.1021/ed060p968
31. Johnstone, A. H., (1991). Why is science difficult to learn? Things are seldom what they seem. Journal of Computer Assisted Learning, 7, 75–83. doi:10.1111/j.1365-2729.1991.tb00230.x
32. Johnstone, A. H., (2010). You can’t get there from here. Journal of Chemical Education, 87, 22–29. doi:10.1021/ed800026d
33. Kahveci, A., (2013). Diagnostic assessment of student understanding of the particulate nature of matter:Decades of research. In G., Tsaparlis, & H., Sevian (Eds.), Concepts of matter in science education (pp. 249–278). Dordrecht:Springer Science+Business Media.
34. Kind, V., (2004). Beyond appearances:Students’ misconceptions about basic chemical ideas (2nd ed.). Royal Society of Chemistry:London.
35. Kirbulut, Z. D., & Beeth, M. E., (2013). Consistency of students’ ideas across evaporation, condensation, and boiling. Research in Science Education, 43, 209–232. doi:10.1007/s11165-011-9264-z
36. Krause, S., Birk, J., Bauer, R., Jenkins, B., & Pavelich, M. J., (2004). Development, testing and application of a chemistry concept inventory. Paper presented at the 34th ASEE/IEEE frontiers in education conference, Savannah, Georgia, USA.
37. Krnel, D., Watson, R., & Glažar, S. A., (2005). The development of the concept of ‘matter’:A cross-age study of how children describe materials. International Journal of Science Education, 27, 367–383. doi:10.1080/09500690412331314441
38. Kruse, R. A., & Roehrig, G. H., (2005). A comparison study:Assessing teachers’ conceptions with the chemistry concepts inventory. Journal of Chemical Education, 82, 1246–1250. doi:10.1021/ed082p1246
39. Lawrie, G. A., Schultz, M., Bailey, C. H., Mamun, A. A., Micallef, A. S., Williams, M., & Wright, A. H., (2016). Development of scaffolded online modules to support self-regulated learning in chemistry concepts. In M., Schultz, T. A., Holme, & S., Schmid (Eds.), Technology and assessment strategies for improving student learning in chemistry (pp. 1–21). Washington, DC:American Chemical Society.
40. Lawrie, G. A., Schultz, M., O'Brien, G., Tasker, R., Bedford, S. B., Dargaville, T. R., … Wright, A. H., (in preparation). Shared perspectives, strategies and an inter-institutional collaborative community in development of concept items.
41. Lawrie, G. A., Schultz, M., & Wright, A. H., (in review). Insights into students’ conceptions as they enter tertiary chemistry studies:A longitudinal study exploring the utility of concept inventories.
42. Lawrie, G., Wright, A. H., Schultz, M., Dargaville, T., O'Brien, G., Bedford, S., & Thompson, C., (2013). Using formative feedback to identify and support first-year chemistry students with missing or misconceptions. A practice report. International Journal of the First Year in Higher Education, 4, 111–116. doi:10.5204/intjfyhe.v4i2.179
43. Lawrie, G. A., Wright, A., Schultz, M., Dargaville, T., Tasker, R., Williams, M., & Thompson, C., (2015). Closing the loop:A model for inter-institutional collaboration through delivering formative assessment in large, first-year stem classes. In G., Weaver, W. D., Burgess, A. L., Childress, & L., Slakey (Eds.), Transforming institutions:Undergraduate stem education for the 21st century (pp. 399–410). Purdue, IN:Purdue University Press.
44. Lewis, E. L., & Linn, M. C., (1994). Heat energy and temperature concepts of adolescents, adults, and experts:Implications for curricular improvements. Journal of Research in Science Teaching, 31, 657–677. doi:10.1002/tea.3660310607
45. Luxford, C. J., & Bretz, S. L., (2014). Development of the bonding representations inventory to identify student misconceptions about covalent and ionic bonding representations. Journal of Chemical Education, 91, 312–320. doi:10.1021/ed400700q
46. McInnis, C., James, R., & McNaught, C., (1995). First year on campus:Diversity in the initial experiences of Australian undergraduates. Report for centre for the study of higher education, University of Melbourne (Canberra, Australia).
47. Mulford, D. R., & Robinson, W. R., (2002). An inventory for alternate conceptions among first-semester general chemistry students. Journal of Chemical Education, 79, 739–744. doi:10.1021/ed079p739
48. Murphy, K., Holme, T., Zenisky, A., Caruthers, H., & Knaus, K., (2012). Building the ACS exams anchoring concept content map for undergraduate chemistry. Journal of Chemical Education, 89, 715–720. doi:10.1021/ed300049w
49. Nicol, D. J., & Macfarlane-Dick, D., (2006). Formative assessment and self-regulated learning:A model and seven principles of good feedback practice. Studies in Higher Education, 31, 199–218. doi:10.1080/03075070600572090
50. Osborne, R. J., & Cosgrove, M. M., (1983). Children’s conceptions of the changes of state of water. Journal of Research in Science Teaching, 20, 825–838. doi:10.1002/tea.3660200905
51. Othman, J., Treagust, D. F., & Chandrasegaran, A. L., (2008). An investigation into the relationship between students’ conceptions of the particulate nature of matter and their understanding of chemical bonding. International Journal of Science Education, 30, 1531–1550. doi:10.1080/09500690701459897
52. Pavelich, M. J., Jenkins, B., Birk, J., Bauer, R., & Krause, S., (2004). Development of a chemistry concept inventory for use in chemistry, materials and other engineering courses. Paper presented at the American Society for Engineering Education annual conference and exposition, Salt Lake City, UT.
53. Pentecost, T. C., & Langdon, L. S., (2008). Development of a covalent bonding and molecular structure learning progression and concept inventory for first semester general chemistry. Paper presented at the 20th biennial conference on Chemical Education, Bloomington, Indiana.
54. Peterson, R. F., Treagust, D. F., & Garnett, P., (1989). Development and application of a diagnostic instrument to evaluate grade 11 and 12 students’ concepts of covalent bonding and structure following a course of instruction. Journal of Research in Science Teaching, 26, 301–314. doi:10.1002/tea.3660260404
55. Posner, G. J., Strike, K. A., Hewson, P. W., & Gertzog, W. A., (1982). Accommodation of a scientific conception:Toward a theory of conceptual change. Science Education, 66, 211–227. doi:10.1002/sce.3730660207
56. Potgieter, M., & Davidowitz, B., (2011). Preparedness for tertiary chemistry:Multiple applications of the chemistry competence test for diagnostic and prediction purposes. Chemistry Education Research and Practice, 12, 193–204. doi:10.1039/C1RP90024B
57. Potgieter, M., Davidowitz, B., & Blom, B., (2005). Chemical concepts inventory of first year students at two tertiary institutions in South Africa. Paper presented at the conference of the South African association of Research in Mathematics, Science and Technology and Education, Namibia.
58. Potgieter, M., Davidowitz, B., & Venter, E., (2008). Assessment of preparedness of first-year chemistry students:Development and application of an instrument for diagnostic and placement purpose. African Journal of Research in SMT Education, Special Edition, 12, 1–18. doi:10.1080/10288457.2008.10740638
59. Prince, M., Vigeant, M., & Nottis, K., (2012). Development of the heat and energy concept inventory:Preliminary results on the prevalence and persistence of engineering students’ misconceptions. Journal of Engineering Education, 101, 412–438. doi:10.1002/j.2168-9830.2012.tb00056.x
60. Rasch, G., (1993). Probabilistic models for some intelligence and attainment tests. Chicago, IL:MESA Press.
61. Sadler, P. M., (1998). Psychometric models of student conceptions in science:Reconciling qualitative studies and distractor-driven assessment instruments. Journal of Research in Science Teaching, 35, 265–296. doi:10.1002/(SICI)1098-2736(199803)35:3<265::AID-TEA3>3.0.CO;2-P
62. Schmidt, H.-J., (1997). Students’ misconceptions–looking for a pattern. Science Education, 81, 123–135. doi:10.1002/(SICI)1098-237X(199704)81:2<123::AID-SCE1>3.0.CO;2-H
63. Schwartz, P., & Barbera, J., (2014). Evaluating the content and response process validity of data from the chemical concepts inventory. Journal of Chemical Education, 91, 630–640. doi:10.1021/ed400716p
64. diSessa, A. A., (1988). Knowledge in pieces. In G., Forman & P. B., Pufall (Eds.), Constructivism in the computer age (pp. 49–70). Hillsdale, NJ:Lawrence Erlbaum Associates.
65. diSessa, A. A., (2008). A bird’s-eye view of the “pieces” vs. “coherence” controversy (from the “pieces” side of the fence). In S., Vosniadou (Ed.), International handbook of research on conceptual change (pp. 35–60). New York, NY:Routledge.
66. diSessa, A. A., Gillespie, N. M., & Esterly, J. B., (2004). Coherence versus fragmentation in the development of the concept of force. Cognitive Science, 28, 843–900. doi:10.1016/j.cogsci.2004.05.003
67. Sijtsma, K., (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107–120. doi:10.1007/S11336-008-9101-0
68. Singer, S. R., Nielsen, N. R., & Schweingruber, H. A., (2012). Discipline-based education research:Understanding and improving learning in undergraduate science and engineering. Washington, DC:National Academies Press.
69. Solano-Flores, G., & Nelson-Barber, S., (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38, 553–573. doi:10.1002/tea.1018
70. Solomonidou, C., & Stavridou, H., (2000). From inert object to chemical substance:Students’ initial conceptions and conceptual development during an introductory experimental chemistry sequence. Science Education, 84, 382–400. doi:10.1002/(SICI)1098-237X(200005)84:3<382::AID-SCE4>3.0.CO;2-D
71. Stains, M., Escriu-Sune, M., De Santizo, Molina Alvarez de Santizo, M. L., & Sevian, H., (2011). Assessing secondary and college students’ implicit assumptions about the particulate nature of matter:Development and validation of the structure and motion of matter survey. Journal of Chemical Education, 88, 1359–1365. doi:10.1021/ed1002509
72. Taber, K. S., (2015). Exploring the language(s) of chemistry education. Chemistry Education Research and Practice, 16, 193–197. doi:10.1039/c5rp90003d
73. Talanquer, V., (2007). Explanations and teleology in chemistry education. International Journal of Science Education, 29, 853–870. doi:10.1080/09500690601087632
74. Talanquer, V., (2013). How do students reason about chemical substances and reactions?. In G., Tsaparlis, & H., Sevian (Eds.), Concepts of matter in science education (pp. 331–346). Dordrecht:Springer Science+Business Media.
75. Talanquer, V., (2016). Central ideas in chemistry:An alternative perspective. Journal of Chemical Education, 93, 3–8. doi:10.1021/acs.jchemed.5b00434
76. Tasker, R., (2014). Visualising the molecular world for a deep understanding of chemistry. Teaching Science, 60, 16–27.
77. Tasker, R., & Dalton, R., (2006). Research into practice:Visualisation of the molecular world using animations. Chemistry Education Research and Practice, 7, 141–159. doi:10.1039/B5RP90020D
78. Tavakol, M., & Dennick, R., (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. doi:10.5116/ijme.4dfb.8dfd
79. Treagust, D. F., (1986). Evaluating students’ misconceptions by means of diagnostic multiple choice items. Research in Science Education, 16, 199–207. doi:10.1007/BF02356835
80. Treagust, D. F., (1988). Development and use of diagnostic tests to evaluate students’ misconceptions in science. International Journal of Science Education, 10, 159–169. doi:10.1080/0950069880100204
81. Treagust, D. F., Chandrasegaran, A. L., Crowley, J., Yung, B. H. W., Cheong, I. P.-A., & Othman, J., (2010). Evaluating students’ understanding of kinetic particle theory concepts relating to the states of matter, changes of state and diffision:A cross-national study. International Journal of Science and Mathematics Education, 8, 141–164. doi:10.1007/s10763-009-9166-y
82. Treagust, D. F., Chandrasegaran, A. L., Zain, A. N. M., Ong, E. T., Karpudewan, M., & Halim, L., (2011). Evaluation of an intervention instructional program to facilitate understanding of basic particle concepts among students enrolled in several levels of study. Chemistry Education Research and Practice, 12, 251–261. doi:10.1039/C1RP90030G
83. Tyson, L., Treagust, D. F., & Bucat, R. B., (1999). The complexity of teaching and learning chemical equilibrium. Journal of Chemical Education, 76, 554–558. doi:10.1021/ed076p554
84. Tytler, R., (2000). A comparison of year 1 and year 6 students’ conceptions of evaporation and condensation:Dimensions of conceptual progression. International Journal of Science Education, 22, 447–467. doi:10.1080/095006900289723
85. Vosniadou, S., (2012). Reframing the classical approach to conceptual change:Preconceptions, misconceptions and synthetic models. In B. J., Fraser, K. G., Tobin, & C. J., McRobbie (Eds.), Second international handbook of science education (pp. 119–130). Dordrecht:Springer Science+Business Media.
86. Vosniadou, S., Vamvakoussi, X., & Skopeliti, I., (2008). The framework theory approach to the problem of conceptual change. In S., Vosniadou, (Ed.), International handbook of research on conceptual change (pp. 3–34). New York, NY:Routledge.
87. Wren, D., & Barbera, J., (2013). Gathering evidence for validity during the design, development, and qualitative evaluation of thermochemistry concept inventory items. Journal of Chemical Education, 90, 1590–1601. doi:10.1021/ed400384g
88. Wren, D., & Barbera, J., (2014). Psychometric analysis of the thermochemistry concept inventory. Chemistry Education Research and Practice, 15, 380–390. doi:10.1039/c3rp00170a
89. Yezierski, E. J., & Birk, J. P., (2006). Misconceptions about the particulate nature of matter:Using animations to close the gender gap. Journal of Chemical Education, 83, 954–960. doi:10.1021/ed083p954
90. Yorke, M., (2001). Formative assessment and its relevance to retention. Higher Education Research & Development, 20, 115–126. doi:10.1080/758483462