Theoretically grounded guidelines for assessing learning progress: Cognitive changes in ill-structured complex problem-solving contexts

Educational Technology Research and Development

Volumn 60, Issue 4, 2012, Pages 601-622

  Kim, Min Kyu ^a  

^a UNIVERSITY OF GEORGIA   (United States)

Author keywords

Cognitive change; Learning progress; Mental models; Problem solving

Indexed keywords

EID: 84865612370     PISSN: 10421629     EISSN: 15566501     Source Type: Journal    
DOI: 10.1007/s11423-012-9247-4     Document Type: Article

References (131)

1. Alexander, P. A. (2003). The development of expertise: The journey from acclimation to proficiency. Educational Researcher, 32(8), 10-14.
2. Alexander, P. A. (2004). A model of domain learning: Reinterpreting expertise as a multidimensional, multistage process. In D. Y. Dai & R. J. Sternberg (Eds.), Motivation, emotion, and cognition: Integrative perspectives on intellectual functioning and development (pp. 273-298). Mahwah, NJ: Lawrence Erlbaum.
3. Anderson, J. R. (1980). Cognitive psychology and its implications. New York: Freeman.
4. Anzai, Y., & Yokoyama, T. (1984). Internal models in physics problem solving. Cognition and Instruction, 1, 397-450.
5. Axelrod, R. (1976). Structure of decision: The cognitive maps of political elites. Princeton, NJ: Princeton University Press.
6. Barnett, S. M., & Ceci, S. J. (2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128(4), 612-637.
7. Bransford, J. D., Barclay, J. R., & Franks, J. J. (1972). Sentence memory: A constructive versus interpretive approach. Cognitive Psychology, 3, 193-209.
8. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). Learning and transfer. In How people learn: Brain, mind, experience, and school (pp. 31-78). Washington, DC: National Academy Press.
9. Bransford, J. D., & Franks, J. J. (1972). The abstraction of linguistic ideas. Cognitive Psychology, 2, 331-350.
10. Bransford, J. D., & Johnson, M. K. (1972). Contextual prerequisites for understanding: Some investigations of comprehension and recall. Journal of Verbal Learning and Verbal Behavior, 11(6), 717-726. doi: 16/S0022-5371(72)80006-9.
11. Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple implications. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of research in education (Vol. 24, pp. 61-100). Washington, DC: American Educational Research Association.
12. Bruner, J. S. (1961). The act of discovery. Harvard Educational Review, 31, 21-32.
13. Cañas, A. J. (2009). What are Propositions?... from a Concept Mapping Perspective. Retrieved 31 July 2011, from http://cmap. ihmc. us/docs/Proposition. html.
14. Carley, K., & Palmquist, M. (1992). Extracting, representing and analyzing mental models. Social Forces, 70, 215-225.
15. Carlson, J. (2002). Intelligence: Perspectives and a model. In G. M. van der Aalsvoort, W. C. M. Resing, & A. J. J. M. Ruijssenaars (Eds.), Learning potential assessment and cognitive training: Actual research and perspectives in theory building and methodology (pp. 11-27). New York: JAI.
16. Carver, S. M. (2006). Assessing for deep understanding. In R. K. Sawyer (Ed.), Cambridge handbook of the learning science (pp. 205-221). New York: Cambridge University Press.
17. Case, R. (Ed.). (1992). The mind's staircase: Exploring the conceptual underpinnings of children's thought and knowledge. Hillsdale, NJ: Erlbaum.
18. Chen, Z., & Siegler, R. S. (2000). Intellectual development in childhood. In R. Sternberg (Ed.), Handbook of intelligence (pp. 92-116). New York: Cambridge University Press.
19. Chi, M. T. H. (2006). Two approaches to the study of experts' characteristics. In K. A. Ericsson, N. Charness, P. Feltovich, & R. Hoffman (Eds.), Cambridge handbook of expertise and expert performance (pp. 121-130). New York: Cambridge University Press.
20. Chi, M. T. H., Glaser, R., & Farr, M. (Eds.). (1988). The nature of expertise. Hillsdale, NJ: Lawrence Erlbaum.
21. Clariana, R. B., & Wallace, P. (2007). A computer-based approach for deriving and measuring individual and team knowledge structure from essay questions. Journal of Educational Computing Research, 37(3), 211-227. doi: 10. 2190/EC. 37. 3. a.
22. Collins, A., & Gentner, D. (1987). How people construct mental models. In D. Holland & N. Quinn (Eds.), Cultural models in language and thought (pp. 243-265). Cambridge, MA: Cambridge University Press.
23. Collins, L. M., & Wugalter, S. E. (1992). Latent class models for stage-sequential dynamic latent variables. Multivariate Behavioral Research, 27(1), 131-157.
24. Cox, B. D. (1997). The rediscovery of the active learner in adaptive contexts: A developmental-historical analysis of transfer of training. Educational Psychologist, 32(1), 41-55.
25. Craik, K. J. W. (1943). The nature of explanation. Cambridge: Cambridge University Press.
26. Cronbach, J. L. (1957). The two disciplines of scientific psychology. American Psychologist, 12, 671-684.
27. Dochy, F., Segers, M., van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: A meta-analysis. Learning and Instruction, 13(5), 533-568. doi: 10. 1016/S0959-4752(02)00025-7.
28. Dreyfus, H. L., & Dreyfus, S. E. (2005). Peripheral vision: Expertise in real world contexts. Organization Studies, 26(5), 779-792.
29. Dreyfus, H. L., & Dreyfus, S. E. (1986). Mind over machine: The power of human intuition and expertise in the era of the computer. Oxford: B. Blackwell.
30. Ericsson, K. A. (2003). The acquisition of expert performance as problem solving: Construction and modification of mediating mechanisms through deliberate practice. In J. E. Davidson & R. J. Sternberg (Eds.), The psychology of problem solving (pp. 31-83). New York: Cambridge University Press.
31. Ericsson, K. A. (2005). Recent advances in expertise research: A commentary on the contributions to the special issue. Applied Cognitive Psychology, 19, 233-241.
32. Ericsson, K. A. (2006). The influence of experience and deliberate practice on the development of superior expert performance. In K. A. Ericsson, N. Charness, P. Feltovich, & R. R. Hoffman (Eds.), Cambridge handbook of expertise and expert performance (pp. 685-706). Cambridge: Cambridge University Press.
33. Ericsson, K., & Simon, H. (1980). Verbal reports as data. Psychological Review, 87(3), 215-251. doi: 10. 1037/0033-295X. 87. 3. 215.
34. Ericsson, K., & Simon, H. (1993). Protocol analysis: Verbal reports as data (2nd ed.). Boston: MIT Press. ISBN 0262050293.
35. Festinger, L. (1962). Cognitive dissonance. Scientific American, 207(4), 93-102.
36. Fischer, K. W., & Silvern, L. (1985). Stages and individual differences in cognitive development. Annual Review of Psychology, 36, 613-648.
37. Flavell, J. H. (1985). Cognitive development (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall.
38. Flavell, J. H. (1992). Cognitive development: Past, present, and future. Developmental Psychology, 28(6), 998-1005.
39. Flavell, J. H., & Miller, P. H. (1998). Social cognition. In W. Damon, D. Kuhn, & R. S. Siegler (Eds.), Handbook of child psychology (Vol. 2, pp. 851-898)., Cognition, perception, and language New York: Wiley.
40. Forbus, K. D., Gentner, D., & Law, K. (1995). MAC/FAC: A model of similarity-based retrieval. Cognitive Science, 19, 141-205.
41. Garnham, A. (1987). Mental models as representations of discourse and text. Chichester: Ellis Horwood Ltd.
42. Garnham, A. (2001). Mental models and the interpretation of anaphora. Hove: Psychology Press.
43. Gentner, D., Holyoak, K. J., & Kokinov, B. N. (2001). The analogical mind. Cambridge, MA: MIT Press.
44. Gentner, D., Loewenstein, J., & Thompson, L. (2003). Learning and transfer: A general role for analogical encoding. Journal of Educational Psychology, 95(2), 393-408. doi: 10. 1037/0022-0663. 95. 2. 393.
45. Gentner, D., & Medina, J. (1998). Similarity and the development of rules. Cognition, 65, 263-297.
46. Gentner, D., & Stevens, A. L. (1983). Mental models. Hillsdale, NJ: Lawrence Erlbaum.
47. Gijbels, D., Dochy, F., den Bossche, P. V., & Segers, M. (2005). Effects of problem-based learning: A meta-analysis from the angle of assessment. Review of Educational Research, 75(1), 27-61.
48. Glaser, R. (1992). Expert knowledge and process of thinking. In D. F. Halpern (Ed.), Enhancing thinking skills in the sciences and mathematics (pp. 63-75). Hillsdale, NJ: Erlbaum.
49. Glaser, E. M., Abelson, H. H., & Garrison, K. N. (1983). Putting knowledge to use: Facilitating the diffusion of knowledge and the implementation of planned change (1st ed.). San Francisco: Jossey-Bass.
50. Goldsmith, T. E., & Kraiger, K. (1997). Applications of structural knowledge assessment to training evaluation. In J. K. Ford (Ed.), Improving training effectiveness in organizations (pp. 73-95). Mahwah, NJ: Lawrence Erlbaum.
51. Goodman, C. M. (1987). The Delphi technique: A critique. Journal of Advanced Nursing, 12(6), 729-734.
52. Greeno, J. G. (1989). Situations, mental models, and generative knowledge. In D. Klahr & K. Kotovsky (Eds.), Complex information processing (pp. 285-318). Hillsdale, NJ: Lawrence Erlbaum.
53. Hoffman, R. R. (1998). How can expertise be defined?: Implications of research from cognitive psychology. In R. Williams, W. Faulkner, & J. Fleck (Eds.), Exploring expertise (pp. 81-100). New York: Macmillan.
54. Holyoak, K. J., & Koh, K. (1987). Surface and structural similarity in analogical transfer. Memory & Cognition, 15, 332-340.
55. Hsu, C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment Research & Evaluation, 12(10). http://pareonline. net/getvn. asp?v=12&n=10
56. Ifenthaler, D. (2006). Diagnose lernabhängiger Veränderung mentaler Modelle. Entwicklung der SMD-Technologie als methodologische Verfahren zur relationalen, strukturellen und semantischen Analyse individueller Modellkonstruktionen. Universitäts-Dissertation, Freiburg.
57. Ifenthaler, D. (2010). Relational, structural, and semantic analysis of graphical representations and concept maps. Educational Technology Research and Development, 58(1), 81-97. doi: 10. 1007/s11423-008-9087-4.
58. Ifenthaler, D., Masduki, I., & Seel, N. M. (2009). The mystery of cognitive structure and how we can detect it: Tracking the development of cognitive structures over time. Instructional Science,. doi: 10. 1007/s11251-009-9097-6.
59. Ifenthaler, D., & Seel, N. M. (2005). The measurement of change: Learning-dependent progression of mental models. Technology, Instruction, Cognition, and Learning, 2(4), 321-340.
60. Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Cambridge: Cambridge University Press.
61. Johnson-Laird, P. N. (2005a). Mental models and thoughts. In K. J. Holyoak (Ed.), The Cambridge handbook of thinking and reasoning (pp. 185-208). Cambridge: Cambridge University Press.
62. Johnson-Laird, P. N. (2005b). The history of mental models. In K. I. Manktelow & M. C. Chung (Eds.), Psychology of reasoning: Theoretical and historical perspectives (pp. 179-212). New York: Psychology Press.
63. Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology: Research and Development, 45(1), 65-94.
64. Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology: Research and Development, 48(4), 63-85.
65. Jonassen, D. H., Beissner, K., & Yacci, M. (1993). Structural knowledge: Techniques for representing, conveying, and acquiring structural knowledge. Hillsdale, NJ: Lawrence Erlbaum.
66. Judd, C. H. (1908). The relation of special training to general intelligence. Educational Review, 36, 28-42.
67. Kail, R. V. (2004). Cognitive development includes global and domain-specific processes. Merrill Palmer Quarterly Journal of Developmental Psychology, 50(4), 445-455.
68. Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38, 23-31.
69. Katz, J. J., & Postal, P. M. (1964). An integrated theory of linguistic descriptions. Cambridge: MIT Press.
70. Kendler, H. H., & Kendler, T. S. (1962). Vertical and horizontal processes in problem solving. Psychological Review, 69, 1-16.
71. Kernis, M. H., Brockner, J., & Frankel, B. S. (1989). Self-esteem and reactions to failure: The mediating role of overgeneralization. Journal of Personality and Social Psychology, 57, 707-714.
72. Kieras, D. E., & Bovair, S. (1984). The role of a mental model in learning to operate a device. Cognitive Science, 8, 255-273.
73. Kim, M. (2011a). Development of an assessment technology for measuring knowledge structures using natural language response to a complex problem scenario. Cognition and Instruction (submitted for publication).
74. Kim, M. (2011b). Investigation of a model of stage-sequential learning progress in a complex problem-solving situation: Application of the Log-Linear Cognitive Diagnostic Model (LCDM). Contemporary Educational Psychology (submitted for publication).
75. Kim, M. (2012). Cross-validation study on methods and technologies to assess mental models in a complex problem solving situation. Computers in Human Behavior, 28(2), 703-717.
76. Kintsch, W., & van Dijk, T. A. (1978). Toward a model of text comprehension and production. Psychological Review, 85(5), 363-394. doi: 10. 1037/0033-295X. 85. 5. 363.
77. Kubose, T. T., Holyoak, K. J., & Hummel, J. E. (2002). The role of textual coherence in incremental analogical mapping. Journal of Memory and Language, 47(3), 407-435.
78. Lave, J. (1988). Cognition in practice: Mind, mathematics, and culture in everyday life. Cambridge: Cambridge University Press.
79. Lee, J., & Park, O. (2007). Adaptive instructional systems. In J. M. Spector, M. D. Merill, J. van Merrienboer, & M. P. Driscoll (Eds.), Handbook of research for educational communications and technology (pp. 469-484). New York: Routledge, Taylor & Francis Group.
80. Lemaire, P., & Siegler, R. S. (1995). Four aspects of strategic change: Contributions to children's learning of multiplication. Journal of Experimental Psychology: General, 124(1), 83-97.
81. Lobato, J. (2006). Alternative perspectives on the transfer of learning: History, issues, and challenges for future research. Journal of the Learning Sciences, 15(4), 431-449.
82. Mayer, R. E. (1989). Models for understanding. Review of Educational Research, 59(1), 43-64.
83. Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
84. Newell, A., & Simon, H. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice Hall.
85. Norman, D. (1983). Some observations on mental models. In D. Gentner & A. L. Stevens (Eds.), Mental models (pp. 7-14). Hillsdale, NJ: Lawrence Erlbaum.
86. Norman, D. A., Gentner, D. R., & Stevens, A. L. (1976). Comments on learning schemata and memory representation. In D. Klahr (Ed.), Cognition and instruction (pp. 177-196). Hillsdale, NJ: Lawrence Erlbaum.
87. Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1), 15-29. doi: 10. 1016/j. im. 2003. 11. 002.
88. Opfer, J. E., & Siegler, R. S. (2004). Revisiting preschoolers' living things concept: A microgenetic analysis of conceptual change in basic biology. Cognitive Psychology, 49, 301-332.
89. Piaget, J. (1964). Development and learning. In R. E. Ripple & V. N. Rockcastle (Eds.), Piaget rediscovered (pp. 7-20). Ithaca, NY: Cornell University.
90. Piaget, J. (1976). Science of education and the psychology of the child. New York: Orion Press.
91. Pirnay-Dummer, P. (2006). Expertise und Modellbildung-MITOCAR. Universitäts-Dissertation, Freiburg.
92. Pirnay-Dummer, P., Ifenthaler, D., & Spector, J. (2010). Highly integrated model assessment technology and tools. Educational Technology Research and Development, 58(1), 3-18. doi: 10. 1007/s11423-009-9119-8.
93. Pretz, J. E., Naples, A. J., & Sternberg, R. J. (2003). Recognizing defining, and representing problems. In J. E. Davidson & R. J. Sternberg (Eds.), The psychology of problem solving (pp. 3-30). New York: Cambridge University Press.
94. Reeves, L. M., & Weisberg, R. W. (1994). The role of content and abstract information in analogical transfer. Psychological Bulletin, 115(3), 381-400.
95. Rumelhart, D. E., & Norman, D. A. (1978). Accretion, tuning and restructuring: Three model of learning. In R. L. Klatzky & J. W. Cotton (Eds.), Semantic factors in cognition (pp. 37-53). Hillsdale, NJ: Lawrence Erlbaum.
96. Rupp, A. A., Templin, J. L., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York: Guilford.
97. Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144.
98. Salomon, G., & Perkins, D. N. (1989). Rocky roads to transfer: Rethinking mechanisms of a neglected phenomenon. Educational Psychologist, 24, 113-142.
99. Schlomske, N., & Pirnay-Dummer, P. (2008). Model based assessment of learning dependent change during a two semester class. In Kinshuk, Sampson, D., & Spector, M. (Eds.), Proceedings of IADIS International Conference Cognition and Exploratory Learning in Digital Age 2008, Freiburg, pp. 45-53.
100. Schvaneveldt, R. W. (Ed.). (1990). Pathfinder associative networks: Studies in knowledge organization. Norwood, NJ: Ablex.
101. Schvaneveldt, R. W., Durso, F. T., Goldsmith, T. E., Breen, T. J., & Cooke, N. M. (1985). Measuring the structure of expertise. International Journal of Man-Machine Studies, 23, 699-728.
102. Seel, N. M. (2001). Epistemology, situated cognition, and mental models: Like a bridge over troubled water. Instructional Science, 29(4-5), 403-427.
103. Seel, N. M. (2003). Model-centered learning and instruction. Technology, Instruction, Cognition, and Learning, 1(1), 59-85.
104. Seel, N. M. (2004). Model-centered learning environments: Theory, Instructional design, and ethics. In N. M. Seel & S. Dijkstra (Eds.), Curriculum, plans, and processes in instruction design: International perspectives (pp. 49-74). Mahwah, NJ: Lawrence Erlbaum.
105. Seel, N. M., & Dinter, F. R. (1995). Instruction and mental model progression: Learner-dependent effects of teaching strategies on knowledge acquisition and analogical transfer. Educational Research and Evaluation, 1(1), 4-35.
106. Segers, M. (1997). An alternative for assessing problem-solving skills: The overall test. Studies in Educational Evaluation, 23(4), 373-398.
107. Shaklee, H. (1979). Bounded rationality and cognitive development: Upper limits on growth? Cognitive Psychology, 11, 327-345.
108. Shute, V. J., & Zapata-Rivera, D. (2007). Adaptive technologies. In J. M. Spector, M. D. Merill, J. van Merrienboer, & M. P. Driscoll (Eds.), Handbook of research for educational communications and technology (pp. 227-294). New York: Routledge, Taylor & Francis Group.
109. Siegler, R. S. (2005). Children's learning. American Psychologist, 60, 769-778.
110. Siegler, R. S., & Klahr, D. (1982). When do children learn: The relationship between existing knowledge and the ability to acquire new knowledge. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 2, pp. 121-211). Hillsdale, NJ: Lawrence Erlbaum.
111. Siegler, R. S., Thompson, C. A., & Opfer, J. E. (2009). The logarithmic-to-linear shift: One learning sequence, many tasks, many time scales. Mind, Brain, and Education, 3, 143-150.
112. Simon, H. A., & Hayes, J. R. (1976). The understanding process: Problem isomorphs. Cognitive Psychology, 8, 165-190.
113. Singley, M. K., & Anderson, J. R. (1989). The transfer of cognitive skill. Cambridge, MA: Harvard University Press.
114. Smith, J. P., diSessa, A. A., & Roschelle, J. (1993). Misconceptions reconceived: A constructivist analysis of knowledge in transition. The Journal of the Learning Sciences, 3(2), 115-163.
115. Snow, R. E. (1990). New approaches to cognitive and conative assessment in education. International Journal of Educational Research, 14(5), 455-473.
116. Spector, J. M. (2004). Problems with problem-based learning: Comments on model-centered learning and instruction in Seel (2003). Technology, Instruction, Cognition and Learning, 1(4), 359-374.
117. Spector, J. M. (2008a). Cognition and learning in the digital age: Promising research and practice. Computers in Human Behavior, 24(2), 249-262.
118. Spector, J. M. (2008b). Expertise and dynamic tasks. In H. Qudrat-Allah, J. M. Spector, & P. I. Davidsen (Eds.), Complex decision making: Theory and practice. Berlin: Springer.
119. Spector, J. M. (2008c). Complex domain learning. In H. H. Adelsberger, Kinshuk, J. M. Pawlowski, & D. Sampson (Eds.), Handbook of information technologies for education and training (pp. 261-275). Heidelberg: Springer.
120. Spector, J. M., & Koszalka, T. A. (2004). The DEEP methodology for assessing learning in complex domains. Final report to the National Science Foundation Evaluative Research and Evaluation Capacity Building, Syracuse University, Syracuse, NY.
121. Sternberg, R. J. (2008). Applying psychological theories to educational practice. American Educational Research Journal, 45(1), 150-165.
122. Sternberg, R. J., & Frensch, P. A. (1993). Mechanisms of transfer. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 25-38). Norwood, NJ: Ablex.
123. Sugrue, B. (1995). A theory-based framework for assessing domain-specific problem-solving ability. Educational Measurement: Issues and Practice, 14(3), 29-35. doi: 10. 1111/j. 1745-3992. 1995. tb00865. x.
124. Taricani, E. M., & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 54(1), 65-82.
125. Thorndike, E. L. (1906). Principles of teaching. New York: Seiler.
126. van Gog, T., Ericsson, K. A., Rikers, R. M., & Paas, F. (2005). Instructional design for advanced learners: Establishing connections between the theoretical frameworks of cognitive load and deliberate practice. Educational Technology: Research and Development, 53(3), 73-81.
127. Vosniadou, S., & Skopeliti, I. (2005). Developmental shifts in children's categorization of the earth. In B. G. Bara, L. Barsalou, & M. Bucciarelli (Eds.), Proceedings of the XXVII Annual Conference of the Cognitive Science Society (pp. 2325-2330), Stresa.
128. Vosniadou, S., Vamvakoussi, X., & Skopeliti, I. (2008). The framework theory approach to the problem of conceptual change. In S. Vosniadou (Ed.), International handbook of research on conceptual change (pp. 3-34). New York: Routledge.
129. Vygotsky, L. (1934/1978). Mind in society. Cambridge, MA: Harvard University Press.
130. Werner, H. (1957). The concept of development from a comparative and organismic point of view. In D. B. Harris (Ed.), The concept of development: An issue in the study of human behavior (pp. 125-148). Minneapolis, MN: University of Minnesota.
131. Wood, P. K. (1983). Inquiring systems and problem structures: Implications for cognitive development. Human Development, 26, 249-265.