An exploratory study to examine the feasibility of measuring problem-solving processes using a click-through interface

Journal of Technology, Learning, and Assessment

Volumn 2, Issue 2, 2003, Pages 1-30

  Chung, Gregory K W K ^a,b,c,d   Baker, Eva L ^b,e,f  

^a Natl Ctr Res Eval Std S   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c PEPPERDINE UNIVERSITY   (United States)

^d UNIVERSITY OF HAWAII   (United States)

^e Office of Technology Assessment   (United States)

^f US Dept Educ Advsy Cncl E   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 3042666207     PISSN: 15402525     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (38)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.
2. Ainsworth, E., Guidry, T., Chung, G. K. W. K., Delacruz, G. C., & Kaiser, W. L. (2003, April). UCLA/Hewlett-Packard Computer Science/Engineering Retention Project (Quarterly Report). Los Angeles: University of California, Center for Excellence in Engineering & Diversity.
3. Atman, C. J., Chimka, J. R., Bursic, K. M., & Nachtmann, H. L. (1999). A comparison of freshman and senior engineering design processes. Design Studies, 20, 131-152.
4. Bakeman, R., & Gottman, J. M. (1997). Observing interaction (2nd ed.). New York: Cambridge University Press.
5. Bakeman, R., McArthur, D., & Quera, V. (1996). Detecting group differences in sequential association using sampled permutations: Log odds, kappa, and phi compared. Behavior Research Methods, Instruments, & Computers, 28, 446-457.
6. Bakeman, R., & Quera, V. (1995). Analyzing interaction. New York: Cambridge University Press.
7. Baker, E. L., & Mayer, R. E. (1999). Computer-based assessment of problem solving. Computers in Human Behavior, 15, 269-282.
8. Baker, E. L., & O'Neil, H. F., Jr. (2002). Measuring problem solving in computer environments: Current and future states. Computers in Human Behavior, 18, 609-622.
9. Bennett, R. E. (1999). Using new technology to improve assessment. Educational Measurement: Issues and Practice, 18(3), 5-12.
10. Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It's not only the scoring. Educational Measurement, 17(4), 9-17.
11. Bowers, C. A., Jentsch, F., Salas, E., & Braun, C. C. (1998). Analyzing communication sequences for team training needs assessment. Human Factors, 40, 672-679.
12. Chung, G. K. W. K., de Vries, L. F., Cheak, A. M., Stevens, R. H., & Bewley, W. L. (2002). Cognitive process validation of an online problem solving assessment. Computers in Human Behavior, 18, 669-684.
13. Chung, G. K. W. K., Kim, J.-O., de Vries, L. F., & Phan, C. H. (2003). Validating a method to infer learning processes from users' clickstream data using Bayesian networks. Unpublished manuscript.
14. Chung, G. K. W. K., & Baker, E. L. (2003). Issues in the reliability and validity of automated scoring of constructed responses. In M. D. Shermis & J. E. Burstein (Eds.), Automated essay grading: A cross-disciplinary approach (pp. 23-40). Mahwah, NJ: Erlbaum.
15. Clauser, B. E. (2000). Recurrent issues and recent advances in scoring performance assessments. Applied Psychological Measurement, 24, 310-324.
16. Gottman, R. M., Markman, H., & Notarius, C. (1977). The topology of marital conflict: A sequential analysis of verbal and nonverbal behavior. Journal of Marriage and the Family, 39, 461-477.
17. Gottman, J. M., & Roy, A. K. (1990). Sequential analysis: A guide for behavioral researchers. New York: Cambridge University Press.
18. Herl, H. E., Baker, E. L., & Niemi, D. (1996). Construct validation of an approach to modeling cognitive structure of U.S. history knowledge. Journal of Educational Research, 89, 206-218.
19. Herl, H. E., O'Neil, H. F., Jr., Chung, G. K. W. K., Bianchi, C., Wang, S.L., Mayer, R. E., et al. (1999). Final report for validation of problem-solving measures. (CSE Tech. Rep. No. 501). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
20. Hirokawa, R. Y. (1980). A comparative analysis of communication patterns within effective and ineffective decision-making groups. Communication Monographs, 47, 312-321.
21. Huff, K. L., & Sireci, S. G. (2001). Validity issues in computer-based scoring. Educational Measurement: Issues and Practice, 20(3), 16-25.
22. Katz, I. R., & James, C. M. (1998). Toward assessment of design skill in engineering (GRE Research Report 97-16). Princeton, NJ: Educational Testing Service.
23. Lawson, A. E. (1987). Classroom test of scientific reasoning: Revised paper-pencil edition. Tempe: Arizona State University.
24. Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.
25. Mayer, R. E., & Gallini, J. (1990). When is an illustration worth ten-thousand words? Journal of Educational Psychology, 82, 715-726.
26. Mayer, R. E., & Wittrock, M. C. (1996). Problem-solving transfer. In D.C. Berliner & R.C. Calfee (Eds.), Handbook of educational psychology (pp. 47-62). New York: Macmillian Library Reference USA, Simon & Schuster Macmillan.
27. Mullins, C. A., Atman, C. J., & Shuman, L. J. (1999). Freshman engineers' performance when solving design problems. IEEE Transactions on Education, 42, 281-287.
28. Munro, A., Johnson, M., Pizzini, Q., Surmon, D., Towne, D., & Wogulis, J. (1997). Authoring simulation-centered tutors with RIDES. International Journal of Artificial Intelligence in Education, 8, 284-316.
29. Munro, A., & Pizzini, Q. A. (1998). VIVIDS reference manual. Los Angeles: University of Southern California, Behavioral Technology Laboratories.
30. National Research Council. (2001). Knowing what students know: The science and design of educational assessment. Committee on the Foundations of Assessment, Pellegrino, J., Chudowsky, N., & Glaser, R. (Eds.). [Board on Testing and Assessment, Center for Education. Division of Behavioral and Social Sciences and Education.] Washington, DC: National Academy Press.
31. Nhouyvanisvong, A., & Katz, I. R. (1998). The structure of generate-and-test in algebra problem-solving. In Proceedings of the Annual Meeting of the Cognitive Science Society (pp. 770-775). Hillsdale, NJ: Erlbaum.
32. O'Neil, H. F., Jr. (1999). Perspectives on computer-based performance assessment of problem solving. Computers in Human Behavior, 15, 225-268.
33. O'Neil, H. F., Jr., Chuang, S. S., & Chung, G. K. W. K. (in press). Issues in the computer-based assessment of collaborative problem solving. Assessment in Education.
34. Sanderson, P. M., & Fisher, C. (1994). Exploratory sequential data analysis: Foundations. Human-Computer Interaction, 9, 251-317.
35. Schacter, J., Herl, H. E., Chung, G. K. W. K., Dennis, R. A., & O'Neil, H. F., Jr. (1999). Computer-based performance assessments: A solution to the narrow measurement and reporting of problem-solving. Computers in Human Behavior, 15, 403-418.
36. Stevens, R., Ikeda, J., Casillas, A., Palacio-Cayetano, J., & Clyman, S. (1999). Artificial neural network-based performance assessments. Computers in Human Behavior, 15, 295-313.
37. Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4, 295-312.
38. Towne, D. M., Munro, A., Pizzini, Q. A., Surmon, D. S., Coller, L. D., & Wogulis, J. L. (1990). Model-building tools for simulation-based training. Interactive Learning Environments, 1, 33-35.