Sophisticated tasks in e-assessment: What are they and what are their benefits?

Assessment and Evaluation in Higher Education

Volumn 34, Issue 3, 2009, Pages 305-319

  Boyle, Andrew ^a   Hutchison, Dougal ^b  

^a Office of the Qualifications and Examinations Regulator (Ofqual)   (United Kingdom)

^b NATIONAL FOUNDATION FOR EDUCATIONAL RESEARCH   (United Kingdom)

Author keywords

Benefits of e assessment; E assessment; Formative assessment; Higher order learning outcomes; Sophisticated tasks; Taxonomies and definitions of task types

Indexed keywords

EID: 78650788172     PISSN: 02602938     EISSN: 1469297X     Source Type: Journal    
DOI: 10.1080/02602930801956034     Document Type: Article

References (65)

1. Ahmed, A., and A. Pollitt. 1999. Curriculum demands and question difficulty. Paper presented at IAEA Conference, May, in Slovenia. http://www.cambridgeassessment.org.uk/research/confproceedingsetc/IAEA1999AAAP/file (accessed August 2, 2007).
2. Baggott, G., and R. Rayne. 2002. The online assessment and feedback project: Project rationale. http://www.bbk.ac.uk/olaaf/extras/rationale-extra.pdf (accessed August 2, 2007).
3. Bennett, R.E. 1993. Towards intelligent assessment: An integration of constructed-response testing, artificial intelligence and model-based measurement. In Test theory for a new generation of tests, ed. N. Frederiksen, R.J. Mislevy, and I.I. Bejar, 99-123. Hillsdale, NJ: Lawrence Erlbaum.
4. Bennett, R.E. 1998. Reinventing assessment: Speculations on the future of large-scale educational testing. Princeton, NJ: Educational Testing Service Policy Information Center. ftp://ftp.ets.org/pub/res/reinvent.pdf (accessed August 2, 2007).
5. Bennett, R.E. 2002. Inexorable and inevitable: The continuing story of technology and assessment. Journal of Technology, Learning and Assessment 1, no. 1. http://www.bc.edu/research/intasc/jtla/journal/pdf/v1n1_jtla.pdf (accessed August 2, 2007).
6. Bennett, R.E., M. Goodman, J. Hessinger, J. Ligget, G. Marshall, H. Kahn, and J. Zack. 1999. Using multimedia in large-scale computer-based testing programs. Computers in Human Behavior 15: 283-94.
7. Bennett, R.E., M. Morley, and D. Quardt. 2000. Three response types for broadening the conception of mathematical problem solving in computerised tests. Applied Psychological Measurement 24, no. 4: 294-309.
8. Black, P., and D. Wiliam. 1998. Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice 5, no. 1: 7-73.
9. Black, P., and D. Wiliam. 2003. In praise of educational research: Formative assessment. British Educational Research Journal 29, no. 5: 623-37.
10. Boyle, A. 2007. The formative use of e-assessment: Some early implementations, and suggestions for how we might move on. In Proceedings of the 11th International Computer Assisted Assessment (CAA) Conference, ed. F. Khandia. http://www.caaconference.com (accessed August 2, 2007).
11. Boyle, A., and D. O'Hare. 2003. Finding appropriate methods to assure quality Computer-Based Assessment development in UK Higher Education. In Proceedings of the 7th International Computer Assisted Assessment (CAA) Conference, ed. J. Christie. http://www.caaconference.com (accessed August 2, 2007).
12. Boyle, A., D. Hutchison, D. O'Hare, and A. Patterson. 2002. Item selection and application in higher education. In Proceedings of the 6th International Computer Assisted Assessment (CAA) Conference, ed. M. Danson. http://www.caaconference.com (accessed August 2, 2007).
13. Brennan, R.L. 2001. Generalizability theory. New York: Springer Verlag.
14. Bunderson, C.V., D.K. Inouye, and J.B. Olsen. 1989. The four generations of computerized educational measurement. In Educational measurement, ed. R.L. Linn, 367-407, 3rd ed. New York: American Council on Education.
15. Conole, G., and B. Warburton. 2005. A review of computer-assisted assessment. Association for Learning Technology Journal 13, no. 1: 17-31.
16. Crossouard, B., J. Pryor, and H. Torrance. 2004. Creating an alternative assessment regime with online formative assessment: Developing a researcher identity. Paper presented at the European Conference on Education Research, September, in Crete. http://www.esri.mmu.ac.uk/respapers/papers-pdf/ECER%20Creteppr.pdf (accessed August 2, 2007).
17. Dermo, J. 2007. Benefits and obstacles: Factors affecting the uptake of CAA in undergraduate courses. In Proceedings of the 11th International Computer Assisted Assessment (CAA) Conference, ed. F. Khandia. http://www.caaconference.com (accessed August 2, 2007).
18. DfES (Department for Education and Skills). 2005. White paper on 14-19 education and skills. http://www.dfes.gov.uk/publications/14-19educationandskills/ (accessed August 2, 2007).
19. Dikli, S. 2006. An overview of automated scoring of essays. Journal of Technology, Learning and Assessment 5, no. 1. http://www.jtla.org (accessed August 2, 2007).
20. Duke-Williams, E., and T. King. 2001. Using computer-aided assessment to test higher level learning. In Proceedings of the 5th International Computer Assisted Assessment (CAA) Conference, ed. M. Danson and C. Eabry. http://www.caaconference.com (accessed August 2, 2007).
21. Harlen, W., and R. Deakin Crick. 2002. A systematic review of the impact of summative assessment and tests on students' motivation for learning. http://eppi.ioe.ac.uk/EPPIWeb/dl.aspx?Path=reel%5Creview_groups%5Cassessment%5Cass_rv1&FileName=ass_rv1 (accessed August 2, 2007).
22. Harlen, W., and R. Deakin Crick. 2003. A systematic review of the impact on students and teachers of the use of ICT for assessment of creative and critical thinking skills. http://eppi.ioe.ac.uk/EPPIWeb/dl.aspx?Path=reel%5Creview_groups%5Cassessment%5Cass_rv2&FileName=ass_rv2 (accessed August 2, 2007).
23. Huff, K.L., and S.G. Sireci. 2001. Validity issues in computer based testing. Educational Measurement: Issues and Practice 20, no. 3: 16-25.
24. Katz, I.R., D.M. Williamson, H.L. Nadelman, I. Kirsch, R.G. Almond, P.L. Cooper, M.L. Redman, and D. Zapata. 2004. Assessing information and communications technology literacy for higher education. Paper presented at the 30th annual conference of the International Association, Educational Assessment (IAEA), June 13-18, in Philadelphia, PA.
25. Keppell, M., E. Au, A. Ma, and C. Chan. 2006. Peer learning and learning-oriented assessment in technology-enhanced environments. Assessment & Evaluation in Higher Education 31, no. 4: 453-64.
26. Kirriemuir, J., and A. McFarlane. 2006. Literature review in games and learning. http://www.futurelab.org.uk/download/pdfs/research/lit_reviews/Games_Review1.pdf (accessed August 2, 2007).
27. Koch, D.A. 1993. Testing goes graphical. Journal of Interactive Instruction Development 5: 14-21.
28. Mackenzie, D. 1999. Recent developments in the Tripartite Interactive Assessment Delivery system (TRIADs). Paper presented at the 3rd international Computer-Assisted Assessment Conference, July. http://www.caaconference.com (accessed August 2, 2007).
29. Mackenzie, D. 2004. Online assessment: Quality production and delivery for higher education. www.enhancementthemes.ac.uk/documents/events/20040416/Mackenziepaperrevised.doc (accessed August 2, 2007).
30. Maughan, S., and D. Mackenzie. 2004. Bioscope: The assessment of process and outcomes using the TRIAD system. In Proceedings of the 8th International Computer Assisted Assessment (CAA) Conference, ed. M. Ashby. http://www.caaconference.com/ (accessed August 2, 2007).
31. Maughan, S., D. Peet, and A. Willmott. 2001. On-line formative assessment item banking and learning support. In Proceedings of the 5th International Computer Assisted Assessment (CAA) Conference, ed. M. Danson and C. Eabry. http://www.caaconference.com (accessed August 2, 2007).
32. McLaughlin, P.J., S.L. Fowell, P.H. Dangerfield, D.J. Newton, and S.E. Perry. 2004. Development of computerised assessment (TRIADs) in an undergraduate medical school. In Advances in computer aided assessment, ed. D. O'Hare and D. Mackenzie, 25-32. Birmingham: SEDA.
33. Mislevy, R.J. 1993. Foundations of a new test theory. In Test theory for a new generation of tests, ed. N. Frederiksen, R.J. Mislevy, and I.I. Bejar, 19-39. Hillsdale, NJ: Lawrence Erlbaum.
34. Mislevy, R.J., R.G. Almond, and J.F. Lukas. 2003. A brief introduction to evidence-centered design. http://www.education.umd.edu/EDMS/mislevy/papers/BriefIntroECD.pdf (accessed August 2, 2007).
35. Mullis, I.V.S., M.O. Martin, G.J. Ruddock, C.Y. O'Sullivan, A. Arora, and E. Erberber. 2007. Trends in International Mathematics and Science Study (TIMSS) 2007 Assessment Frameworks. Boston, MA: TIMSS & Progress in International Reading Literacy Study (PIRLS) International Study Center, Lynch School of Education, Boston College.
36. Nicol, D., and D. Macfarlane-Dick. 2004. Rethinking formative assessment in HE: A theoretical model and seven principles of good feedback practice. In Enhancing student learning through effective formative feedback, ed. C. Juwah, D. MacFarlane-Dick, B. Matthew, D. Nicol, D. Ross, and B. Smith. York: The Higher Education Academy.
37. OECD (Organisation for Economic Co-operation and Development). 2003. The PISA 2003 assessment framework. Paris: Author.
38. O'Neil, H., and D.C.D. Klein. 1997. Feasibility of machine scoring of concept maps (CSE Technical Report 460). www.cse.ucla.edu/products/Reports/TECH460.pdf (accessed August 2, 2007).
39. Osmundson, E., G. Chung, H.E. Herl, and D.C.D. Klein. 1999. Knowledge mapping in the classroom: A tool for examining the development of students' conceptual understandings (CSE Technical Report 507). http://www.cse.ucla.edu/products/Reports/TECH507.pdf (accessed August 2, 2007).
40. Parshall, C.G., T. Davey, and P.J. Pashley. 2000. Innovative item types for computerized testing. In Computerized adaptive testing: Theory and practice, ed. W.J. van der Linden and C.A.W. Glas, 129-148. Dordrecht, The Netherlands: Kluwer.
41. Pellegrino, J.W., N. Chudowsky, and R. Glaser, eds. 2001. Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
42. QCA (Qualifications and Curriculum Authority). 2002. Assessing gifted and talented children. Papers presented at an international seminar, February, in London.
43. QCA (Qualifications and Curriculum Authority). 2004. World class arena. http://www.qca.org.uk/qca_5450.aspx (accessed August 2, 2007).
44. Raikes, N., J. Greatorex, and S. Shaw. 2004. From paper to screen: Some issues on the way. Paper presented at the International Association of Educational Assessment Conference, June 13-18. http://www.cambridgeassessment.org.uk/research/confproceedingsetc/IAEA2000NRJGSS/file/(accessed August 2, 2007).
45. Raikes, N., and R. Harding. 2003. The horseless carriage stage: Replacing conventional measures. Assessment in Education: Principles, Policy and Practice 10, no. 3: 267-77.
46. Rayne, R., and G. Baggott. 2004. Computer-based and computer-assisted tests to assess procedural and conceptual knowledge. In Proceedings of the 8th International Computer Assisted Assessment (CAA) Conference, ed. M. Ashby. http://www.caaconference.com/ (accessed August 2, 2007).
47. Ricketts, C., P. Filmore, R. Lowry, and S. Wilks. 2003. How should we measure the costs of computeraided assessment? In Proceedings of the 7th International Computer Assisted Assessment (CAA) Conference, ed. J. Christie. http://www.caaconference.com (accessed August 2, 2007).
48. Ridgway, J., and S. McCusker. 2003. Using computers to assess new educational goals. Assessment in Education: Principles, Policy and Practice 10, no. 3: 309-28.
49. Ridgway, J., S. McCusker, and D. Pead. 2004. Literature review of e-assessment. http://www.nestafuturelab.org/research/reviews/10_01.htm (accessed August 2, 2007).
50. Ripley, M. 2007. E-assessment - An update on research, policy and practice. http://www.futurelab.org.uk/resources/documents/lit_reviews/Assessment_Review_update.pdf (accessed August 2, 2007).
51. Ruiz-Primo, M.A., R. Shavelson, and S. Elise Shultz. 1997. On the validity of concept map-based assessment interpretations: An experiment testing the assumption of hierarchical concept maps in science (CSE Technical Report 455). http://www.cse.ucla.edu/products/Reports/TECH455.pdf (accessed August 2, 2007).
52. Ruiz-Primo, M.A., M. Li, Y. Yin, R.J. Shavelson, J. Vanides, S. Schultz, and C. Ayala. 2004. Concept maps as an assessment tool: A framework for examining their cognitive validity. Paper presented at the American Educational Research Association (AERA) Annual Meeting, April, in San Diego, CA.
53. Sainsbury, M. 2002. Test development research: A short guide, TOPIC 25, no. 9.
54. Scalise, K., and B. Gifford. 2006. Computer-Based Assessment in e-learning: A framework for constructing 'intermediate constraint' questions and tasks for technology platforms. Journal of Technology, Learning and Assessment 4, no. 6. http://escholarship.bc.edu/cgi/viewcontent.cgi?article=1036&context=jtla (accessed August 2, 2007).
55. Shavelson, R., H. Lang, and B. Lewin. (1994) On concept maps as potential 'authentic' assessments in science (CSE Technical Report 388). http://www.cse.ucla.edu/products/Reports/TECH388.pdf (accessed August 2, 2007).
56. Shermis, M.D., and J. Burstein, eds. 2003. Automated essay scoring: A cross-disciplinary perspective. Hillsdale, NJ: Lawrence Erlbaum.
57. Sim, G., P. Holifield, and M. Brown. 2004. Implementation of computer assisted assessment: Lessons from the literature. Association for Learning Technology Journal 12, no. 3: 215-29.
58. Snow, R.E., and D.F. Lohman. 1993. Cognitive psychology, new test design and new test theory: An introduction. In Test theory for a new generation of tests, ed. N. Frederiksen, R.J. Mislevy, and I.I. Bejar, 1-17. Hillsdale, NJ: Lawrence Erlbaum.
59. Sturman, L., and A. Kispal. 2003. To e or not to e? A comparison of electronic marking and paperbased marking. Paper presented at the 29th International Association for Educational Assessment Conference, October 5-10, in Manchester. http://www.aqa.org.uk/support/iaea/papers/STURMAN-KISPAL.PDF (accessed August 2, 2007).
60. Threlfall, J., and N. Nelson. 2006. A taxonomy of sources of difficulty in the assessment of ICT. Paper presented at the Association for Educational Assessment - Europe (AEA-E) Conference, November 9-11, in Naples. http://www.aea-europe.net/userfiles/C3%20John%20Threlfall%20&%20Nick%20Nelson.pdf (accessed August 2, 2007).
61. Wainer, H. 2000. CATs: Whither and whence. Psicologica 21, no. 1-2: 121-33.
62. Warburton, B. 2006. Quick win or slow burn? Modelling UK HE CAA uptake. In Proceedings of the 10th International Computer Assisted Assessment (CAA) Conference, ed. M. Danson. http://www.caaconference.com (accessed August 2, 2007).
63. Working Group on 14-19 Reform. 2004. 14-19 Curriculum and qualifications reform: Final report of the Working Group on 14-19 Reform. Nottinghamshire: DfES.
64. World Class Arena. 2004. World class arena: Challenging minds, recognising talent. http://www.worldclassarena.org/v5/default.htm (accessed August 2, 2007).
65. Zenisky, A.L., and S.G. Sireci. 2002. Technological innovations in large-scale assessment. Applied Measurement in Education 15, no. 4: 337-62.