Writing pal: Feasibility of an intelligent writing strategy tutor in the high school classroom

Journal of Educational Psychology

Volumn 105, Issue 4, 2013, Pages 1010-1025

  Roscoe, Rod D ^a   McNamara, Danielle S ^a  

^a ARIZONA STATE UNIVERSITY   (United States)

Author keywords

Ill defined learning domains; Intelligent tutoring systems; Usability and feasibility testing; Writing instruction

Indexed keywords

EID: 84887543012     PISSN: 00220663     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0032340     Document Type: Article

References (55)

1. Aleven, V., & Koedinger, K. (2002). An effective metacognitive strategy:Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26, 147-179. doi:10.1207/ s15516709cog2602_1
2. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V. 2. Journal of Technology, Learning, and Assessment, 4(3). Retrieved from http://www.jtla.org
3. Beal, C., Arroyo, I., Cohen, P., & Woolf, B. (2010). Evaluation of AnimalWatch:In intelligent tutoring system for arithmetic and fractions. Journal of Interactive Online Learning, 9, 64-77.
4. Bell, C., & McNamara, D. (2007). Integrating iSTART into a high school curriculum. Proceedings of the 29th annual meeting of the Cognitive Science Society (pp. 809-814). Austin, TX: Cognitive Science Society.
5. Burstein, J., Chodorow, M., & Leacock, C. (2004). Automated essay evaluation: The Criterion online writing system. AI Magazine, 25, 27-36.
6. Caccamise, D., Franzke, M., Eckhoff, A., Kintsch, E., & Kintsch, W. (2007). Guided practice in technology-based summary writing. In D. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 375-396). Mahwah, NJ: Erlbaum.
7. Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29, 241-250. doi:10.1080/ 1358165042000283101
8. Crossley, S., & McNamara, D. S. (2011). Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing. International Journal of Continuing Engineering Education and Life-Long Learning, 21, 170-191. doi:10.1504/IJCEELL.2011.040197
9. Crossley, S., Roscoe, R., Graesser, A., & McNamara, D. (2011). Predicting human scores of essay quality using computational indices of linguistic and textual features. Proceedings of the 15th international conference on artificial intelligence in education (pp. 438-440). Auckland, New Zealand:AIED.
10. Crossley, S., Weston, J., McLain Sullivan, S., & McNamara, D. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28, 282-311. doi:10.1177/ 0741088311410188
11. Deane, P., Odendahl, N., Quinlan, T., Fowles, M., Welsh, C., & Bivens-Tatum, J. (2008). Cognitive models of writing: Writing proficiency as a complex integrated skill (Research Report No. RR-08-55). Princeton, NJ: Educational Testing Service.
12. De la Paz, S., & Graham, S. (2002). Explicitly teaching strategies, skills, and knowledge: Writing instruction in middle school classrooms. Journal of Educational Psychology, 94, 687-698. doi:10.1037/0022-0663.94.4.687
13. Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5(1), Retrieved from http://www.jtla.org
14. Flower, L., & Hayes, J. (1981). A cognitive process theory of writing. College Composition and Communication, 32, 365-387. doi:10.2307/ 356600
15. Gamper, J., & Knapp, J. (2002). A review of intelligent CALL systems. Computer Assisted Language Learning, 15, 329-342. doi:10.1076/call .15.4.329.8270
16. Graesser, A., Lu, S., Jackson, G., Mitchell, H., Ventura, M., Olney, A., & Louwerse, M. (2004). AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments & Computers, 36, 180-192. doi:10.3758/BF03195563
17. Graesser, A., & McNamara, D. (2012). Use of computers to analyze and score essays and open-ended verbal responses. In H. Cooper, P. Camic, R. Gonzalez, D. Long, & A. Panter (Eds.), APA handbook of research methods in psychology (pp. 307-325). Washington, DC: American Psychological Association.
18. Graesser, A., McNamara, D., & VanLehn, K. (2005). Scaffolding deep comprehension strategies through Point & Query, AutoTutor, and iSTART. Educational Psychologist, 40, 225-234. doi:10.1207/ s15326985ep4004_4
19. Graham, S., McKeown, D., Kiuhara, S., & Harris, K. (2012). A metaanalysis of writing instruction for students in the elementary grades. Journal of Educational Psychology, 104, 879-896. doi:10.1037/a0029185
20. Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99, 445-476. doi:10.1037/0022-0663.99.3.445
21. Grimes, D., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8, 4-43.
22. Hausmann, R., VanLehn, K., Nokes, T., & Gershman, S. (2009). The design of self-explanation prompts: The fit hypothesis. Paper presented at the 31st annual meeting of the Cognitive Sciences Society, Amsterdam, the Netherlands.
23. Hillocks, G. (1984). What works in teaching composition: A meta-analysis of experimental treatment studies. American Journal of Education, 93, 133-170. doi:10.1086/443789
24. Huot, B. (1996). Toward a new theory of writing assessment. College Composition and Communication, 47, 549-566. doi:10.2307/358601
25. Jackson, G., & McNamara, D. (2013). Motivation and performance in a game-based intelligent tutoring system. Journal of Educational Psychology, 105, 1036-1049. doi:10.1037/a0032580
26. Johnson, W., & Wu, S. (2008). Assessing aptitude for learning with a serious game for foreign language and culture. In B. Woolf, E. Aimeur, R. Nkambo, & S. Lajoie (Eds.), Intelligent tutoring systems (pp. 520-529). Berlin, Germany: Springer-Verlag. doi:10.1007/978-3-540-69132-7_55
27. Kellogg, R., Whiteford, A., & Quinlan, T. (2010). Does automated feedback help students learn to write? Journal of Educational Computing Research, 42, 173-196. doi:10.2190/EC.42.2.c
28. Kintsch, E., Caccamise, D., Franzke, M., Johnson, N., & Dooley, S. (2007). Summary Street®: Computer-guided summary writing. In T. K. Landauer, D. M. McNamara, S. Dennis, & W. Kintsch (Eds.), Latent semantic analysis (pp. 263-277). Mahwah, NJ: Erlbaum.
29. Landauer, T., Laham, D., & Foltz, P. (2003). Automatic essay assessment. Assessment in Education: Principles, Policy & Practice, 10, 295-308. doi:10.1080/0969594032000148154
30. Landauer, T., Lochbaum, K., & Dooley, S. (2009). A new formative assessment technology for reading and writing. Theory Into Practice, 48, 44-52. doi:10.1080/00405840802577593
31. Lynch, C., Ashley, K., Pinkwart, N., & Aleven, V. (2009). Concepts, structures, and goals: Redefining ill-definedness. International Journal of Artificial Intelligence in Education, 19, 253-266.
32. Malvern, D., Richards, B., Chipere, N., & Durán, P. (2004). Lexical diversity and language development: Quantification and assessment. Basingstoke, England: Palgrave. doi:10.1057/9780230511804
33. McGarrell, H., & Verbeem, J. (2007). Motivating revision of drafts through formative feedback. ELT Journal, 61, 228-236. doi:10.1093/elt/ccm030
34. McNamara, D., Crossley, S., & McCarthy, P. (2010). Linguistic features of writing quality. Written Communication, 27, 57-86. doi:10.1177/ 0741088309351547
35. McNamara, D., Crossley, S., & Roscoe, R. (2012). Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-012-0258-1
36. McNamara, D. S., O'Reilly, T., Best, R., & Ozuru, Y. (2006). Improving adolescent students' reading comprehension with iSTART. Journal of Educational Computing Research, 34, 147-171. doi:10.2190/1RU5-HDTJ-A5C8-JVWE
37. McNamara, D., Raine, R., Roscoe, R., Crossley, S., Dai, J., Cai, Z.,.. Graesser, A. (2011). The Writing-Pal: Natural language algorithms to support intelligent tutoring on writing strategies. In P. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 298-311) Hershey, PA: IGI Global.
38. Meadows, M., & Billington, L. (2005). A review of the literature on marking reliability Retrieved from AQA Centre for Education Research and Policy website: https://orderline.education.gov.uk/gempdf/1849625344/QCDA104983_review_of_the_literature_on_marking_reliability.pdf
39. Michael, J., Rovick, A., Glass, M., Zhou, Y., & Evens, M. (2003). Learning from a computer tutor with natural language capabilities. Interactive Learning Environments, 11, 233-262. doi:10.1076/ilee.11.3.233.16543
40. Pajares, F. (2003). Self-efficacy beliefs, motivation, and achievement in writing: A review of the literature. Reading & Writing Quarterly:Overcoming Learning Difficulties, 19, 139-158. doi:10.1080/ 10573560308222
41. Proske, A., Narciss, S., & McNamara, D. (2012). Computer-based scaffolding to facilitate students' development of expertise in academic writing. Journal of Research in Reading, 35, 136-152. doi:10.1111/j .1467-9817.2010.01450.x
42. Rock, J. (2007). The impact of short-term use of Criterion on writing skills in 9th grade (Research Report no. RR-07-07). Princeton, NJ: Educational Testing Service.
43. Roscoe, R., Varner, L., Cai, Z., Weston, J., Crossley, S., & McNamara, D. (2011). Internal usability testing of automated essay feedback in an intelligent writing tutor. In R. Murray & P. McCarthy (Eds.), Proceedings of the 24th international Florida Artificial Intelligence Research Society conference (pp. 543-548). Menlo Park, CA: AAAI Press.
44. Roscoe, R., Varner, L., Weston, J., Crossley, S., & McNamara, D. (in press). The Writing Pal Intelligent Tutoring System: Usability testing and development. Computers and Composition.
45. Rowley, K., & Meyer, N. (2003). The effect of a computer tutor for writers on student writing achievement. Journal of Educational Computing Research, 29, 169-187. doi:10.2190/3WVD-BKEY-PK0D-TTR7
46. Rudner, L., Garcia, V., & Welch, C. (2006). An evaluation of the Intelli-Metric essay scoring system. Journal of Technology, Learning, and Assessment, 4, 3-21.
47. Shank, R., & Neeman, A. (2001). Motivation and failure in educational systems design. In K. Forbus & P. Feltovich (Eds.), Smart machines in education (pp. 37-69). Cambridge, MA: AAAI Press/MIT Press.
48. Shermis, M., & Burstein, J. (Eds.). (2003). Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Erlbaum.
49. Shermis, M., Burstein, J., & Bliss, L. (2004). The impact of automated essay scoring on high stakes writing assessments. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
50. Shute, V. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189. doi:10.3102/0034654307313795
51. Simon, H. (1973). The structure of ill structured problems. Artificial Intelligence, 4, 181-201. doi:10.1016/0004-3702(73)90011-8
52. VanLehn, K., Lynch, C., Schulze, K., Shapiro, J., Shelby, R., Taylor, L., .. Wintersgill, M. (2005). The Andes Physics Tutoring System:Lessons learned. International Journal of Artificial Intelligence in Education, 15, 147-204.
53. Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10, 157-180. doi:10.1191/1362168806lr190oa
54. Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal, 3, 22-36.
55. Wolfe, C., Britt, M., Petrovic, M., Albrecht, M., & Kopp, K. (2009). The efficacy of a web-based counterargument tutor. Behavior Research Methods, 41, 691-698. doi:10.3758/BRM.41.3.691