Evaluative misalignment of 10th-grade student and teacher criteria for essay quality: An automated textual analysis

Journal of Writing Research

Volumn 5, Issue 1, 2013, Pages 35-59

  Varner, Laura K ^a   Roscoe, Rod D ^a   McNamara, Danielle S ^a  

^a ARIZONA STATE UNIVERSITY   (United States)

Author keywords

Computational linguistics; Self assessment; Teacher essay evaluation; Textual analysis; Writing assessment

Indexed keywords

EID: 84881536335     PISSN: 20301006     EISSN: 22943307     Source Type: Journal    
DOI: 10.17239/jowr-2013.05.01.2     Document Type: Article

References (48)

1. Andrade, H., & du Boulay, B. (2003). The role of rubric-referenced self-assessment in learning to write. Journal of Educational Research, 97, 21-34.
2. Andrade, H., Du, Y., & Wang, X. (2008). Putting rubrics to the test: The effect of a model, criteria generation, and rubric-referenced self-assessment on elementary school students' writing. Educational Measurement: Issues and Practices, 27, 3-13.
3. Camara, W. J. (2003). Scoring the essay on the SAT writing section. New York: College Entrance Examination Board.
4. Cho, K., & Schunn, C. D. (2007). Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system. Computers and Education, 48, 409-426.
5. Crossley, S. A. & McNamara, D. S. (2010). Cohesion, coherence, and expert evaluations of writing proficiency. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 984-989). Austin, TX: Cognitive Science Society.
6. Crossley, S. A., & McNamara, D. S. (2011). Understanding expert ratings of essay quality: Coh- Metrix analyses of first and second language writing. International Journal of Continuing Engineering Education and Life-Long Learning, 21, 170-191.
7. Crossley, S. A., & McNamara, D. S. (2012). Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35, 115-136.
8. Crossley, S. A., Weston, J., McLain-Sullivan, S. T., & McNamara, D. S. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28, 282-311.
9. Donovan, C. A., & Smolkin, L. B. (2006). Children's understanding of genre and writing development. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 131-143). New York: Guilford.
10. Dunning, D., Johnson, K., Ehrlinger, J., & Kruger, J. (2003). Why people fail to recognize their own incompetence. Current Directions in Psychological Science, 12, 83-86.
11. Falchikov, N., & Boud, D. (1989). Student self-assessment in higher education: A meta-analysis. Review of Educational Research, 59, 395-430.
12. Fellbaum, C. (Ed.). (1998). WordNet: An electronic lexical database. Cambridge, MA: MIT Press.
13. Gesier, S., & Studley, R. (2001). UC and SAT: Predictive validity and differential impact of the SAT I and SAT II at the University of California. Oakland, CA: University of California.
14. Gill, A. J., French, R. M., Gergle, D., & Oberlander, J. (2008). The language of emotion in short blog texts. Proceedings of CSCW (pp. 299-302). New York, NY: ACM Press.
15. Graesser, A. C. & McNamara, D. S. (2011). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science, 2, 371-398.
16. Graesser, A. C., McNamara, D. S., Louwerse, M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36, 193-202.
17. Graham, S. (2006). Strategy instruction and the teaching of writing: A meta-analysis. In C. MacArthur, S. Graham, & J. Fitzgerald. Handbook of writing research (pp. 187-207). New York: Guilford Press.
18. Graham, S. & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99, 445-476.
19. Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.
20. Hancock, J. T., Curry, L. E., Goorha, S., & Woodworth, M. (2008). On lying and being lied to: A linguistic analysis of deception in computer-mediated communication. Discourse Processes, 45, 1-23.
21. Hancock, J. T., Landrigan, C., & Silver, C. (2007). Expressing emotion in text-based communication. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 929-932). New York, NY: ACM.
22. Hillocks, G. (1984). What works in teaching composition: A meta-analysis of experimental treatment studies. American Journal of Education, 93, 133-170.
23. Hillocks, G. (1986). Research on written composition: New directions for teaching. Urbana, IL: Eric Clearinghouse on Reading and Communication Skills.
24. Kos, R., & Maslowski, C. (2001). Second graders' perceptions of what is important in writing. The Elementary School Journal, 101, 567-585.
25. Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121-1134.
26. Landauer, T. K., McNamara, D. S., Dennis, S., & Kintsch, W. (2007). LSA: A road to meaning. Mahwah, NJ: Erlbaum.
27. Light, R. J. (2001). Making the most of college: Students speaking their minds. Cambridge: Harvard University Press.
28. Lin, S., Monroe, B., & Troia, G. A. (2007). Development of writing knowledge in grades 2-8: A comparison of typically developing writers and their struggling peers. Reading and Writing Quarterly, 23, 207-230.
29. Lipstein, R., & Renninger, K. A. (2006). "Putting things into words": 12-15 year-old students' interest for writing. In P. Boscolo & S. Hidi. Motivation and writing: Research and school practice. New York: Kluwer Academic/Plenum.
30. Louwerse, M. M. (2001). An analytic and cognitive parameterization of coherence relations. Cognitive Linguistics, 12, 291-315.
31. McNamara, D. S., Cai, Z., & Louwerse, M. M. (2007). Optimizing LSA measures of cohesion. In T. K. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch. Handbook of latent semantic analysis (pp. 379-400). Mahwah, NJ: Erlbaum.
32. McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27, 57-86.
33. McNamara, D. S., & Graesser, A. C. (2012). Coh-Metrix: An automated tool for theoretical and applied natural language processing. In P. M. McCarthy & C. Boonthum-Denecke. Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 188-205). Hershey, PA: IGI Global.
34. McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47, 292-330.
35. McNamara, D. S., & Magliano, J.P. (2009). Self-explanation and metacognition: the dynamics of reading. In D.J. Hacker, J. Dunlosky, & A.C. Graesser. Handbook of metacognition in education (pp. 60-81). New York: Routledge.
36. National Assessment of Educational Progress. (2007). The Nation's Report Card: Writing 2007. Retrieved Nov. 5, 2010, nces.ed.gov/nationsreportcard/writing/
37. Nelson, M. M., & Schunn, C. D. (2009). The nature of feedback: How different types of peer feedback affect writing performance. Instructional Science, 37, 375-401.
38. Pennebaker, J. W., Booth, R. J., & Francis, M. E. (2007). Linguistic Inquiry and Word Count: LIWC [computer software]. Austin, TX.
39. Pennebaker, J. W., Mayne, T. J., & Francis, M. E. (1997). Linguistic predictors of adaptive bereavement. Journal of Personality and Social Psychology, 72, 863-871.
40. Powell, P. (2009). Retention and writing instruction: Implications for access and pedagogy. College Composition and Communication, 60, 664-682.
41. Rogers, L. A., & Graham, S. (2008). A meta-analysis of single subject design writing intervention research. Journal of Educational Psychology, 100, 879-906.
42. Roscoe, R. D., Crossley, S. A., Weston, J. L., & McNamara, D. S. (2011). Automated assessment of paragraph quality: Introductions, body, and conclusion paragraphs. In R. C. Murray & P. M. McCarthy (Eds.), Proceedings of the 24th International Florida Artificial Intelligence Research Society Conference. (pp. 281-286). Menlo Park, CA: AAAI Press.
43. Ross, J. A., Rolheiser, C., & Hogaboam-Gray, A. (1999). Effects of self-evaluation training on narrative writing. Assessing Writing, 6, 107-132.
44. Rus, V., Lintean, M., Graesser, A. C., & McNamara, D. S. (2009). Assessing student paraphrases using lexical semantics and word weighting. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. C. Graesser (Eds.), Artificial intelligence in education: Building learning systems that care; From knowledge representation to affective modeling (pp. 165-172). Amsterdam, The Netherlands: IOS Press.
45. Schoonen, R., & De Glopper, K. D. (1996). Writing performance and knowledge about writing. In G. Rijlaarsdam, H. Van Den Bergh, & M. Couzijn (Eds.), Theories, models and methodology in writing research (pp. 87-107). Amsterdam: Amsterdam University Press.
46. Tousignant, M., & DesMarchais, J. E. (2002). Accuracy of student self-assessment ability compared to their own performance in a problem-based learning medical program: A correlation study. Advances in Health Sciences Education, 7, 19-27.
47. Wilson, M. D. (1988). The MRC Psycholinguistic Database: Machine Readable Dictionary, Version 2. Behavior Research Methods, Instruments, & Computers, 20, 6-11.
48. Wong, B. (1999). Metacognition in writing. In R. Gallimore, L. P. Bernheimer, D. L. MacMillan, D. L. Speech, & S. Vaughn (Eds.), Developmental perspectives on children with high-incidence disabilities (pp. 183-198). Mahwah, NJ: Erlbaum.