Automated scoring in context: Rapid assessment for placed students

Assessing Writing

Volumn 18, Issue 1, 2013, Pages 62-84

  Klobucar, Andrew ^a   Elliot, Norbert ^a   Deess, Perry ^a   Rudniy, Oleksandr ^a   Joshi, Kamal ^a  

^a NEW JERSEY INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Automated essay scoring (AES); Validation methods; Writing assessment; Writing placement

Indexed keywords

EID: 84872395238     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2012.10.001     Document Type: Article

References (73)

1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education Standards for educational and psychological testing 1999, American Educational Research Association, Washington, DC.
2. Attali Y., Burstein J. Automated essay scoring with e-rater v.2.0. Journal of Technology, Learning, and Assessment 2006, 4(3):1-30. Retrieved from. http://ejournals.bc.edu/ojs/index.php/jtla/index.
3. Bailey T., Wook D., Jeong D.W., Cho S.W. Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review 2010, 29:255-270. 10.1016/j.econedurev.2009.09.002.
4. Bennett R.E. On the meanings of constructed response. Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment 1993, 1-27. Erlbaum, Hillsdale, NJ. R.E. Bennett, W.C. Ward (Eds.).
5. Bennett R.E. Automated scoring of constructed-response literacy and mathematics items 2011, Arabella Philanthropic Investment Advisors, Washington, DC.
6. Bowen W.G., Chingos M.M., McPherson M.S. Crossing the finish line: Completing college at America's public universities 2009, Princeton University Press, Princeton, NJ.
7. Brent E., Townsend M. Automated essay grading in the sociology classroom: Finding common ground. Machine scoring of student essays: Truth and consequences 2006, 177-198. Utah State UP, Logan, UT. P. Ericcson, R. Haswell (Eds.).
8. Bridgeman B., Tripani C., Attali Y. Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education 2012, 25:27-40. doi:10.1080/08957347.2012.635502.
9. Bridgeman, B., Trapani, C., & Williamson, D. M. (2011, April). The question of validity of automated essay scores and differentially valued evidence. Paper presented at the meeting of the National Council on Measurement in Education, New Orleans, LA.
10. Burstein J., Chodorow M. Progress and new directions in technology for automated essay evaluation. The Oxford handbook of applied linguistics 2010, 529-538. Oxford University Press, Oxford, UK. 2nd ed. R. Kaplan (Ed.).
11. Burstein J., Marcu D., Knight K. Finding the WRITE stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems: Special Issue on Advances in Natural Language Processing 2003, 18:32-39.
12. Camili G. Test fairness. Educational measurement 2006, 221-256. American Council on Education/Praeger, Westport, CT. 4th ed. R.L. Brennan (Ed.).
13. Cizek G.J., Bunch M.B. Standard setting: A guide to establishing and evaluating performance standards on tests 2007, Sage Press, Thousand Oaks, CA.
14. State profile report: New Jersey 2011, College Board, College Board, New York, NY, Retrieved from. http://professionals.collegeboard.com/profdownload/NJ_11_03_03_01.pdf.
15. Collins R., Elliot N., Klobucar A., Deek F.P. Web-based portfolio assessment: Validation of an open source platform. Journal of Interactive Learning Research 2013, 24:417-444.
16. Complete College America Remediation: Higher education's bridge to nowhere 2012, Complete College America, Washington, DC, Retrieved from. http://www.completecollege.org/.
17. Condon W. Large-scale assessment, locally-developed measures, and automated scoring of essays: Fishing for red herrings?. Assessing Writing 2013, 18:100-108. 10.1016/j.asw.2012.10.002.
18. Council of Writing Program Administrators, National Council of Teachers of English, National Writing Project Framework for success in postsecondary writing 2011, Retrieved from. http://wpacouncil.org/files/framework-for-success-postsecondary-writing.pdf.
19. Craven P. Editorial. International Journal of e-Assessment 2011, 1. Retrieved from. http://journals.sfu.ca/ijea/index.php/journal/article/viewFile/5/4.
20. Cronbach L.J. Aptitudes and instructional methods: A handbook for research on interactions 1971, Irvington Publishers, New York, NY.
21. Deane, P. (2011). Writing assessment and cognition (ETS RR-11-14). Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/research/policy_research_reports/ets.
22. Deane P., Quinlan T. What automated analyses of corpora can tell us about students' writing skills. Journal of Writing Research 2010, 2:151-177.
23. Elliot N., Briller V., Joshi K. Portfolio assessment: Quantification and community. Journal of Writing Assessment 2007, 3:5-30. Retrieved from. http://www.journalofwritingassessment.org/archives/3-1.2.pdf.
24. Elliot N., Deess P., Rudniy A., Joshi K. Placement of students into first-year writing courses. Research in the Teaching of English 2012, 46:285-313.
25. Forgione P.D. Coming together to raise achievement: New assessments for the Common Core State Standards 2012, Center for K-12 Assessment & Performance Management, Princeton, NJ, Retrieved from. http://www.k12center.org/rsc/pdf/Assessments_for_the_Common_Core_Standards.pdf.
26. Fowles M. Writing assessment for admission to graduate and professional programs: Lessons learned and a note for the future. Writing assessment in the 21st century: Essays in honor of Edward M. White 2012, 136-146. Hampton Press, New York, NY. N. Elliot, L. Perelman (Eds.).
27. Graham S., Perrin D. A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology 2007, 99:446-467. 10.1037/a0029185.
28. Grimes D., Warschauer M. Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment 2010, 8(6):1-43. Retrieved from. http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1625/1469.
29. Hambleton R.K., Pitoniak M.M. Setting performance standards. Educational measurement 2005, 433-470. American Council on Education/Praeger, Westport, CT. 4th ed. R.L. Brennan (Ed.).
30. Hamp-Lyons L. Scoring procedures for ESL contexts. Assessing second language writing in academic contexts 1991, 241-276. Ablex, Norwood, NJ. L. Hamp-Lyons (Ed.).
31. Haswell R. Teaching of writing in higher education. Handbook on writing research: History, society, school, individual, text 2008, 332-346. Lawrence Erlbaum, New York, NY. C. Bazerman (Ed.).
32. SM: Promoting the standard. Race and writing assessment 2012, 47-61. Peter Lang, New York, NY. A.B. Inoue, M. Poe (Eds.).
33. Hillocks G. Research on written composition: New directions for teaching 1986, NCTE, Urbana, IL.
34. Hussar, W. J., & Bailey, T. M. (2011). Projections of education statistics to 2020 (NCES 2011-026). U.S. Department of Education, National Center for Education Statistics. Washington, DC: U.S. Government Printing Office. http://nces.ed.gov/pubs2011/2011026.pdf.
35. Inoue, A.B., & Poe, M. (Eds.). (2012a). Introduction. In: Race and writing assessment (pp. 1-11). New York, NY: Peter Lang.
36. Inoue A.B., Poe M. Racial formations in two writing assessments: Revisiting White and Thomas's findings on the English Placement Test after 30 years. Writing assessment in the 21st century: Essays in honor of Edward M. White 2012, 341-359. Hampton Press, New York, NY. N. Elliot, L. Perelman (Eds.).
37. Kane M.T. Current concerns in validity theory. Journal of Educational Measurement 2001, 38:319-342. 10.1111/j.1745-3984.2001.tb01130.x.
38. Kane M.T. Validation. Educational measurement 2006, 17-64. American Council on Education/Praeger, Westport, CT. 4th ed. R.L. Brennan (Ed.).
39. Klobucar A., Deane P., Elliot N., Ramineni C., Deess P., Rudniy A. Automated essay scoring and the search for valid writing assessment. International advances in writing research: Cultures, places, measures 2012, 103-119. WAC Clearinghouse/Parlor Press, Fort Collins, Colorado/Anderson, SC. http://wac.colostate.edu/books/wrab2011/chapter6.pdf, C. Bazerman, C. Dean, J. Early, K. Lunsford, S. Null, P. Rogers, A. Stansell (Eds.).
40. Kobrin J.L., Deng H., Shaw E.J. The association between SAT prompt characteristics, response features, and essay scores. Assessing Writing 2011, 16:154-169. 10.1016/j.asw.2011.01.001.
41. Kobrin, J. L., Patterson, B. F., Shaw, E. J, Mattern, K. D., & Barbuti, S. M. (2008). Validity of the SAT® for predicting first-year college grade point average (College Board Report 2008-5). New York: College Board. Retrieved from http://professionals.collegeboard.com/.
42. Leacock C., Chodorow M., Gamon M., Tetreault J. Automated grammatical error detection for language learners 2010, Morgan and Claypool, San Rafael, CA.
43. Lipnevich, A. A., & Smith, J. K. (2008). Response to assessment feedback: The effects of grades, praise, and source of information. ETS Research Report (ETS RR-08-30). Princeton, NJ: Educational Testing Service. http://www.ets.org/Media/Research/pdf/RR-08-30.pdf.
44. Livingston S.A. Constructed-response test questions: Why we use them; How we score them. ETS R & D Connections 2009, September, 11. http://www.ets.org/Media/Research/pdf/RD_Connections11.pdf.
45. Mattern, K, Camara, W., & Kobrin, J. L. (2007). SAT Writing®: An overview of research and psychometrics to date (RN-32). New York: College Board. Retrieved from https://professionals.collegeboard.com/data-reports-research/cb/sat-writing-overview-research.
46. Mattern K., Kobrin J., Patterson B., Shaw E., Camara W. Validity is in the eye of the beholder: Conveying SAT research findings to the general public. The concept of validity: Revisions, new directions, and applications 2009, 213-240. Information Age Publishing, Charlotte, NC. R.W. Lissitz (Ed.).
47. Maxwell S.E. Sample size and multiple regression analysis. Psychological Methods 2000, 5:434-458. 10.1037/1082-989X.5.4.434.
48. Mazzeo J., Lazer S., Zieky M. Monitoring educational progress with group score assessments. Educational measurement 2006, 681-699. American Council on Education/Praeger, Westport, CT. 4th ed. R.L. Brennan (Ed.).
49. Mislevy R.J. Cognitive psychology and educational assessment. Educational measurement 2006, 257-305. American Council on Education/Praeger, Westport, CT. 4th ed. R.L. Brennan (Ed.).
50. National Governors Association Common core state standards initiative 2012, National Governors Association, Washington, DC, Retrieved from http://www.corestandards.org.
51. Odendahl, N.V. (2011). Testwise: Understanding educational assessment. Vol. 1. Lanham, MD: Rowman and Littlefield.
52. Organization for Economic Co-operation and Development International migration outlook 2012 2012, OECD Publishing, Paris, FR.
53. Page E.B. The imminence of grading essays by computer. Phi Delta Kappan 1966, 48:238-243.
54. Perelman L. Construct validity, length, score, and time in holistically graded writing assessments: The case against automated essay scoring (AES). International advances in writing research: Cultures, places, measures 2012, 121-131. WAC Clearinghouse/Parlor Press, Fort Collins, Colorado/Anderson, SC. http://wac.colostate.edu/books/wrab2011/chapter7.pdf, C. Bazerman, C. Dean, J. Early, K. Lunsford, S. Null, P. Rogers, A. Stansell (Eds.).
55. Plakans L. Comparing composing processes in writing-only and reading-to-write test tasks. Assessing Writing 2008, 13:111-112. 10.1016/j.asw.2008.07.001.
56. Powers, D. E., Burstein, J. Chodorow, M., Fowles, M. E., & Kukich, K. (2000). Comparing the validity of automated and human essay scoring (GRE Board Research Rep. No. 98-08a, ETS RR-00-10). Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/research/policy_research_reports/ets.
57. Quinlan, T., Higgins, D., & Wolf, S. (2009). Evaluating the construct coverage of the e-rater scoring engine (ETS RR-09-01). Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/research/policy_research_reports/ets.
58. Ramineni C. Automated essay scoring for online writing placement: A case study. Assessing Writing 2013, 18:40-61. 10.1016/j.asw.2012.10.002.
59. Ramineni C., Williamson D. Automated essay scoring: Psychometric guidelines and practices. Assessing Writing 2013, 18:25-39. 10.1016/j.asw.2012.10.002.
60. Scott-Clayton, J. (2012). Do high-stakes placement exams predict college success? (CCRC Working Paper 41). New York, NY: Community College Research Center. Retrieved from http://ccrc.tc.columbia.edu/Home.asp.
61. Shermis, M.D. & Burstein, J. (Eds.). (2003). Introduction. Automated essay scoring: A cross-disciplinary perspective (pp. xiii-xvi). Mahwah, NJ: Lawrence Erlbaum Associates.
62. Handbook of automated essay evaluation: Current applications and new directions. 2013, Routledge, New York, NY. M.D. Shermis, J. Burstein (Eds.).
63. Shermis, M. D., & Hamner, B. (2012). Contrasting state-of-the-art automated scoring of essays: Analysis. Paper presented at the annual meeting of the National Council of Measurement in Education, Vancouver, BC, Canada. Retrieved from http://www.scoreright.org/NCME_2012_Paper3_29_12.pdf.
64. Smith C.S. Modes of discourse: The local structure of texts 2003, Cambridge University Press, Cambridge, UK.
65. Stufflebeam D.L. Evaluation models. New Directions for Evaluation 2001, 89:7-98.
66. Su K.Y. Preface: General chair. Joint conference of the 47th annual meeting of the Association for Computational Linguistics and 4th international joint conference on Natural Language Processing of the AFNLP: Proceedings 2009, iii. Retrieved from. http://www.aclweb.org/anthology-new/P/P09/P09-1.pdf, J. Su, J. Wiebe (Eds.).
67. Venezia A., Bracco K.R., Nodine T. One shot deal? Students' perceptions of assessment and course placement in California's community colleges 2010, WestEd, San Francisco, CA, Retrieved from. http://www.wested.org/online_pubs/OneShotDeal.pdf.
68. White E.M. Holisticism. College Composition and Communication 1984, 35:400-409.
69. Williamson M. Validity of automated scoring: Prologue for a continuing discussion of machine scoring student writing. Journal of Writing Assessment 2003, 1:85-104. Retrieved from. http://www.journalofwritingassessment.org/.
70. Williamson D.D., Xi X., Breyer F.J. A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practices 2012, 31:2-13.
71. Willingham W.W., Cole N.S., Lewis C., Leung S.W. Test performance. Gender and fair assessment 1997, 55-126. Lawrence Erlbaum, Mahwah, NJ. W.W. Willingham, N.S. Cole (Eds.).
72. Willingham W.W., Pollack J.M., Lewis C. Grades and test scores: Accounting for observed differences. Journal of Educational Measurement 2002, 39:1-97.
73. Yang Y., Buckendahl C.W., Juszkiewicz P.J., Bhola D.S. A review of strategies for validating computer automated scoring. Applied Measurement in Education 2002, 15:391-412.