Computational assessment of text readability: A survey of current and future research

ITL - International Journal of Applied Linguistics (Belgium)

Volumn 165, Issue 2, 2014, Pages 97-135

  Collins Thompson, Kevyn ^a  

^a UNIVERSITY OF MICHIGAN   (United States)

Author keywords

Computational linguistics; Machine learning; Readability; Reading difficulty; Text complexity

Indexed keywords

EID: 85013118583     PISSN: 00190829     EISSN: 17831490     Source Type: Journal    
DOI: 10.1075/itl.165.2.01col     Document Type: Article

References (97)

1. Abedi, J., Leon, S., Kao, J., Bayley, R., Ewers, N., Herman, J., & Mundhenk, K. (2011). Accessible reading assessments for students with disabilities: The role of cognitive, grammatical, lexical, and textual/visual features. CRESST Report #785. Univ. of California, Los Angeles. Jan 2011. http://www.cse.ucla.edu/products/reports/R785.pdf
2. Agrawal, R., Gollapudi, S., Kannan, A., & Kenthapadi, K. (2011). Identifying enrichment candidates in textbooks. Proceedings of the 20th International Conference on World Wide Web (WWW’11) (pp. 483–492). ACM, New York, NY, USA.
3. Akamatsu, K., Pattanasri, N., Jatowt, A., & Tanaka, K. (2011). Measuring comprehensibility of web pages based on link analysis. Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology (Vol. 1, pp. 40–46).
4. Al-Khalifa, H. S., & Al-Ajlan, A. A. (2010). Automatic readability measurements of the Arabic text: An exploratory study. Arabian Journal for Science and Engineering, 35(2c) (pp. 103–124).
5. Aluisio, S., Specia, L., Gasperin, C. and Scarton, C. Readability Assessment for Text Simplification. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, 2010 (pp. 1–9).
6. Barzilay, R., & Elhadad, N., (2003). Sentence alignment for monolingual comparable Corpora. Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing (EMNLP’03) (pp. 25–32).
7. Bates, E. (2003). On the nature and nurture of language. In R. Levi-Montalcini, D. Baltimore, R. Dulbecco & F. Jacob (Series Eds.) & E. Bizzi, P. Calissano & V. Volterra (Vol. Eds.), Frontiers of biology: The brain of homo sapiens (pp. 241–265). Rome: Istituto della Enciclopedia Italiana fondata da Giovanni Trecanni S.p. A.
8. Becker, S. A. (2004). A study of web usability for older adults seeking online health resources. ACM Transactions on Computer-Human Interaction (TOCHI), 11(4), 387–406. DOI: 10.1145/1035575.1035578
9. Beinborn, L., Zesch, T., & Gurevych, I. (2012). Towards fine-grained readability measures for self-directed language learning. Proceedings of the SLTC 2012 Workshop on NLP for CALL: Linkoping Electronic Conference, 80, 11–19.
10. Benjamin, R. (2012). Reconstructing readability: Recent developments and recommendations in the analysis of text difficulty. Educational Psychology Review, 24(1), 63–88. DOI: 10.1007/s10648-011-9181-8
11. Carroll, J. B., Davies, P., & Richman, B. (1971). Word frequency book. Boston: Houghton Mifflin.
12. Chall, J. S. (1958). Readability: An appraisal of research and application. Bureau of Educational Research Monographs, No. 34. Columbus: Ohio State University Press.
13. Chall, J. S., & Dale, E. (1995). Readability revisited: The New Dale-Chall readability formula. Cambridge, MA: Brookline Books.
14. Chang, K. M., Nelson, J., Pant, U., & Mostow, J. (2013). Toward exploiting EEG input in a reading tutor. International Journal of Artificial Intelligence in Education, 22(1-2), 19–38.
15. Chen, X., Bennett, P. N., Collins-Thompson, K., & Horvitz, E. (2013). Pairwise ranking aggregation in a crowdsourced setting. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM’13) (pp. 193–202). ACM, New York, NY, USA.
16. Chen, Y.-T., Chen, Y.-H., & Cheng, Y.-C. (2013). Assessing Chinese readability using term frequency and lexical chains. Computational Linguistics and Chinese Language Processing, 18, 1–18.
17. Cole, M. J., Gwizdka, J., Liu, C., Belkin, N. J., & Zhang, X. (2012). Inferring user knowledge level from eye movement patterns. Information Processing and Management. DOI: http://dx.doi.org/10.1016/j.ipm.2012.08.004
18. Collins-Thompson, K., Bennett, P. N., White, R. W., de la Chica, S., & Sontag, D. (2011). Personalizing web search results by reading level. Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM’11) (pp. 403–412). ACM, New York, NY, USA.
19. Collins-Thompson, K., & Callan, J. (2004b). Information retrieval for language tutoring: An overview of the REAP project. Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’04) (pp. 544–545). ACM, New York, NY, USA.
20. Collins-Thompson, K, & Callan, J. (2004c). A language modeling approach to predicting reading difficulty. Proceedings of HLT-NAACL 2004 (pp. 193–200).
21. Collins-Thompson, K., & Callan, J. (2005). Predicting reading difficulty with statistical language models. Journal of the American Society for Information Science and Technology, 56, 1448–1462. DOI: 10.1002/asi.20243
22. Collins-Thompson, K. (2013). Enriching the web by modeling reading difficulty. Proceedings of the Sixth International Workshop on Exploiting Semantic Annotations in Information Retrieval (ESAIR ‘13) (pp. 3–4). ACM, New York, NY, USA.
23. Crossley, S. A., Greenfield, J., & McNamara, D. S. (2008). Assessing text readability using cognitively based indices. TESOL Quarterly, 42(3), 475–493.
24. Dale, E., & Chall, J. S. (1949). The concept of readability. Consciousness and Cognition, 26(23).
25. Dale, E., & O’Rourke, J. (1981). The living word vocabulary. Chicago, IL: World Book/Childcraft International.
26. Daowadung, P., & Chen, Y.-H. (2011). Using word segmentation and SVM to assess readability of Thai text for primary school students. Proceedings of the International Joint Conference on Computer Science and Software Engineering: JCSSE.
27. Dascalu, M. (2014). ReaderBench (2)-individual assessment through reading strategies and tex-tualcomplexity.AnalyzingDiscourseandTextComplexityforLearningandCollaborating(pp. 161–188). Springer International Publishing.
28. De Clercq, O., Hoste, V., Desmet, B., van Oosten, P., De Cock, M., & Macken, L. (2013). Using the crowd for readability prediction. Natural Language Engineering, 1(1). Cambridge University Press.
29. Dell’Orletta, F., Montemagni, S. and Venturi, G. READ-IT: Assessing Readability of Italian Texts with a View to Text Simplification. In Proceedings of the 2nd Workshop on Speech and Language Processing for Assistive Technologies, 2011 (pp. 73–83).
30. Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391–407. DOI: 10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9
31. Duarte Torres, S., & Weber, I. (2011). What and how children search on the web. Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM 2011) (pp. 393–402).
32. Feng, L., Elhadad, N., & Huenerfauth, M. (2009). Cognitively motivated features for readability assessment. Proceedings of the the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2009).
33. Ferguson, G., & Maclean, J. (1991). Assessing the readability of medical journal articles: An analysis of teacher judgements. Edinburgh Working Papers in Linguistics, 2, 112–125. http://files.eric.ed.gov/fulltext/ED353790.pdf
34. Fernández Huerta, J. (1959). Medidas sencillas de lecturabilidad. Consigna, 214, 29–32.
35. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221–233, Jun 1948. DOI: 10.1037/h0057532
36. Flor, M., Klebanov, B. B., & Sheehan, K. M. (2013). Lexical tightness and text complexity. Proceedings of the Second Workshop on Natural Language Processing for Improving Textual Accessibility.
37. François, T. L. (2009). Combining a statistical language model with logistic regression to predict the lexical and syntactic difficulty of texts for FFL. Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop. Association for Computational Linguistics.
38. François, T., & Fairon, C. (2012). An AI readability formula for French as a foreign language. Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP 2012) (pp. 466–477).
39. François, T., & Miltsakaki, E. (2012). Do NLP and machine learning improve traditional readability formulas? Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations (pp. 49–57). Association for Computational Linguistics.
40. François, T., Brouwers, L., Naets, H., & Fairon, C. (2014). AMesure: une formule de lisibilité pour les textes administratifs. Actes de la 21e Conférence sur le Traitement automatique des Langues Naturelles (TALN 2014) (pp. 467–472). Marseille.
41. Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon mechanical turk: Gold mine or coal mine? last words editorial. Computational Linguistics, 37(2). DOI: 10.1162/COLI_a_00057
42. Fry, E. (1990). A readability formula for short passages. Journal of Reading, Vol. 33, No. 8, 594–597, May 1990.
43. Gibson, E. (1998) Linguistic complexity: Locality of syntactic dependencies. Cognition, 68,1–76. DOI: 10.1016/S0010-0277(98)00034-1
44. Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, and Computers, 36(2), 193–202. DOI: 10.3758/BF03195564
45. Gyllstrom K., & Moens, M-F. (2010). Wisdom of the ages: Toward delivering the children’s web with the link-based agerank algorithm. Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM’10) (pp. 159–168). ACM, New York, NY, USA.
46. Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.
47. Hancke, J., Vajjala, S., & Meurers, D. (2012). Readability classification for German using lexical, syntactic, and morphological features. Proceedings of COLING 2012 (pp. 1063–1080).
48. Heilman, M., Collins-Thompson, K., Callan, J., & Eskenazi, M. (2007). Combining lexical and grammatical features to improve readability measures for first and second language Texts. Proceedings of HLT-NAACL’07 (pp. 460–467).
49. Heilman, M., Collins-Thompson, K., & Eskenazi, M. (2008). An analysis of statistical models and features for reading difficulty prediction. Proceedings of the ACL 2008 BEA Workshop on Innovative Use of NLP for Building Educational Applications.
50. Heilman, M., Collins-Thompson, K., Eskenazi, M., Juffs, A., & Wilson, L. (2010). Personalization of reading passages improves vocabulary acquisition. International Journal of Artificial Intelligence in Education, 20(1).
51. Honkela, T., Izzatdust, Z., & Lagus, K. (2012). Text mining for wellbeing: Selecting stories using semantic and pragmatic features. Artificial Neural Networks and Machine Learning–ICANN 2012 (pp. 467–474). Springer Berlin Heidelberg.
52. Jameel, S., Lam, W., & Qian, X. (2012). Ranking text documents on conceptual difficulty using term embedding and sequential discourse cohesion. Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (pp. 145–152).
53. Kandel, L., & Moles, A. (1958). Application de l’Indice de Flesch à la langue français. Cahiers d’Etudes de Radio-Television, 19, 253–274.
54. Kanungo, T., & Orr, D. (2009). Predicting the readability of short web summaries. Proceedings of the Second ACM International Conference on Web Search and Data Mining (WSDM’09) (pp. 202–211). ACM, New York, NY, USA.
55. Kate, R. J., Luo, X., Patwardhan, S., Franz, M., Florian, R., Mooney, R. J., Roukos, S., & Welty, C. (2010). Learning to predict readability using diverse linguistic features. Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010).
56. Kidwell, P., Lebanon, G., & Collins-Thompson, K. (2009). Statistical estimation of word acquisition with application to readability prediction. Proceedings of EMNLP’09 (pp. 900–909).
57. Kidwell, P., Lebanon, G., & Collins-Thompson, K. (2011). Statistical estimation of word acquisition with application to readability prediction. Journal of the American Statistical Association, 106(493), 21–30. DOI: 10.1198/jasa.2010.ap09318
58. Kim, J. Y., Collins-Thompson, K., Bennett, P. N., & Dumais, S. T. (2012). Characterizing web content, user interests, and search behavior by reading level and topic. Proceedings of the fifth ACM International Conference on Web Search and Data Mining (WSDM’12) (pp. 213–222). ACM, New York, NY, USA.
59. Kincaid, J. P., Fishburne, R. P., Rogers, R. L., & Chissom, B. S. (1975). Derivation of new readability formulas (Automated readability index, fog count, and flesch reading ease formula) for navy enlisted personnel. Research Branch Report 8–75. Chief of Naval Technical Training: Naval Air Station Memphis.
60. Kireyev, K., & Landauer, T. K. (2011). Word maturity: Computational modeling of word knowledge. Proceedings of ACL 2011 (pp. 299–308).
61. Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. Proceedings of the 26th Annual ACM Conference on Human Factors in Computing Systems (CHI ‘08) (pp. 453–456). ACM.
62. Klare, G. R. (1963). The measurement of readability. Ames, IA: Iowa State University Press.
63. Klerke, S., & Søgaard, A. DSim, a Danish Parallel Corpus for Text Simplification. In LREC May 2012 (pp. 4015–4018).
64. Landauer, T. K., Kireyev, K., & Panaccione, C. (2011). Word maturity: A new metric for word knowledge.ScientificStudiesofReading,15(1),92–108.DOI:10.1080/10888438.2011.536130
65. Lau, T. P. (2006). Chinese readability analysis and its applications on the internet. CUHK, Masters Thesis, Hong Kong.
66. Lennon, C., & Burdick, H. (2004). The lexile framework as an approach for reading measurement and success. Technical Report. Metametrics, Inc. April 2004. http://www.lexile.com/research/1/ (Retrieved Dec. 10, 2013)
67. Malvern, D., & Richards, B. (2012). Measures of lexical richness. Encyclopedia of Applied Linguistics, Blackwell Publishing Ltd.
68. McCullagh, P. (1980). Regression models for ordinal data. Journal of the Royal Statistical Society, Series B, 42(2), 109–142.
69. Mitchell, J. V. (1985). The ninth mental measurements yearbook. Lincoln, Nebraska: University of Nebraska Press.
70. Nandhini, K., & Balasundaram, S. R. (2011). Improving readability of dyslexic learners through document summarization. Proceedings of the Technology for Education (T4E), 2011 IEEE International Conference on. IEEE (pp. 246–249).
71. Nelson, J., Perfetti, C., Liben, D., & Liben, M. (2012). Measures of text difficulty: Testing their predictive value for grade levels and student performance. Technical Report submitted to the Gates Foundation. Feb. 1, 2012. URL: http://achievethecore.org/content/upload/nelson_perfetti_liben_measures_of_text_difficulty_research_ela.pdf
72. Paivio, A., Yuille, J. C., & Madigan, S. A. (1968). Concreteness, imagery, and meaningfulness: Values for 925 nouns. Journal of Experimental Psychology, 76(1), 1–25. Part 2 (1968). DOI: 10.1037/h0025327
73. Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2), 1–135. DOI: 10.1561/1500000011
74. Petersen, S. E. and Ostendorf, M.; A machine learning approach to reading level assessment; In Computer Speech and Language, 2009, 23, (pp. 86–106).
75. Pilán, I., Volodina, E., & Johansson, R. (2014). Rule-based and machine learning approaches for second language sentence-level readability. Proceedings of BEA Workshop 2014.
76. Pitler, E., & Nenkova, A. (2008). Revisiting readability: A unified framework for predicting text quality. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’08) (pp. 186–195). Stroudsburg, PA, USA, Association for Computational Linguistics. http://dl.acm.org/citation.cfm?id=1613715.1613742
77. Rello, L., Saggion, H., Baeza-Yates, R., & Graells, E. (2012). Graphical schemes may improve readability but not understandability for people with dyslexia. Proceedings of NAACL-HLT 2012.
78. Richardson, J. T. E. (1975). Imagery, concreteness, and lexical complexity, Quarterly Journal of Experimental Psychology, 27(2), 211–223. Psychology Press.
79. Russell, D. M. (2011). SearchReSearch: Search by reading level [Web log post]. Retrieved from http://searchresearch1.blogspot.com/2011/02/search-by-reading-level.html
80. Schwarm, S. E., & Ostendorf, M. (2005). Reading level assessment using support vector machines and statistical language models. Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL’05) (pp. 523–530). Stroudsburg, PA, USA, Association for Computational Linguistics.
81. Sato, S., Matsuyoshi, S., & Kondoh, Y. (2008). Automatic assessment of Japanese text readability based on a textbook corpus. Proceedings of LREC’08.
82. Si, L., & Callan, J. P. (2001). A statistical model for scientific readability. Proceedings of CIKM’01 (pp. 574–576).
83. Sitbon, L., & Bellot, P. (2008). A readability measure for an information retrieval process adapted to dyslexics. Proceedings of the Second International Workshop on Adaptive Information Retrieval (AIR 2008) (pp. 52–57).
84. Sjöholm, J. (2012). Probability as readability: A new machine learning approach to readability assessment for written Swedish. Masters Thesis, Linköpings universitet, 2012.
85. Sung, Y. T., Chen, J. L., Cha, J. H., Tseng, H. C., Chang, T. H., & Chang, K. E. (2014). Constructing and validating readability models: The method of integrating multilevel linguistic features with machine learning. Behavior Research Methods, 2014 April 2, 1–15.
86. Stenner, A. J., Burdick, H., Sanford, E. E., & Burdick, D. S. (2007). The lexile framework for reading technical report. MetaMetrics, Inc.
87. Tan, C., Gabrilovich, E., & Pang, B. (2012). To each his own: Personalized content selection based on text comprehensibility. Proceedings of the 5th ACM International Conference on Web Search and Data Mining, February 2012.
88. Tanaka, S., Jatowt, A., Kato, M. P., & Tanaka, K. (2013). Estimating content concreteness for finding comprehensible documents. Proceedings of WSDM’13. 475–484.
89. Tanaka-Ishii, K., Tezuka, S., & Terada, H. (2010). Sorting by readability. Computational Linguistics, 36(2), 203–227. DOI: 10.1162/coli.09-036-R2-08-050
90. Todirascu, A., François, T., Gala, N., Fairon, C., Ligozat, A. L., & Bernhard, D. (2013). Coherence and cohesion for the assessment of text readability. Natural Language Processing and Cognitive Science, 11, (pp. 11–19).
91. Vapnik, V. N. (1995). The nature of statistical learning theory. New York: Springer-Verlag Inc. DOI: 10.1007/978-1-4757-2440-0
92. Vajjala, S., & Meurers, D. (2012). On improving the accuracy of readability classification using insights from second language acquisition. Proceedings of the Seventh Workshop on Building Educational Applications Using NLP (pp. 163–173). ACL.
93. Vajjala, S., & Meurers, D. (2014). Readability assessment for text simplification: From analyzing documents to identifying sentential simplifications. ITL International Journal of Applied Linguistics, Sept. 2014.
94. von Ahn, L., & Dabbish, L. (2008). Designing games with a purpose. Communications of the ACM, 51(8),58–67. (August 2008), DOI: 10.1145/1378704.1378719
95. Vor Der Brück, T., & Hartrumpf, S. (2007). A semantically oriented readability checker for German. Proceedings of the 3rd Language & Technology Conference (pp. 270–274). Poznan, Poland. October 2007.
96. Wang, Y. (2006). Automatic recognition of text difficulty from consumers health information. Proceedings of the IEEE Symposium on Computer-Based Medical Systems (pp. 131–136). Los Alamitos, CA, USA, IEEE Computer Society.
97. Zakaluk, B. L., & Samuels, S. J. (1988). Readability: Its past, present and future. International Reading Association. Newark, Del..