Constructing and validating readability models: the method of integrating multilevel linguistic features with machine learning

Behavior Research Methods

Volumn 47, Issue 2, 2015, Pages 340-354

  Sung, Yao Ting ^a   Chen, Ju Ling ^a   Cha, Ji Her ^a   Tseng, Hou Chiang ^a   Chang, Tao Hsing ^b   Chang, Kuo En ^a  

^a NATIONAL TAIWAN NORMAL UNIVERSITY   (Taiwan)

^b NATIONAL KAOHSIUNG UNIVERSITY OF APPLIED SCIENCES   (Taiwan)

Author keywords

Linguistic features; Multilevel; Readability; Support vector machine; Validity

Indexed keywords

COMPREHENSION; HUMAN; LINGUISTICS; MACHINE LEARNING; PSYCHOLOGICAL MODEL; READING; REPRODUCIBILITY; SUPPORT VECTOR MACHINE;

COMPREHENSION; HUMANS; LINGUISTICS; MACHINE LEARNING; MODELS, PSYCHOLOGICAL; READING; REPRODUCIBILITY OF RESULTS; SUPPORT VECTOR MACHINE;

EID: 84897120854     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-014-0459-x     Document Type: Article

References (73)

1. Ahrens, K. (2006). The effect of visual target presentation times on lexical ambiguity resolution. Language and Linguistics, 7(3), 677–696.
2. Anderson, R. C., & Chen, X. (2012). Chinese reading development in monolingual and bilingual learners: Introduction to the special issue. Scientific Studies of Reading, 17(1), 1–4.
3. Bailin, A., & Grafstein, A. (2001). The linguistic assumptions underlying readability formulae: A critique. Language & Communication, 21(3), 285–301.
4. Bazi, Y., & Melgani, F. (2006). Toward an optimal SVM classification system for hyperspectral remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 44(11), 3374–3385.
5. Beck, I. L., McKeown, M. G., & Kucan, L. (2002). Bringing words to life: Robust vocabulary instruction. New York, NY: Guilford Press.
6. Benjamin, R. (2012). Reconstructing readability: Recent developments and recommendations in the analysis of text difficulty. Educational Psychology Review, 24(1), 63–88.
7. Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. In D. Haussler (Ed.), 5th Annual ACM Workshop on Computational learning theory (pp. 144–152). Pittsburgh, PA: ACM Press.
8. Bruce, B., Rubin, A., & Starr, K. S. (1981). Why readability formulas fail. IEEE Transactions on Professional Communication, PC-24, 50–52.
9. Cardoso-Cachopo, A., & Oliveira, A. L. (2003). An empirical comparison of text categorization methods. In M. A. Nascimento, E. S. de Moura, & A. L. Oliveira (Eds.), String processing and information retrieval (pp. 183–196). Heidelberg: Springer.
10. Carpenter, P. A., & Just, M. A. (1983). What your eyes do while your mind is reading. In K. Rayner (Ed.), Eye movements in reading: Perceptual and language processes (pp. 275–307). New York, NY: Academic Press.
11. Chae, J., & Nenkova, A. (2009). Predicting the fluency of text with shallow structural features: case studies of machine translation and human-written text. Paper presented at the Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, Athens, Greece.
12. Chang, Y. W., & Lin, C. J. (2008). Feature ranking using linear svm. Causation and Predicition Challenge in Machine Learning, Volume 2, 47.
13. Chang, C. C. & Lin, C. J. (2011). LIBSVM [a library for support vector machines]. Retrieved from http://www.csie.ntu.edu.tw/~cjlin/libsv
14. Chang, T. H., Sung, Y. T., & Lee, Y. T. (2012, November). A Chinese word segmentation and POS tagging system for readability research. Paper presented at the 42nd Annual Meeting of the Society for Computers in Psychology, Minneapolis, MN.
15. Chen, Y. W., & Lin, C. J. (2006). Combining SVMs with various feature selection strategies. Studies in Fuzziness and Soft Computing, 207, 315–324.
16. Chen, J. L., & Su, Y. F. (2010). The effect of decoding ability on character complexity effect and word length effect in Taiwanese beginning reader. Bulletin of Educational Psychology, 41(3), 579–604.
17. Chen, S., Zhou, S., Yin, F. F., Marks, L. B., & Das, S. K. (2007). Investigation of the support vector machine algorithm to predict lung radiation-induced pneumonitis. International Journal of Medical Physics Research and Practice, 34, 3808–3814.
18. Crossley, S. A., Allen, D. B., & McNamara, D. S. (2011). Text readability and intuitive simplification: A comparison of readability formulas. Reading in a Foreign Language, 23(1), 84–101.
19. Crossley, S. A., Greenfield, J., & McNamara, D. S. (2008). Assessing text readability using cognitively based indices. TESOL Quarterly, 42(3), 475–493.
20. Dale, E., & Chall, J. S. (1948). A formula for predicting readability: Instructions. Educational Research Bulletin, 27, 37–54.
21. Dale, E., & Chall, J. S. (1949). The concept of readability. Elementary English, 26, 23.
22. Ding, S. (2009). Feature selection based F-score and ACO algorithm in support vector machine. Second International Symposium on Knowledge Acquisition and Modeling, 1, 19–23.
23. DuBay, W. H. (2007). Smart language: Reader, readability, and the grading of text. Costa Mesa, CA: Impact Information.
24. Duda, R., Hart, P., & Stork, D. (2000). Pattern classification (2nd ed.). NewYork: Wiley.
25. Dumais, S. T., Furnas, G. W., Landauer, T. K., Deerwester, S., & Harshman, R. (1988). Using latent semantic analysis to improve information retrieval. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 281–285. Washington, D.C.
26. Feng, L., Jansche, M., Huenerfauth, M., & Elhadad, N. (2010). A comparison of features for automatic readability assessment. Proceedings of the 23rd International Conference on Computational Linguistics, 276-284.
27. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221–223.
28. Foltz, P. W., Kintsch, W., & Landauer, T. K. (1998). The measurement of textual coherence with Latent Semantic Analysis. Discourse Processes, 25, 285–307.
29. François, T., & Fairon, C. (2012). An "AI readability" formula for French as a foreign language. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (pp. 466-477). Jeju Island, Korea: Association for Computational Linguistics.
30. Gershkoff-Stowe, L., & Hahn, E. R. (2007). Fast mapping skills in the developing lexicon. Journal of Speech, Language, and Hearing Research, 50, 682–697.
31. Givón, T. (1979). On understanding grammar. New York, NY: Academic Press.
32. Glass, G. V., & Hopkins, K. D. (1984). Statistical methods in education and psycholgoy. Englewood, NJ: Prentice-Hall.
33. Graesser, A. C., & McNamara, D. S. (2011). Computational analyses of multi-level discourse comprehension. Topics in Cognitive Science, 3, 371–398.
34. Graesser, A. C., McNamara, D. S., & Kulikowich, J. M. (2011). Coh-Metrix: Providing multi-level analyses of text characteristics. Educational Researcher, 40(5), 223–234.
35. Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2), 193–202.
36. Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Science, 17(8), 684–691.
37. Grainger, J., & Frenck-Mestre, C. (1998). Masked priming by translation equivalents in proficient bilinguals. Language and Cognitive Processes, 13(6), 601–623.
38. Heilman, M., Thompson, K. C., & Eskenazi, M. (2008). An analysis of statistical models and features for reading difficulty prediction. Paper presented at the Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications, Columbus, OH.
39. Huang, S. (2000). The story of heads and tails─ on a sequentially sensitive lexicon. Language and Linguistics, 1(2), 79–107.
40. Jiang, S., Pang, G., Wu, M., & Kuang, L. (2012). A improved K-nearest-neighbor algorithm for text categorization. Expert Systems with Applications, 39(1), 1503–1509.
41. Johansson, V. (2008). Lexical diversity and lexical density in speech and writing. Lund, Sweden: Lund University.
42. Jordan, M. P. (1998). The power of negation in English: Text, context and relevance. Journal of Pragmatics, 29, 705–752.
43. Keerthi, S. S., & Lin, C. J. (2003). Asymptotic behaviors of support vector machines with Gaussian kernel. Neural Computation, 15(7), 1667–1689.
44. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.
45. Klare, G. R. (1985). Matching reading materials to readers: The role of readability estimates in conjunction with other information about comprehensibility. In T. L. Harris & E. J. Cooper (Eds.), Reading, Thinking, and concept development. New York: College Entrance Examination Board.
46. Klare, G. R. (2000). The measurement of readability: Useful information for communicators. ACM Journal of Computer Documentation, 24(3), 107–121.
47. Lakoff, G., & Johnson, M. (1980). Metaphors we live by. London, UK: University of Chicago Press.
48. Lee, Y. S., Tseng, H. C., Chen, J. L., Peng, C. Y., Chang, T. H., & Sung, Y. T. (2012, July). Constructing a novel Chinese readability classification model using principal component analysis and genetic programming. Paper presented at the 12th International Conference on Advanced Learning Technologies (ICALT), Rome, Italy.
49. Lemaire, B., Denhiere, G., Bellissens, C., & Jhean-Iarose, S. (2006). A computational model for simulating text comprehension. Behavior Research Methods, 38(4), 628–637.
50. Lin, H. T., & Lin, C. J. (2003). A study on sigmoid kernels for SVM and the training of non-PSD kernels by SMO-type methods. Technical report, Department of Computer Science and Information Engineering, National Taiwan University.
51. Liu, Y., Shu, H., & Li, P. (2007). Word naming and psycholinguistic norms: Chinese. Behavior Research Methods, 39, 192–198.
52. Louwerse, M. M. (2004). Semantic variation in idiolect and sociolect: Corpus linguistic evidence from literary texts. Computers and the Humanities, 38, 207–221.
53. Louwerse, M. M., & Mitchell, H. H. (2003). Toward a taxonomy of a set of discourse markers in dialog: A theoretical and computational linguistic account. Discourse Processes, 23, 441–470.
54. McBride-Chang, C., Cho, J-R., Liu, H., Wagner, R. K., Shu, H., Zhou, A., … Muse, A. (2005). Changing models across cultures: Associations of phonological awareness and morphological structure awareness with vocabulary and word recognition in second graders from Beijing, Hong Kong, Korea, and the United States. Journal of Experimental Child Psychology, 92, 140-160.
55. McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47, 292–330.
56. Noble, C. H., Rowland, C. F., & Pine, J. M. (2011). Comprehension of argument structure and semantic roles: Evidence from English–learning children and the forced–choice pointing paradigm. Cognitive Science, 35, 963–982.
57. Perfetti, C. A. (2007). Reading ability: Lexical quality to comprehension. Scientific Studies of Reading, 11, 357–383.
58. Petersen, S. E., & Ostendorf, M. (2009). A machine learning approach to reading level assessment. Computer Speech and Language, 23, 89–106.
59. Ravid, D., & Berman, R. A. (2010). Developing noun phrase complexity at school age: A text-embedded cross-linguistic analysis. First Language, 30, 3–26.
60. Rayner, K., Foorman, B., Perfetti, C., Pesetsky, D., & Seidenberg, M. (2001). How psychological science informs the teaching of reading. Psychological Science in the Public Interest, 2(2), 31–74.
61. Samuels, S. J. (2006). Toward a model of reading fluency. In S. J. Samuels & E. E. Farstrup (Eds.), What research says about reading instruction (pp. 24–46). Newark, DE: International Reading Association.
62. Sanders, T. J. M., Spooren, W. P. M., & Noordman, L. G. M. (1992). Toward a taxonomy of coherence relations. Discourse Processes, 15, 1–35.
63. Schriver, K. (2000). Readability formula in the new millennium: What's the use? ACM Journal of Computer Documentation, 24(3), 138–140.
64. Shapiro, A. M., & McNamara, D. S. (2000). The use of latent semantic analysis as a tool for the quantitative assessment of understanding and knowledge. Journal of Educational Computing Research, 22, 1–36.
65. Singer, M., Harkness, D., & Stewart, S. T. (1997). Constructing inferences in expository text comprehension. Discourse Processes, 24, 199–228.
66. Sung, Y. T., Chen, J. L., Lee, Y. S., Cha, J. H., Tseng, H. C., Lin, W. C., & Chang, K. E. (2013). Investigating chinese text readability: Linguistic features, modeling, validation. Chinese Journal of Psychology, 55(1), 75–106.
67. Sung, Y. T., Chen, J. L., Lee, Y. T., Lee, Y. S., Peng, C. Y., Tseng, H. C., & Chang, T. H. (2012, July). Constructing and Validating a Readability Modal with LSA: A case study of Chinese and social science textbooks. Paper presented at the 22th Annual Meeting of Society for Text and Discourse Process, Montreal, Canada.
68. Tan, L. H., & Peng, D. L. (1990). The effects of semantic context on the feature analyses of single Chinese characters. Journal of Psychology, 4, 5–10.
69. van den Broek, P., & Kremer, K. E. (2000). The mind in action: What it means to comprehend during reading. In B. M. Taylor, M. F. Graves, & P. v. d. Broek (Eds.), Reading for meaning: Fostering comprehension in the middle grades (pp. 1–31). Newark, DE: International Reading Association.
70. van den Broek, P., Risden, K., Fletcher, C. R., & Thurlow, R. (1996). A “landscape” view of reading: Fluctuating patterns of activation and the construction of a stable memory representation. In B. K. Britton & A. C. Graesser (Eds.), Models of understanding text (pp. 165–187). Mahwah, NJ: Erlbaum.
71. Vapnik, V. N., & Chervonenkis, A. Y. (1974). Teoriya raspoznavaniya obrazov: Statisticheskie problemy obucheniya [Theory of pattern recognition: Statistical problems of learning]. Moscow, Russia: Nauka.
72. Verhoeven, L., & Perfetti, C. (2008). Advances in text comprehension: Model, process and development. Applied Cognitive Psychology, 22, 293–301.
73. Yang, J., Wang, S., Chen, H., & Rayner, K. (2009). The time course of semantic and syntactic processing in Chinese sentence comprehension: Evidence from eye movements. Memory & Cognition, 37(8), 1164–1176.