The role of time on task in computer-based low-stakes assessment of cross-curricular skills

Journal of Educational Psychology

Volumn 106, Issue 3, 2014, Pages 627-638

  Kupiainen, Sirkku ^a   Vainikainen, Mari Pauliina ^a   Marjanen, Jukka ^a   Hautamäki, Jarkko ^a  

^a UNIVERSITY OF HELSINKI   (Finland)

Author keywords

Cross curricular skills; Log data; Low stakes assessment; Motivational attitudes; Time on task

Indexed keywords

EID: 84894265413     PISSN: 00220663     EISSN: 19392176     Source Type: Journal    
DOI: 10.1037/a0035507     Document Type: Article

References (70)

1. Adey, P., Csapó, B., Demetriou, A., Hautamäki, J., & Shayer, M. (2007). Can we be intelligent about intelligence? Why education needs the concept of plastic general ability. Educational Research Review, 2, 75-97. doi:10.1016/j.edurev.2007.05.001
2. America Achieves. (2013). Middle class or middle of the pack? What can we learn when benchmarking U.S. schools against the world's best? Retrieved from http://www.americaachieves.org/docs/OECD/Middle-Class-Or-Middle-Of-Pack.pdf
3. Atkinson, R. C., & Geiser, S. (2009). Reflections on a century of college admission tests. Educational Researcher, 38, 665-676. doi:10.3102/ 0013189X09351981
4. Bachman, J. G., & O'Malley, P. M. (1984). Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles. Public Opinion Quarterly, 48, 491-509. doi:10.1086/268845
5. Barry, C. L., Horst, S. J., Finney, S. J., Brown, A. R., & Kopp, J. P. (2010). Do examinees have similar test-taking effort? A high-stakes question for low-stakes testing. International Journal of Testing, 10, 342-363. doi: 10.1080/15305058.2010.508569
6. Bloom, B. S. (1980). The new direction in educational research: Alterable variables. The Journal of Negro Education, 49, 337-349. doi:10.2307/ 2295092
7. Buckley, J. (2009, June). Cross-national response styles in international educational assessments: Evidence from PISA 2006. Paper presented at the National Center for Education Statistics Conference on the Program for International Student Assessment (PISA): What We Can Learn From PISA, Washington, DC. https://edsurveys.rti.org/PISA/
8. Carroll, J. B. (1963). A model of school learning. Teachers College Record, 64, 723-733.
9. Carroll, J. B. (1989). The Carroll model: A 25-year retrospective and prospective view. Educational Researcher, 18, 26-31. doi:10.3102/ 0013189X018001026
10. Cattell, R. B. (1987). Intelligence: Its structure, growth and action. Amsterdam, the Netherlands: Elsevier.
11. Chang, S.-R., Plake, B. S., & Ferdous, A. A. (2005). Response times for correct and incorrect item responses on computerized adaptive tests [Paper presented at the 2005 annual meeting of the American Educational Research Association (AERA), Montréal, Canada]. Retrieved from http://www.eric.ed.gov/
12. Chapman, M., Skinner, E. A., & Baltes, P. B. (1990). Interpreting correlations between children's perceived control and cognitive performance: Control, agency or means-ends beliefs. Developmental Psychology, 26, 246-253. doi:10.1037/0012-1649.26.2.246
13. Cheung, G. W., & Lau, R. S. (2008). Testing mediation and suppression effects of latent variables: Bootstrapping with structural equation models. Organizational Research Methods, 11, 296 -325. doi:10.1177/ 1094428107300343
14. Cousineau, D., & Chartier, S. (2010). Outliers detection and treatment: A review. International Journal of Psychological Research, 3(1), 58-67.
15. Csapó, B. (2007). Research into learning to learn through the assessment of quality and organization of learning outcomes. The Curriculum Journal, 18, 195-210. doi:10.1080/09585170701446044
16. Deary, I. J., Strand, S., Smith, P., & Fernandes, C. (2007). Intelligence and educational achievement. Intelligence, 35, 13-21. doi:10.1016/j.intell .2006.02.001
17. Deci, E. L., & Ryan, R. M. (2000). The "what" and "why" of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11, 227-268. doi:10.1207/S15327965PLI110401
18. Demetriou, A., & Kazi, S. (2006). Self-awareness in g (with processing efficiency and reasoning). Intelligence, 34, 297-317. doi:10.1016/j.intell .2005.10.002
19. Demetriou, A., Pachaury, A., Metallidou, Y., & Kazi, S. (1996). Universals and specificities in the structure and development of quantitativerelational thought: A cross-cultural study in Greece and India. International Journal of Behavioral Development, 19, 255-290. doi:10.1080/ 016502596385785
20. Demetriou, A., Platsidou, M., Efklides, A., Metallidou, Y., & Shayer, M. (1991). The development of quantitative-relational abilities from childhood to adolescence: Structure, scaling, and individual differences. Learning and Instruction, 1, 19-43.
21. Demetriou, A., Spanoudis, G., & Mouyi, A. (2011). Educating the developing mind: Towards an overarching paradigm. Educational Psychology Review, 23, 601-663. doi:10.1007/s10648-011-9178-3
22. Elliott, E. S., & Dweck, C. S. (1988). Goals: An approach to motivation and achievement. Journal of Personality and Social Psychology, 54, 5-12. doi:10.1037/0022-3514.54.1.5
23. Gagné, F., & St Père, F. (2002). When IQ is controlled, does motivation still predict achievement? Intelligence, 30, 71-100. doi:10.1016/S0160- 2896(01)00068-X
24. Gettinger, M. (1985). Time allocated and time spent relative to time needed for learning as determinants of achievement. Journal of Educational Psychology, 77, 3-11. doi:10.1037/0022-0663.77.1.3
25. Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106, 608-626. doi:10.1037/a0034716
26. Greiff, S., Wüstenberg, S., Molnár, G., Fischer, A., Funke, J., & Csapó, B. (2013). Complex problem solving in educational settings-Something beyond g: Concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology, 105, 364 -379. doi: 10.1037/a0031856
27. Grek, S. (2009). Governing by numbers: The PISA 'effect' in Europe. Journal of Education Policy, 24, 23-37. doi:10.1080/02680930802412669
28. Gustafsson, J.-E., & Carlstedt, B. (2006, August). Abilities and grades as predictors of achievement: The encapsulation theory. Paper presented at the 114th Annual Convention of the American Psychological Association, New Orleans, LA.
29. Harackiewicz, J. M., Barron, K. E., Pintrich, P. R., Elliot, A. J., & Thrash, T. M. (2002). Revision of achievement goal theory: Necessary and illuminating. Journal of Educational Psychology, 94, 638-645. doi: 10.1037/0022-0663.94.3.638
30. Harter, S. (1999). The construction of the self: A developmental perspective. New York, NY: Guilford Press.
31. Hattie, J. (2005). The paradox of reducing class size and improving learning outcomes. International Journal of Educational Research, 43, 387-425. doi:10.1016/j.ijer.2006.07.002
32. Hautamäki, J. (1984). Peruskoululaisten loogisen ajattelun mittaamisesta ja esiintymisestä [The measurement and distribution of Piagetian stages of thinking in Finnish comprehensive school]. Joensuu, Finland: University of Joensuu, Department of Social Sciences.
33. Hautamäki, J., Arinen, P., Eronen, S., Hautamäki, A., Kupiainen, S., Lindblom, B., . . . Scheinin, P. (2002). Assessing learning-to-learn: A framework. Retrieved from the Finland National Board of Education website: http://www.oph.fi/download/47716_learning.pdf
34. Hautamäki, J., Kupiainen, S., Arinen, P., Hautamäki, A., Niemivirta, M., Rantanen, P., & Scheinin, P. (2006). Learning-to-learn assessment in Finland: Versatile tools to monitor and improve effectiveness and equity of the education system. In R. Jakku-Sihvonen & H. Niemi (Eds.), Research-based teacher education in Finland: Reflections by Finnish teacher educators (pp. 189-202). Jyväskylä, Finland: Finnish Educational Research Association.
35. Hoskins, B., & Fredriksson, U. (2008). Learning to learn: What is it and can it be measured? (JRC Scientific and Technical Report EUR 23432 EN). Ispra, Italy: European Commission, Joint Research Centre, Centre for Research on Lifelong Learning.
36. Inhelder, B., & Piaget, J. (1958). The early growth of logic in the child. London, United Kingdom: Routledge & Kegan Paul.
37. Karweit, N. (1982). Time on task: A research review [Report No. 332]. Retrieved from http://www.eric.ed.gov/PDFS/ED228236.pdf
38. Karweit, N., & Slavin, R. E. (1981). Measurement and modeling choices in studies of time and learning. American Educational Research Journal, 18, 157-171. doi:10.3102/00028312018002157
39. Klauer, K. J. (1988). Teaching for learning-to-learn: A critical appraisal with some proposals. Instructional Science, 17, 351-367. doi:10.1007/ BF00056221
40. Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd ed.). New York, NY: Guilford Press.
41. Kong, X. J., Wise, S. L., & Bhola, D. S. (2007). Setting the response time threshold parameter to differentiate solution behavior from rapidguessing behavior. Educational and Psychological Measurement, 67, 606-619. doi:10.1177/0013164406294779
42. Kupiainen, S., Marjanen, J., Vainikainen, M.-P., & Hautamäki, J. (2011). Oppimaan oppiminen Vantaan peruskouluissa: Kolmas-, kuudes- ja yhdeksäsluokkalaiset oppijoina keväällä 2010 [Learning to learn in comprehensive schools in Vantaa: 3rd, 6th and 9th graders as learners in spring 2010]. Department of Education, University of Helsinki, Vantaa, Finland.
43. Lee, Y.-H., & Chen, H. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53, 359-379.
44. Leys, C., Ley, C., Klein, O., Bernard, P., & Licata, L. (2013). Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49, 764-766. doi:10.1016/j.jesp.2013.03.013
45. Little, T. D., Lopez, D. F., Oettingen, G., & Baltes, P. B. (2001). A comparative-longitudinal study of action-control beliefs and school performance: On the role of context. International Journal of Behavioral Development, 25, 237-245. doi:10.1080/01650250042000258
46. MacKinnon, D. P., Krull, J. L., & Lockwood, C. M. (2000). Equivalence of the mediation, confounding and suppression effect. Prevention Science, 1, 173-181. doi:10.1023/A:1026595011371
47. MacKinnon, D. P., Lockwood, C. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83-104. doi:10.1037/1082-989X.7 .1.83
48. Muthén, L. K., & Muthén, B. O. (1998 -2010). Mplus user's guide (6th ed.). Los Angeles, CA: Muthén & Muthén.
49. Niemivirta, M. (2002). Individual differences and developmental trends in motivation: Integrating person-centred and variable-centred methods. In P. R. Pintrich & M. L. Maehr (Eds.), New directions in measures and methods: Advances in motivation and achievement (Vol. 12, pp. 241-275). Amsterdam, the Netherlands: JAI Press.
50. Olson, D. (2003). Psychological theory and educational reform: How school remakes mind and society. Cambridge, United Kingdom: Cambridge University Press.
51. Rohde, T. E., & Thompson, L. A. (2007). Predicting academic achievement with cognitive ability. Intelligence, 35, 83-92. doi:10.1016/j.intell .2006.05.004
52. Ross, J. D., & Ross, C. M. (1979). Ross test of higher cognitive processes. Novato, CA: Academic Therapy.
53. Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54-67. doi:10.1006/ceps.1999.1020
54. Scheerens, J., & Bosker, R. J. (1997). The foundations of educational effectiveness. Oxford, United Kingdom: Pergamon Press.
55. Schnipke, D. L., & Scrams, D. J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213-232. doi:10.1111/j .1745-3984.1997.tb00516.x
56. Schnipke, D. L., & Scrams, D. J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In C. N. Mills, M. T. Potenza, J. J. Fremer, & W. C. Ward (Eds.), Computer-based testing: Building the foundation for future assessments (pp. 237-266). Mahwah, NJ: Erlbaum.
57. Shayer, M. (1979). Has Piaget's construct of formal operational thinking any utility? British Journal of Educational Psychology, 49, 265-276. doi:10.1111/j.2044-8279.1979.tb02425.x
58. Snow, R. E. (1990). New approaches to cognitive and conative assessment in education. International Journal of Educational Research, 14, 455-473.
59. Spinath, B., Spinath, F. M., Harlaar, N., & Plomin, R. (2006). Predicting school achievement from general cognitive ability self-perceived ability and intrinsic value. Intelligence, 34, 363-374. doi:10.1016/j.intell.2005 .11.004
60. Sternberg, R., Castejon, J. L., Prieto, M. D., Hautamäki, J., & Grigorenko, E. (2001). Confirmatory factor analysis of the Sternberg Triarchic Abilities Test in three international samples. European Journal of Psychological Assessment, 17, 1-16. doi:10.1027/1015-5759.17.1.1
61. Sundre, D. L., & Wise, S. L. (2003, April). 'Motivation filtering': An exploration of the impact of low examinee motivation on the psychometric quality of tests. Paper presented at the National Council on Measurement in Education Annual Conference, Chicago, IL.
62. Thorsen, C., & Cliffordson, C. (2012). Teachers' grade assignment and the predictive validity of criterion-referenced grades. Educational Research and Evaluation: An International Journal on Theory and Practice, 18, 153-172. doi:10.1080/13803611.2012.659929
63. Urdan, T., & Midgley, M. (2001). Academic self-handicapping: What we know and what more there is to learn. Educational Psychology Review, 13, 115-138. doi:10.1023/A:1009061303214
64. Wang, S., Jiao, H., Young, M. J., Brooks, T., & Olson, J. (2008). Comparability of computer-based and paper-and-pencil testing in K-12 reading assessments: A meta-analysis of testing mode effects. Educational and Psychological Measurement, 68, 5-24. doi:10.1177/ 0013164407305592
65. Wise, S. L. (2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19, 95-114. doi:10.1207/s15324818ame1902_2
66. Wise, S. L. (2009). Strategies for managing the problem of unmotivated examinees in low-stakes testing programs. Journal of General Education, 58, 152-166. doi:10.1353/jge.0.0042
67. Wise, S. L., & DeMars, C. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1-17. doi:10.1207/s15326977ea1001_1
68. Wise, S. L., & DeMars, C. (2008, March). Examinee non-effort and the validity of program assessment results. Paper presented at the 2008 annual meeting of the National Council on Measurement in Education, New York, NY.
69. Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163-183. doi:10.1207/s15324818ame1802_2
70. Zhao, X., Lynch, J. G., & Chen, Q. (2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37, 197-206. doi:10.1086/651257