Do examinees have similar test-taking effort? a high-stakes question for low-stakes testing

International Journal of Testing

Volumn 10, Issue 4, 2010, Pages 342-363

  Barry, Carol L ^a   Horst, S Jeanne ^a   Finney, Sara J ^a   Brown, Allison R ^a   Kopp, Jason P ^a  

^a James Madison University   (United States)

Author keywords

Accountability; Expectancy value theory; Low stakes testing; Mixture modeling; Test taking motivation

Indexed keywords

EID: 79960531670     PISSN: 15305058     EISSN: 15327574     Source Type: Journal    
DOI: 10.1080/15305058.2010.508569     Document Type: Article

References (41)

1. Ackerman, P. L., & Kanfer, R. (2009). Test length and cognitive fatigue: An empirical examination of effects of performance and test-taker reactions. Journal of Experimental Psychology: Applied, 15, 163-181.
2. Akaike, H. (1987). Factor analysis and AIC. Psychometrika, 52, 317-332.
3. American Educational Research Association, American Psychological Association, & National Council for Measurement in Education (AERA, APA, & NCME). (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
4. Asparouhouv, T., & Muthén, B. O. (2007). Wald test of mean equality for potential latent class predictors in mixture modeling. Retrieved January 25, 2008 from http://www.statmodel. com/download/MeanTest1.pdf
5. Bauer, D. J., & Curran, P. J. (2004). The integration of continuous and discrete latent variable models: Potential problems and promising opportunities. Psychological Methods, 9, 3-29.
6. Baumert, J., & Demmrich, A. (2001). Test motivation in the assessment of student skills: The effects of incentives on motivation and performance. European Journal of Psychology of Education, 16, 441-462.
7. Bovaird, J. A. (2002). New applications in testing: Using response time to increase the construct validity of a latent trait estimate. Doctoral dissertation, University of Kansas. Dissertation Abstracts International, 64, 998.
8. Cacioppo, J. T., Petty, R. E., & Kao, C. F. (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48, 306-307.
9. Cao, J., & Stokes, S. L. (2008). Bayesian IRT guessing models for partial guessing behaviors. Psy-chometrika, 73, 209-230.
10. Clark, S. L., & Muthén, B. (2009, April). Relating latent class analysis results to variables not included in the analysis. Paper presented the annual meeting of the American Educational Research Association, San Diego, CA.
11. Eccles, J. S., Adler, T. F., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., et al. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Achievement and achiement motivation (pp. 75-146). San Francisco, CA: W. H. Freeman.
12. Eklöf, H. (2007). Test-taking motivation and mathematics performance in TIMSS 2003. International Journal of Testing, 7, 311-326.
13. Finney, S. J., Pieper, S. L., & Barron, K. E. (2004). Examining the psychometric properties of the Achievement Goal Questionnaire in a general academic context. Educational and Psychological Measurement, 62, 365-382.
14. International Test Commission (ITC). (2000). International Guidelines for Test Use. Retrieved May 22, 2010, from http://www.intestcom.org/Guidelines/Test+Use.php
15. John, O. P., & Srivastava, S. (1999). The big-five trait taxonomy: History, measurement, and theoretical perspectives. In L. A. Pervin & O. P. John (Eds.), Handbook of personality: Theory and research (2nd ed., pp. 102-138). New York: Guilford.
16. Liem, A. D., Lau, S., & Nie, Y. (2008). The role of self-efficacy, task value, and achievement goals in predicting learning strategies, task disengagement, peer relationship, and achievement outcome. Contemporary Educational Psychology, 33, 486-512.
17. Linn, R. L. (2000). Assessments and accountability. Educational Researcher, 29(2), 4-16.
18. Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88, 767-778.
19. Lubke, G. (2010). Latent variable mixture models. In G. R. Hancock & R. O. Mueller (Eds.), The reviewers guide to quantitative methods in the social sciences (pp. 209-219). New York: Routledge.
20. Marsh, H. W., & Hau, K-T. (2006). Applications of latent-variable models in educational psychology: The need for methodological-substantive synergies. Contemporary Educational Psychology, 32, 151-170.
21. Mislevy, R. J. (1995). What can we learn from international assessments? Educational Evaluation and Policy Analysis, 17, 419-437. Retrieved May 26, 2009, from http://www.jstor.org/stable/1164436
22. Muthén, L. K., & Muthén, B. O. (1998-2007). Mplus user's guide (5th ed.). Los Angeles: Muthén & Muthén. No Child Left Behind Act of 2001, 20 U.S.C.S. Section 6301 et seq. (2002). Retrieved November 13, 2009, from http://www.ed.gov/nclb/overview/importance/difference/index.html
23. O'Neil, H. F., Sugrue, B., & Baker, E. L. (1995/1996). Effects of motivational interventions on the national assessment of educational progress mathematics performance. Educational Assessment, 3, 135-157.
24. Pastor, D. A., Barron, K. E., Miller, B. J., & Davis, S. L. (2007). A latent profile analysis of college students' achievement goal orientation. Contemporary Educational Psychology, 32, 8-47.
25. Pieper, S. L. (2004). Refining and extending the 2 × 2 achievement goal framework: Another look at work-avoidance. Doctoral dissertation, James Madison University, 2004. Dissertation Abstracts International, 64, 4436.
26. Schwartz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461-464.
27. Sclove, L. S. (1987). Application of model selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333-343.
28. Sundre, D. L., Thelk, A., & Wigtil, C. (2008). The Natural World Test, version 9: A measure of quantitative and scientific reasoning, test manual. Harrisonburg, VA: James Madison University, Center for Assessment and Research Studies.
29. Swerdzewski, P. J., Harmes, J. C., & Finney, S. J. (2009). Skipping the test: Using empirical evidence to inform policy related to students who avoid taking low-stakes assessment in college. Journal of General Education, 58, 167-195.
30. Thelk, A., Sundre, D. L., Horst, J. S., & Finney, S. J. (2009). Motivation matters: Using the Student Opinion Scale (SOS) to make valid inferences about student performance. Journal of General Education, 58, 129-151.
31. Tofighi, D., & Enders, C. K. (2007). Identifying the correct number of classes in growth mixture modeling. In G. R. Hancock (Ed.), Mixture models in latent variable research (pp. 317-341). Greenwich, CT: Information Age.
32. U.S. DepartmentofEducation. (2006).A test of leadership: Charting the future of U.S. higher education. Washington, DC: U.S. Department of Education. Retrieved November 22, 2009, from http://www. ed.gov/about/bdscomm/list/hiedfuture/index.html
33. Wainer, H. W. (1993). Measurement problems. Journal of Educational Measurement, 30(1), 1-21.
34. Wise, S. L. (2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19, 95-114.
35. Wise, S. L., & DeMars, C. E. (2005). Examinee motivation in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1-17.
36. Wise, S. L. & DeMars, C. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15, 27-41.
37. Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 162-183.
38. Wolf, L. F., Smith, J. K., & Birnbaum, M. E. (1995). Consequence of performance, test motivation, and mentally taxing items. Applied Measurement in Education, 8, 341-351.
39. Xin, Y., & Walker, K. (1997). Autonomy and accountabilityin higher education:An analysis of Chinese higher education reform. Canadian and International Education, 26, 9-27.
40. Yeo, G., & Neal, A. (2008). Subjective cognitive effort: A model of states, traits, and time. Journal of Applied Psychology, 93, 617-631.
41. Zilberberg, A., Brown, A. R., Harmes, J. C., & Anderson, R. D. (2009). How can we increase student motivation during low-stakes testing? Understanding the student perspective. In D. M. McInerney, G. T. L. Brown, & G. A. D. Liem (Eds.), Research on sociocultural influences on motivation and learning: Vol. 9. Student perspectives on assessment: What students can tell us about improving school outcomes (pp. 255-277). Greenwich, CT: Information Age.