Do college student surveys have any validity?

Review of Higher Education

Volumn 35, Issue 1, 2011, Pages 45-76

  Porter, Stephen R ^a  

^a NORTH CAROLINA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 80052554633     PISSN: 01625748     EISSN: 10907009     Source Type: Journal    
DOI: 10.1353/rhe.2011.0034     Document Type: Article

References (61)

1. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (1999). The standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2. Achen, C. H. (1977). Measuring representation: Perils of the correlation coefficient. American Journal of Political Science, 21(4), 805-815.
3. Achen, C. H. (1978). Measuring representation. American Journal of Political Science, 22(3), 475-510.
4. Astin, A. (1993). What matters in college? Four critical years revisited. San Francisco: Jossey-Bass.
5. Bowman, N. A. (2010). Can first-year students accurately report their learning and development? American Educational Research Journal, 47(2), 466-496.
6. Bowman, N. A. (in press-a). Measuring how college affects students: Social desirability and other potential biases in college student self-reported gains. In S. Herzog & N. A. Bowman (Eds.), The validity of college student self-report data. New Directions for Institutional Research. San Francisco: Jossey-Bass.
7. Bowman, N. A. (in press-b). The validity of college seniors' self-reported gains as a proxy for longitudinal growth. Educational Evaluation and Policy Analysis.
8. Bowman, N. A. (2011). The validity of college self-reported gains at diverse institutions. Educational Researcher, 40(1), 22-24.
9. Bradburn, N. M., Rips, L. J., & Shevell, S. K. (1987, April 10). Answering autobiographical questions: The impact of memory and inference on surveys. Science, pp. 157-161.
10. Brener, N. D., McManus, T., Galuska, D. A., Lowry, R., & Wechsler, H. (2003). Reliability and validity of self-reported height and weight among high school students. Journal of Adolescent Health, 32(4), 281-287.
11. Carini, R., Kuh, G. D., & Klein, S. P. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47(1), 1-32.
12. Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Thousand Oaks, CA: Sage.
13. Cole, J. S., & Gonyea, R. M. (2010). Accuracy of self-reported SAT and ACT test scores: Implications for research. Research in Higher Education, 51(4), 305-319.
14. DeMaio, T. J. (1984). Social desirability and survey measurement: A review. In C. F. Turner & E. Martin (Eds.), Surveying subjective phenomena (pp. 257-281). New York: Russell Sage Foundation.
15. DeVellis, R. F. (2003). Scale development: Theory and applications. Los Angeles: Sage Publications.
16. Fowler, F. L. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA: Sage Publications.
17. Garry, M., Sharman, S. J., Feldman, J., Marlatt, G. A., & Loftus, E. F. (2002). Examining memory for heterosexual college students' sexual experiences using an electronic mail diary. Health Psychology, 21(6), 629-634.
18. Garson, D. (2009). Scales and standard measures. Retrieved October 13, 2009, from http://faculty.chass.ncsu.edu/garson/PA765/standard.htm.
19. Gordon, J., Ludlum, J., & Hoey, J. J. (2008). Validating the NSSE against student outcomes: Are they related? Research in Higher Education, 49, 19-39.
20. Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646-675.
21. Groves, R. M., Fowler, F. L., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2004). Survey methodology. Hoboken, NJ: Wiley-Interscience.
22. Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167-189.
23. Kane, M. T. (1992). An argument-based approach to validation. Psychological Bulletin, 112, 527-535.
24. Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319-342.
25. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational Measurement (pp. 17-64). Westport, CT: American Council on Education and Praeger.
26. Klein, S. P., Kuh, G. D., Chun, M., Hamilton, L., & Shavelson, R. (2005). An approach to measuring cognitive outcomes across higher education institutions. Research in Higher Education, 46(3), 251-276.
27. Kozak, M. (2009). What is a strong correlation? Teaching Statistics, 31(3), 85-86.
28. Kreuter, F., Presser, S., & Tourangeau, R. (2008). Social desirability bias in CATI, IVR, and web surveys. Public Opinion Quarterly, 72(5), 847-865.
29. Kuh, G. D. (2004). The national survey of student engagement: Conceptual framework and overview of psychometric properties. Bloomington: Indiana University Center for Postsecondary Research.
30. Kuh, G. D., Hayek, J. C., Carini, R. M., Ouimet, J. A., Gonyea, R. M., & Kennedy, J. (2001). NSSE technical and norms report. Bloomington: Indiana University Center for Postsecondary Research and Planning.
31. Kuncel, N. R., Crede, M., & Thomas, L. L. (2005). The validity of self-reported grade point averages, class ranks, and test scores: A meta-analysis and review of the literature. Review of Educational Research, 75(1), 63-82.
32. LaNasa, S. M., Cabrera, A. F., & Transgrud, H. (2009). The construct validity of student engagement: A confirmatory factor analysis approach. Research in Higher Education, 50, 315-332.
33. LaNasa, S. M., Olson, E., & Alleman, N. (2007). The impact of on-campus student growth on first-year student engagement and success. Research in Higher Education, 48(8), 941-966.
34. Lissitz, R. W., & Samuelsen, K. (2007). A suggested change in terminology and emphasis regarding validity and education. Educational Researcher, 36(8), 437-448.
35. McCartney, K., & Rosenthal, R. (2000). Effect size, practical importance, and social policy for children. Child Development, 71(1), 173-180.
36. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (pp. 13-103). New York: Macmillan.
37. Nederhof, A.J. (1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology, 15(3), 263-280.
38. Niemi, R. G., & Smith, J. (2003). The accuracy of students' reports of course taking in the 1994 National Assessment of Educational Progress. Educational Measurement: Issues and Practice, 22(1), 15-21.
39. NSSE. National Survey of Student Engagement. (2007). Experiences that matter: Enhancing student learning and success. Bloomington: Indiana University Center for Postsecondary Research.
40. NSSE. National Survey of Student Engagement. (2009a). NSSEville State University: Frequency distributions. Retrieved October 13, 2009, from http://nsse.iub.edu/2009_Institutional_Report/pdf/NSSE09%20Frequency%20Distributions%20Report%20(NSSEville%20State).pdf.
41. NSSE. National Survey of Student Engagement. (2009b). NSSEville State University: Mean Comparisons. Retrieved October 13, 2009, from http://nsse.iub.edu/2009_Institutional_Report/pdf/NSSE09%20Mean%20Comparisons%20Report%20(NSSEville%20State).pdf.
42. NSSE. National Survey of Student Engagement. (2009c). Reliability. Retrieved October 13, 2009, from http://nsse.iub.edu/html/PsychometricPortfolio_Reliability.cfm.
43. NSSE. National Survey of Student Engagement. (2009d). Survey development. Retrieved October 13, 2009, from http://nsse.iub.edu/html/PsychometricPortfolio_SurveyDevelopment.cfm.
44. Olivas, M. A. (1986). Financial aid and self-reports by disadvantaged students: The importance of being earnest. Research in Higher Education, 25(3), 245-252.
45. Pace, R. C. (1985). The credibility of student self-reports. Los Angeles: Center for the Study of Evaluation, University of California Los Angeles.
46. Pace, R. C., & Friedlander, J. (1982). The meaning of response categories: How often is "occasionally," "often," and "very often"? Research in Higher Education, 17(3), 267-281.
47. Pascarella, E. T., & Seifert, T. A. (2008). Validation of the NSSE benchmarks and deep approaches to learning against liberal arts outcomes. Paper presented at the Association for the Study of Higher Education.
48. Pike, G. R. (1995). The relationships between self reports of college experiences and achievement test scores. Research in Higher Education, 36, 1-22.
49. Pike, G. R. (1999). The constant error of the halo in educational outcomes research. Research in Higher Education, 40(1), 61-86.
50. Porter, S. R. (2006). Institutional structures and student engagement. Research in Higher Education, 47(5), 521-558.
51. Porter, S. R., Rumann, C., & Pontius, J. (in press). The validity of student engagement survey questions: Can we accurately measure academic challenge? In S. Herzog & N. A. Bowman (Eds.), The validity of college student self-report data. New Directions for Institutional Research. San Francisco: Jossey-Bass.
52. Rosenthal, R., & Rosnow, R. L. (1991). Essentials of behavioral research: Methods and data analysis. Boston: McGraw-Hill.
53. Schwarz, N. (1990). Assessing frequency reports of mundane behaviors. In C. Hendrick & M. S. Clark (Eds.), Research methods in personality and social psychology (pp. 98-119). Newbury Park, CA: Sage.
54. Smyth, J. D., Dillman, D. A., & Christian, L. M. (2007). Context effects in web surveys: New issues and evidence. In A. Joinson, K. McKenna, T. Postmes, & U. Reips (Eds.), Oxford handbook of internet psychology (pp. 427-443). New York: Oxford University Press.
55. Spector, P. E. (1992). Summated rating scale construction: An Introduction. Newbury Park, CA: Sage.
56. Standards for Educational and Psychological Testing (1999). Washington, DC: American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education.
57. Stinebrickner, R., & Stinebrickner, T. R. (2004). Time-use and college outcomes. Journal of Econometrics, 121, 243-269.
58. Swerdzewski, P., Miller, B. J., & Mitchell, R. (2007). Investigating the validity of the National Survey of Student Engagement. Paper presented at the Northeastern Educational Research Association, Rocky Hill, CT.
59. Thompson, C. P. (1982). Memory for unique personal events: The roommate study. Memory and Cognition, 10(4), 324-332.
60. Tourangeau, R., Rips, L., & Rasinski, K. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.
61. Trusheim, D. (1994). How valid is self-reported financial aid information? Research in Higher Education, 35(3), 335-348.