Assessing the Effectiveness of a College Freshman Seminar Using Propensity Score Adjustments

Research in Higher Education

Volumn 52, Issue 6, 2011, Pages 616-639

  Clark, M H ^a   Cundiff, Nicole L ^b  

^a UNIVERSITY OF CENTRAL FLORIDA   (United States)

^b UNIVERSITY OF ALASKA FAIRBANKS   (United States)

Author keywords

Academic success; College retention; First year seminar; Propensity scores; Selection bias

Indexed keywords

EID: 79960584014     PISSN: 03610365     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11162-010-9208-x     Document Type: Article

References (90)

1. ACT. (2006). Retrieved March 12, 2009, from National Collegiate Retention and Persistence to Degree Rates Web site: http://www. act. org/research/policymakers/pdf/retain_2006. pdf.
2. Aitken, N. D. (1982). College student performance, satisfaction and retention: Specification and estimation of a structural model. The Journal of Higher Education, 53, 32-50.
3. Allen, J., & Robbins, S. (2010). Effects of interest-major congruence, motivation and academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23-35.
4. Allison, P. (2001). Missing data. Thousand Oaks, CA: Sage Publications.
5. Astin, A. W. (1993). What matters most in college? Four critical years revisited. San Francisco: Jossey-Bass.
6. Austin, P. C., Grootendorst, P., & Anderson, G. M. (2007). A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: A Monte Carlo study. Statistics in Medicine, 26, 734-753.
7. Barefoot, B. O. (2000). The first-year experience: Are we making it any better? About Campus, 4(6), 12-18.
8. Barefoot, B. O., & Fidler, P. P. (1992). The 1991 national survey of freshman seminar programming (Monograph no. 10). Columbia, SC: National Resource Center for the Freshman Year Experience, University South Carolina.
9. Beck, A. T., & Beck, R. W. (1972). Screening depressed patients in family practice: A rapid technic. Postgraduate Medicine, 52, 81-85.
10. Burgette, J., & Magun-Jackson, S. (2009). Freshman orientation, persistence, and achievement: A longitudinal analysis. Journal of College Retention: Research, Theory and Practice, 10(3), 235-263.
11. Busato, V. V., Prins, F. J., Elshout, J. J., & Hamaker, C. (2000). Intellectual ability, learning style, personality, achievement motivation and academic success among psychology students in higher education. Personality and Individual Differences, 29, 1057-1068.
12. Cabrera, A. F., Nora, A., & Castaneda, M. B. (1993). College persistence: Structural equations modeling test of an integrated model of student retention. The Journal of Higher Education, 64, 123-139.
13. Cepeda, M. S., Boston, R., Farrar, J. T., & Strom, B. L. (2003). Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders. American Journal of Epidemiology, 158, 280-287.
14. Cochran, W. G., & Rubin, D. B. (1973). Controlling bias in observational studies: A review. Sankhya, Series A, 35, 417-446.
15. Cokley, K. O. (2003). What do we know about the motivation of African American students? Challenging the "anti-intellectual" myth. Harvard Educational Review, 73, 524-558.
16. DeBerard, M. S., Spielmans, G. I., & Julka, D. C. (2004). Predictors of academic achievement and retention among college freshman: A longitudinal study. College Student Journal, 38, 66-80.
17. Engberg, M. E., & Mayhew, M. J. (2007). The influence of first-year "success" courses on student learning and democratic outcomes. Journal of College Student Development, 48, 241-258.
18. Fox, R. N. (1984). Reliability and discriminant validity of institutional integration scales for disadvantaged college students. Educational and Psychological Measurement, 44, 1051-1057.
19. Friedman, D. B., & Alexander, J. S. (2007). Investigating a first-year seminar as an anchor course in learning communities. Journal of the First-Year Experience & Students in Transition, 19, 63-74.
20. Gibson, C. M. (2003). Privileging the participant: The importance of sub-group analysis in social welfare evaluations. American Journal of Evaluation, 24, 443-469.
21. Gifford, D. D., Briceno-Perriott, J., & Mianzo, F. (2006). Locus of control: Academic achievement and retention in a sample of university first-year students. Journal of College Admission, 191, 18-25.
22. Ginter, E. J., & Dwinell, P. L. (1994). The importance of perceived duration: Loneliness and its relationship to self-esteem and academic performance. Journal of College Student Development, 35, 456-460.
23. Goldberg, L. R. (1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7-28). Tilburg: Tilburg University Press.
24. Goldberg, L. R. (2004). Big-five factor markers derived from the IPIP item pool (50 items). International personality item pool: A scientific collaboratory for the development of advanced measures of personality and other individual differences. Retrieved July 17, 2004, from http://ipip. ori. org/newQform50b5. htm.
25. Gow, A. J., Whiteman, M. C., Pattie, A., & Deary, I. J. (2005). Goldberg's 'IPIP' Big-Five factor markers: Internal consistency and concurrent validation in Scotland. Personality and Individual Difference, 39, 317-329.
26. Grayson, J. P. (1988). Racial origin and student retention in a Canadian University. Higher Education, 36, 323-352.
27. Grunwald, H. E., & Mayhew, M. J. (2008). Using propensity scores for estimating causal effects: A study in the development of moral reasoning. Research in Higher Education, 49, 758-775.
28. Gu, X. S., & Rosenbaum, P. R. (1993). Comparison of multivariate matching methods: Structures, distances, and algorithms. Journal of Computational and Graphical Statistics, 2, 405-420.
29. Handel, D. D. (2001, April). The relative contribution of participating in a first-year seminar on student satisfaction and retention into the sophomore year. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA. (ERIC Document Reproduction Service No. ED453724).
30. Harackiewicz, J. M., Barron, K. E., Tauer, J. M., & Elliot, A. J. (2002). Predicting success in college: A longitudinal study of achievement goals and ability measures as predictors of interest and performance from freshman year through graduation. Journal of Educational Psychology, 94, 562-575.
31. Hirano, K., & Imbens, G. W. (2001). Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcomes Research Methodology, 2, 259-278.
32. Hunter, M. S. (Summer, 2006). Fostering student learning and success through first-year programs. Association of American Colleges and Universities, 1-7.
33. Jamelske, E. (2009). Measuring the impact of a university first-year experience program on student GPA and retention. Higher Education, 57, 373-391.
34. Keup, J. R. (2005). The impact of curricular interventions on intended second year re-enrollment. Journal of College Student Retention: Research, Theory and Practice, 7, 61-89.
35. Komarraju, M., Karau, S., & Schmeck, R. (2009). Role of the Big Five personality traits in predicting college students' academic motivation and achievement. Learning and Individual Differences, 19, 47-52.
36. Koutsoubakis, D. (1999). A test of the effectiveness of a one-term freshman orientation program at the foreign campus of an accredited private American university. Journal of the First-Year Experience, 11, 33-58.
37. Krause, K. L., Hartley, R., James, R., & McInnis, C. (2005). The first year experience in Australian universities: Findings from a decade of national studies. Centre for the Study of Higher Education University of Melbourne, Government report.
38. Krum, T. E., Costley, L. E., & Mattanah, J. (2009). Longitudinally predicting attrition among first and second year college students. Unpublished manuscript, Towson University, Towson, MD.
39. Kurth, T., Walker, A. M., Glynn, R. J., Chan, K. A., Gaziano, J. M., Berger, K., et al. (2006). Results of multivariable logistic regression, propensity matching, propensity adjustment, and propensity-based weighting under conditions of nonuniform effect. American Journal of Epidemiology, 163, 262-270.
40. Lane, D. J., & Gibbons, F. X. (2007). Am I the typical student? Perceived similarity to student prototypes predicts success. Personality and Social Psychology Bulletin, 33, 1380-1391.
41. Lang, D. J. (2007). The impact of a first-year experience course on the academic performance, persistence, and graduation rates of first-semester college students at a public research university. Journal of the First-Year Experience & Students in Transition, 19, 9-25.
42. Leow, C., Marcus, S., Zanutto, E., & Boruch, R. (2004). Effects of advanced course-taking on math and science achievement: Addressing selection bias using propensity scores. American Journal of Evaluation, 25, 461-478.
43. McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9, 403-425.
44. McInnis, C. (2001). Researching the first year experience: Where to from here? Higher Education Research & Development, 20, 105-114.
45. McKenzie, K., Gow, K., & Schweitzer, R. (2004). Exploring first-year academic achievement through structural equation modeling. Higher Education Research and Development, 23, 95-112.
46. McKenzie, K., & Schweitzer, R. (2001). Who succeeds at University? Factors predicting academic performance in first year Australian students. Higher Education Research and Development, 20, 21-33.
47. Miller, J. W., Janz, J. C., & Chen, C. (2007). The retention impact of a first-year seminar on students with varying pre-college academic performance. Journal of the First-Year Experience & Students in Transition, 19, 47-62.
48. Napoli, A. R., & Wortman, P. M. (1998). Psychosocial factors related to retention and early departure of two-year community college students. Research in Higher Education, 39, 419-455.
49. National Resource Center for the First-Year Experience and Students in Transition and the Policy Center on the First Year of College. (2009). Retrieved March 20, 2009, from National Resource Center for the First-Year Experience and Students in Transition Web site: http://www. sc. edu/fye/index. html.
50. Nes, L. S., Evans, D. R., & Segerstrom, S. C. (2009). Optimism and college retention: Mediation by motivation, performance and adjustment. Journal of Applied Social Psychology, 39, 1887-1912.
51. Nicpon, M. F., Huser, L., Blanks, E. H., Sollenberger, S., Befort, C., & Robinson Kurpius, S. E. (2006). The relationship of loneliness and social support with college freshmen's academic performance and persistence. Journal of College Student Retention: Research, Theory and Practice, 8, 345-358.
52. Noble, K., Flynn, N. T., Lee, J. D., & Hilton, D. (2007). Predicting successful college experiences: Evidence from a first year retention program. Journal of College Student Retention: Research, Theory and Practice, 9, 39-60.
53. Nora, A., Cabrera, A., Hagedorn, L., & Pascarella, E. (1996). Differential impacts of academic and social experiences on college-related behavioral outcomes across different ethnic and gender groups at four-year institutions. Research in Higher Education, 37, 427-451.
54. Norman, D. S. (2003). Loneliness and its effect of schooling and school achievement. Dissertation Abstracts International, 64(10), 3585. (UMI No. 3107462).
55. Pascarella, E. T., & Terenzini, P. T. (1980). Predicting freshman persistence and voluntary dropout decisions from a theoretical model. Journal of Higher Education, 51, 60-75.
56. Peterson, S. L. (1993). Career decision making self-efficacy and institutional integration of underprepared college students. Research in Higher Education, 34, 659-685.
57. Peterson, E. D., Pollack, C. V., Roe, M. T., Parsons, L. S., Littrell, K. A., Canto, J. G., et al. (2003). Early use of glycoprotein IIb/IIIa inhibitors in non-ST-elevation acute myocardial infarction: Observations from the National Registry of Myocardial Infarction 4. Journal of the American College of Cardiology, 2, 45-53.
58. Phillips, G. A. (1997). The utility and distinctiveness of the MMPI-2 College Maladjustment Scale. Unpublished master's thesis, Western Carolina University.
59. Pickett, R. F. (2006). The importance of first-year experience courses. Unpublished manuscript, Southern Illinois University, Carbondale.
60. Porter, S. R., & Swing, R. L. (2006). Understanding how first-year seminars affect persistence. Research in Higher Education, 47, 89-109.
61. Reynolds, W. M., & Gould, J. W. (1981). A psychometric investigation of the standard and short form of the Beck Depression Inventory. Journal of Consulting and Clinical Psychology, 49, 306-307.
62. Rosenbaum, P. R. (1995). Observational studies. New York: Springer.
63. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55.
64. Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516-524.
65. Roweton, W. E. (1994). Predicting rural college retention among first-year undergraduates. (ERIC Document Reproduction Service No. ED370501).
66. Rubin, D. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology, 2, 169-188.
67. Russell, D. W. (1996). UCLA Loneliness Scale (version 3): Reliability, validity, and factor structure. Journal of Personality Assessment, 66, 20-40.
68. Satterfield, C. D. (1999). Effects of institutional integration and career decision making self efficacy on academic persistence among college freshmen. Dissertation Abstracts International, 61, 886.
69. Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of American Statistical Association, 103, 1334-1344.
70. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
71. Shah, B. R., Laupacis, A., Hux, J. E., & Austin, P. C. (2005). Propensity score methods gave similar results to traditional regression modeling in observational studies: A systematic review. Journal of Clinical Epidemiology, 58, 550-559.
72. Southern Illinois University. (2006a, August). University 101 course goal. Retrieved January 9, 2009, from Southern Illinois University's University 101 Web site: http://www. siu. edu/~u101/goal. html.
73. Southern Illinois University. (2006b, August). University 101: Program overview. Retrieved January 9, 2009, from Southern Illinois University's University 101 Web site: http://www. siu. edu/~u101/programoverview. html.
74. Southern Illinois University. (2008). Undergraduate catalog 2008-2009. Carbondale, IL: Southern Illinois University Carbondale.
75. Strauss, L. C., & Volkwein, J. F. (2004). Predictors of student commitment at two-year and four-year institutions. The Journal of Higher Education, 75, 203-227.
76. Stürmer, T., Joshi, M., Glynn, R. J., Avorn, J., Rothman, K. J., & Schneeweiss, S. (2006). A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. Journal of Clinical Epidemiology, 59, 437-447.
77. Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45, 89-125.
78. Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago: University of Chicago.
79. Tinto, V. (1998). Colleges as communities: Taking research on student persistence seriously. The Review of Higher Education, 21, 167-177.
80. Tobolowsky, B. F. (2006). Preliminary summary of results from the 2006 national survey on first-year seminars. Retrieved March 9, 2009, from National Resource Center for the First Year Experience & Students in Transition, University South Carolina Web site: http://www. sc. edu/fye/research/surveyfindings/surveys/survey06. html.
81. Tobolowsky, B. F., Mamrick, M., & Cox, B. E. (2005). The 2003 national survey of first-year seminars: Continuing innovations in the collegiate curriculum (Monograph no. 41). Columbia, SC: National Resource Center for the First Year Experience & Students in Transition, University South Carolina.
82. Vallerand, R. J., & Bissonnette, R. (1992). Intrinsic, extrinsic, and amotivational styles as predictors of behavior: A prospective study. Journal of Personality, 60, 599-620.
83. Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senècal, C., & Vallières, E. F. (1992). The Academic Motivation Scale: A measure of intrinsic, extrinsic, and amotivation in education. Educational and Psychological Measurement, 52, 1003-1017.
84. Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senècal, C. B., & Vallières, E. F. (1993). On the assessment of intrinsic, extrinsic, and amotivation in education: Evidence on the concurrent and construct validity of the Academic Motivation Scale. Educational and Psychological Measurement, 53, 159-172.
85. Vallerand, R. J., Pelletier, L. G., Blais, M. R., Brière, N. M., Senècal, C. B., & Vallières, E. F. (2004). The Academic Motivation Scale (AMS-C 28) college version. Research Laboratory on Social Behavior. Retrieved http://www. er. uqam. ca/nobel/r26710/LRCS/echelles_en. htm.
86. Vansteenkiste, M., Simons, J., Lens, W., Sheldon, K. M., & Deci, E. L. (2004). Motivating learning, performance, and persistence: The synergistic effects of intrinsic goal contents and autonomy-supportive contexts. Journal of Personality and Social Psychology, 87, 246-260.
87. Viswesvaran, C., & Ones, D. S. (2000). Measurement error in "Big Five factors" personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224-235.
88. Wolfe, R. N., & Johnson, S. D. (1995). Personality as a predictor of college performance. Educational and Psychological Measurement, 55, 177-185.
89. Wolf-Wendel, L. E., Tuttle, K., & Keller-Wolff, C. M. (1999). Assessment of a freshman summer transition program in an open-admissions institution. Journal of the First-Year Experience, 11, 7-32.
90. Yockey, F. A., & George, A. A. (1998). The effects of a freshman seminar paired with supplemental instruction. Journal of the First-Year Experience, 10, 57-76.