Know Your Population and Know Your Model: Using Model-Based Regression and Poststratification to Generalize Findings Beyond the Observed Sample

Psychological Methods

Volumn 26, Issue 5, 2021, Pages 547-558

  Kennedy, Lauren ^a   Gelman, Andrew ^b  

^a MONASH UNIVERSITY   (Australia)

^b Columbia University ^*  (United States)

Author keywords

Bayesian statistics; Generalization; Multilevel models; Poststratification; Surveys

Indexed keywords

HUMAN; METHODOLOGY; PERSONALITY;

HUMANS; PERSONALITY; RESEARCH DESIGN;

EID: 85121846943     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000362     Document Type: Article

References (36)

1. Bürkner, P.-C. (2017). brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software, 80(1), 1–28. https://doi.org/10.18637/jss.v080.i01
2. Caughey, D., & Warshaw, C. (2019). Public opinion in subnational politics. University of Chicago Press. https://doi.org/10.1086/700723
3. Donnellan, M. B., & Lucas, R. E. (2008). Age differences in the Big Five across the life span: Evidence from two national samples. Psychology and Aging, 23(3), 558. https://doi.org/10.1037/a0012897
4. Downes, M., Gurrin, L. C., English, D. R., Pirkis, J., Currier, D., Spittal, M. J., & Carlin, J. B. (2018). Multilevel regression and poststratification: A modelling approach to estimating population quantities from highly selected survey samples. American Journal of Epidemiology, 187(8), 1780 –1790. https://doi.org/10.1093/aje/kwy070
5. Gabry, J., Simpson, D., Vehtari, A., Betancourt, M., & Gelman, A. (2019). Visualization in Bayesian workflow. Journal of the Royal Statistical Society: Series A, 182(2), 389 – 402. https://doi.org/10.1111/rssa.12378
6. Gao, Y., Kennedy, L., Simpson, D., & Gelman, A. (2020). Improving multilevel regression and poststratification with structured priors. arXiv.https://arxiv.org/abs/1908.06716
7. Gelman, A. (2005). Analysis of variance-why it is more important than ever. The Annals of Statistics, 33(1), 1–53. https://doi.org/10.1214/009053604000001048
8. Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press. https://doi.org/10.1017/CBO9780511790942
9. Gelman, A., & Little, T. C. (1997). Poststratification into many categories using hierarchical logistic regression. Survey Methodology, 23, 127–135. Gelman, A., & Pardoe, I. (2006). Bayesian measures of explained variance and pooling in multilevel (hierarchical) models. Technometrics, 48(2),241–251. https://doi.org/10.1198/004017005000000517
10. Ghitza, Y., & Gelman, A. (2013). Deep interactions with mrp: Election turnout and voting patterns among small electoral subgroups. American Journal of Political Science, 57(3), 762–776. https://doi.org/10.1111/ajps.12004
11. Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological Assessment, 4(1), 26. https://doi.org/10.1037/1040-3590.4.1.26
12. Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40(1), 84 –96. https://doi.org/10.1016/j.jrp.2005.08.007
13. Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilburg University Press.
14. Green, D. P., & Kern, H. L. (2012). Modeling heterogeneous treatment effects in survey experiments with Bayesian additive regression trees. Public Opinion Quarterly, 76(3), 491–511. https://psycnet.apa.org/doi/10.1093/poq/nfs036
15. Henry, P. J. (2008). College sophomores in the laboratory redux: Influences of a narrow data base on social psychology’s view of the nature of prejudice. Psychological Inquiry, 19(2), 49 –71. https://doi.org/10.1080/10478400802049936
16. Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1), 217–240. https://doi.org/10.1198/jcgs.2010.08162
17. International Personality Item Pool Project. (n.d.). 50-item sample questionnaire. Retrieved from https://ipip.ori.org/new_ipip-50-item-scale.htm
18. Lax, J. R., & Phillips, J. H. (2009a). Gay rights in the states: Public opinion and policy responsiveness. American Political Science Review, 103(3), 367–386.
19. Lax, J. R., & Phillips, J. H. (2009b). How should we estimate public opinion in the states? American Journal of Political Science, 53(1), 107–121. https://doi.org/10.1111/j.1540-5907.2008.00360.x
20. Little, R. J. (1993). Post-stratification: A modeler’s perspective. Journal of the American Statistical Association, 88(423), 1001–1012. https://doi.org/10.1080/01621459.1993.10476368
21. Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38(4), 379. https://doi.org/10.1037/0003-066X.38.4.379
22. Open Source Psychometrics Project. (2019). Open psychology data: Raw data from online personality tests. https://openpsychometrics.org/_rawdata/
23. Park, D. K., Gelman, A., & Bafumi, J. (2004). Bayesian multilevel estimation with poststratification: State-level estimates from national polls. Political Analysis, 12(4), 375–385.
24. Schad, D. J., Betancourt, M., & Vasishth, S. (2019). Toward a principled Bayesian workflow in cognitive science. arXiv. https://arxiv.org/abs/1904.12765
25. Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51(3), 515. https://doi.org/10.1037/0022-3514.51.3.515
26. Si, Y., Trangucci, R., Gabry, J. S., & Gelman, A. (2017). Bayesian hierarchical weighting adjustment and survey inference. arXiv. https://arxiv.org/pdf/1707.08220.pdf
27. Simons, D. J. (2013). Unskilled and optimistic: Overconfident predictions despite calibrated knowledge of relative skill. Psychonomic Bulletin & Review, 20(3), 601– 607. https://doi.org/10.3758/s13423-013-0379-2
28. Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630
29. Smith, P. L., & Little, D. R. (2018). Small is beautiful: In defense of the small-N design. Psychonomic Bulletin & Review, 25(6), 2083–2101. https://doi.org/10.3758/s13423-018-1451-8
30. Sorensen, T., Hohenstein, S., & Vasishth, S. (2016). Bayesian linear mixed models using stan: A tutorial for psychologists, linguists, and cognitive scientists. Quantitative Methods for Psychology, 12(3), 175–200. https://doi.org/10.20982/tqmp.12.3.p175
31. United States Census Bureau. (2012). American Community Survey (ACS): PUMS. https://factfinder.census.gov/faces/nav/jsf/pages/searchresults.xhtml?refresh=t#
32. Wang, W., Rothschild, D., Goel, S., & Gelman, A. (2015). Forecasting elections with non-representative polls. International Journal of Forecasting, 31(3), 980 –991.
33. Weber, S., Gelman, A., Lee, D., Betancourt, M., Vehtari, A., & Racine-Poon, A. (2018). Bayesian aggregation of average data: An application in drug development. The Annals of Applied Statistics, 12(3), 1583– 1604. https://doi.org/10.1214/17-AOAS1122
34. Weisberg, Y. J., DeYoung, C. G., & Hirsh, J. B. (2011). Gender differences in personality across the ten aspects of the Big Five. Frontiers in Psychology, 2, 178. https://doi.org/10.3389/fpsyg.2011.00178
35. Whitsett, D. D., & Shoda, Y. (2014). An approach to test for individual differences in the effects of situations without using moderator variables. Journal of Experimental Social Psychology, 50, 94 –104. https://doi.org/10.1016/j.jesp.2013.08.008
36. Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., Paunesku, D., Romero, C., Flint, K., Roberts, A., Trott, J., Iachan, R., Buontempo, J., Yang, S. M., Carvalho, C. M.,… Dweck C. S. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774), 364 –369. https://doi.org/10.1038/s41586-019-1466-y