Improving web survey quality: Potentials and constraints of propensity score adjustments

Online Panel Research: A Data Quality Perspective

Volumn , Issue , 2014, Pages 273-298

  Steinmetz, Stephanie ^a   Bianchi, Annamaria ^b   Tijdens, Kea ^a,c   Biffignandi, Silvia ^b  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b UNIVERSITY OF BERGAMO   (Italy)

^c ERASMUS UNIVERSITY ROTTERDAM   (Netherlands)

Author keywords

Nonprobability web surveys; Propensity score adjustment; Representativeness; Volunteer web surveys; Webographic variables

Indexed keywords

EID: 84924959852     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9781118763520.ch12     Document Type: Chapter

References (62)

1. Baker, R., Blumberg, S., Brick, J., Couper, M., Courtright, M., Dennis, J., Dillman, D., et al. (2010). Research synthesis: AAPOR report on online panels. Public Opinion Quarterly, 74, 711-781.
2. Baker, R., Brick,M., Bates, N., Battaglia,M., Couper, M., Dever, J., Gile, K., et al. (2013, May). Report of the AAPOR task-force on nonprobability sampling. Retrieved from: http://www.aapor.org/AM /Template.cfm?Section=Reports1&Template=/CM/ContentDisplay.cfm&ContentID=5963.
3. Bandilla, W., Bosnjak, M., & Altdorfer, P. (2003). Survey administration effects? A comparison of web-based and traditional written self-administered surveys using the ISSP environment module. Social Science Computer Review, 21, 235-243.
4. Berk, R., & Newton, P. (1985). Does arrest really deter wife battering? An effort to replicate the findings of the Minneapolis Spouse Abuse Experiment. American Sociological Review, 50, 253-262.
5. Berkhout, E., & Salverda,W. (2012). Development of the public-private wage differential in the Netherlands, 1979-2009. SEO Discussion Paper/AIAS Working Paper. University of Amsterdam.
6. Bethlehem, J. (2009). Applied Survey Methods: A Statistical Perspective. Hoboken, NJ: Wiley-Interscience.
7. Bethlehem, J. (2010). Selection bias in web surveys. International Statistical Review, 78, 161-188.
8. Bethlehem, J. & Biffignandi, S., (2012). Handbook of Web Surveys. Hoboken, United States: Wiley.
9. Bethlehem, J., & Stoop, I. (2007). Online panels: a paradigm theft? InM. Trotman et al. (Eds.), Proceedings of the Fifth International Conference of the Association for Survey Computing: The challenges of a changing world (pp. 113-131). Southampton: Association for Survey Computing.
10. Bianchi, A., & Biffignandi, S. (2013). Web Panel Representativeness. In P. Giudici, S. Ingrassia, & M. Vichi (Eds.), Statistical Models for Data Analysis. Series: Studies in Classification, Data Analysis, and Knowledge Organization, XV (pp. 37-44). Cham: Springer.
11. Bosnjak, M., & Tuten, T. (2003). Prepaid and promised incentives in web surveys. Social Science Computer Review, 21, 208-217.
12. Brick, J. (2011). The future of survey sampling. Public Opinion Quarterly, 75, 872-888.
13. Cochran, W. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24, 295-313.
14. Couper, M. (2000). Web surveys: a review of issues and approaches. Public Opinion Quarterly, 64, 464-481.
15. Couper, M., Kapteyn, A., Schonlau, M., &Winter, J. (2007). Noncoverage and nonresponse in an Internet survey. Social Science Research, 36, 131-148.
16. Couper, M., & Miller, P. (2008). Web survey methods: Introduction. Public Opinion Quarterly, 72, 831-835.
17. De Pedraza, P., Tijdens, K., & Muñoz de Bustillo, R. (2007). Simple bias, weights and efficiency of weights in a continuous voluntary web survey. AIASWorking PaperWP 07-58, Amsterdam Institute for Advanced Labour Studies.
18. Deville, J., & Särndal, C. (1992). Calibration estimation in survey sampling. Journal of the American Statistical Association, 87, 376-382.
19. de Vos, K. (2010). Representativeness of the LISS-panel 2008, 2009, 2010. Tilburg: CentERdata, University of Tilburg.
20. Dillman, D. (2006). Why choice of survey mode makes a difference. Public Health Reports, 121, 11-13.
21. Drake, C. (1993). Effects of misspecification of the propensity score on estimators of treatment effect. Biometrics, 49, 1231-1236.
22. Duffy, B., Smith, K., Terhanian, G., & Bremer, J. (2005). Comparing data from online and face-to-face surveys. International Journal of Market Research, 47, 615-639.
23. Duncan, K., & Stasny, E. (2001). Using propensity scores to control coverage bias in telephone surveys. Survey Methodology, 27, 121-130.
24. Fricker, R. (2008). Sampling methods for web and E-mail surveys. In N. Fielding, R. Lee, & G. Blank (Eds.), The SAGE Handbook of online research methods (pp. 195-216). London: Sage.
25. Fricker, R., & Schonlau, M. (2002). Advantages and disadvantages of Internet research surveys: Evidence from the literature. Field Methods, 14, 347-367.
26. Göritz, A. (2006). Cash lotteries as incentives in online panels. Social Science Computer Review, 24, 445-459.
27. Groves, R. (2004). Survey errors and survey costs. Hoboken, NJ: Wiley-Interscience.
28. Groves, R. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70, 646-675.
29. Groves, R., & Lyberg, L. (2010). Total survey error: Past, present, and future. Public Opinion Quarterly, 74, 849-879.
30. Heckman, J., LaLonde, R., & Smith, J. (1999). The economics and econometrics of Active Labour Market Programs. In O. Ashenfelter, & D. Card (Eds.), Handbook of Labor Economics (pp. 1865-2097, Vol. 3a). Amsterdam: Elsevier.
31. Heerwegh, D. (2005). Effects of personal salutations in e-mail invitations to participate in a web survey. Public Opinion Quarterly, 69, 588-598.
32. Horvitz, D., & Thompson, D. (1952). A generalization of sampling without replacement from finite universe. Journal of the American Statistical Association, 47, 663-685.
33. Klassen, R., & Jacobs, J. (2001). Experimental comparison of web, electronic and mail survey technologies in operations management. Journal of Operations Management, 19, 713-728.
34. Knoef, M., & de Vos, K. (2009). The representativeness of LISS, an online probability panel. Tilburg:CentERdata, University of Tilburg.
35. Kruskal, W., & Mosteller, F. (1979). Representative sampling, II: Scientific literature, excluding statistics. International Statistical Review, 47, 113-127.
36. Lee, S. (2006). Propensity Score Adjustment as a weighting scheme for volunteer panel web surveys. Journal of Official Statistics, 22, 329-349.
37. Lee, S., & Vaillant, R. (2009). Estimation for volunteer panel web surveys using propensity score adjustment and calibration adjustment. Sociological Methods Research, 37, 319-343.
38. Lensvelt-Mulders, G., Lugtig, P.,&Hubregtse, M. (2009). Separating selection bias and non-coverage in Internet panels using propensity matching. Retrieved September 2009 from: www.surveypractice.org.
39. Loosveldt, G., & Sonck, N. (2008). An evaluation of the weighting procedures for online access panel surveys. Survey Research Methods, 2, 93-105.
40. Lozar Manfreda, K., Bosnjak, M., Berzelak, J., Haas, I., & Vehovar, V. (2008). Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research, 50, 79-104.
41. Lynn, P. (2008). The problem of nonresponse. In E. de Leeuw, J. Hox,&D.Dillman (Eds.), International Handbook of Survey Methodology (pp. 35-55). Hillsdale, NJ: Lawrence Erlbaum Associates.
42. Malhotra, N., & Krosnick, J. (2007). The effect of survey mode and sampling on inferences about political attitudes and behaviour: Comparing the 2000 and 2004 ANES to Internet surveys with nonprobability samples. Political Analysis, 15, 286-324.
43. Meng, X., Duan, N., Chen, C., & Algeria, M. (2009). Power-shrinkage: An alternative for dealing with excessive weights. Joint Statistical Meetings.
44. Rosenbaum, P., & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55.
45. Rosenbaum, P., & Rubin, D. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516-524.
46. Scherpenzeel, A., & Bethlehem, J. (2011). How representative are online-panels? Problems of coverage and selection and possible solutions. In M. Das, P. Ester, & L. Kaczmirek (Eds.), Social and behavioral research and the internet: Advances in applied methods and research strategies (pp. 105-132). New York: Routledge.
47. Schonlau, M., van Soest, A., & Kapteyn, A. (2007). Are "Webographic" or attitudinal questions useful for adjusting estimates from Web surveys using propensity scoring? Survey Research Methods, 1, 155-163.
48. Schonlau, M., van Soest, A., Kapteyn, A., & Couper, M. (2009). Selection bias in web surveys and the use of propensity scores. Sociological Methods Research, 37, 291-318.
49. Schonlau, M., Zapert, K., Simon, L., Sanstad, K., Marcus, S., Adams, J., et al. (2004). A comparison between a propensity weighted web survey and an identical RDD survey. Social Science Computer Review, 22, 128-138.
50. Shih, T.-H., & Fan, X. (2008). Comparing response rates from web and mail surveys: A meta-analysis. Field Methods, 20, 249-271.
51. Steinmetz, S., Raess, D., Tijdens, K., & de Pedraza, P. (2013). MeasuringWagesWorldwide: Exploring the potentials and Constraints of VolunteerWeb Surveys. In N. Sappleton (Ed.), Advancing Research Methods with New Technologies (pp. 100-119). Hershey, PA: IGI Global.
52. Steinmetz, S., Tijdens, K.,&de Pedraza, P. (2009). Comparing different weighting procedures for volunteer online panels: Lessons to be learned from German and DutchWageIndicator data. AIASWorking Paper WP 09-76. Amsterdam Institute for Advanced Labour Studies.
53. Taylor, H. (2005). Does Internet research "work"? Comparing online survey results with telephone surveys. International Journal of Market Research, 42, 51-63.
54. Terhanian, G., & Bremer, J. (2012). A smarter way to select respondents for surveys? International Journal of Market Research, 54, 751-780.
55. Terhanian, G., Bremer, J., Smith, R., & Thomas, R. (2000). Correcting data from online survey for the effects of non-random selection and non-random assignment. Research Paper: Rochester, NY: Harris Interactive.
56. Tijdens, K., van Zijl, S., Hughie-Williams,M., van Klaveren. M., & Steinmetz, S. (2010). Codebook and explanatory note on the WageIndicator dataset: A worldwide, continuous, multilingual web-survey on work and wages with paper supplements. AIASWorking PaperWP 10-102. Amsterdam Institute for Advanced Labour Studies.
57. Tuten, T., Urban, D., & Bosnjak, M. (2002). Internet surveys and data quality: A review. In B. Batinic, U. Reips, & M. Bosnjak (Eds.), Online social science (pp. 7-26). Seattle, WA: Hogrefe and Huber Publishers.
58. Valliant, R., & Dever, J. (2011). Estimating propensity adjustments for volunteer web surveys. Sociological Methods, 40, 105-137.
59. Van der Laan, J., (2009). Representativity of the LISS panel. CBS Discussion paper (09041). The Hague:Statistics Netherlands (CBS).
60. Vehovar, V., Batagelj, Z., Manfreda, K., & Zaletel, M. (2002). Nonresponse in web surveys. In R. Groves, D. Dillman, J. Eltinge, & R. Little (Eds.), Survey Nonresponse (pp. 229-242). New York:John Wiley & Sons, Inc..
61. Vehovar, V., & Manfreda, K. (1999). Web surveys: Can the weighting solve the problem? In: Proceedings of the Survey Research Methods Section, American Statistical Association (1999) (pp. 962-967). Retrieved from: http://www.amstat.org/sections/srms/Proceedings/y1999f.html.
62. Yeager, D., Krosnick, J., Chang, L., Javitz, H., Levendusky, M., Simpser, A., &Wang, R. (2011). Comparing the accuracy of RDD telephone surveys and Internet surveys conducted with probability and nonprobability samples. Public Opinion Quarterly, 75, 709-747.