Examination of the equivalence of self-report survey-based paper-and-pencil and internet data collection methods

Psychological Methods

Volumn 18, Issue 1, 2013, Pages 53-70

  Weigold, Arne ^a   Weigold, Ingrid K ^b   Russell, Elizabeth J ^b  

^a Notre Dame College   (United States)

^b UNIVERSITY OF AKRON   (United States)

Author keywords

Auxiliary equivalence; Equivalence testing; Internet; Qualitative equivalence; Quantitative equivalence

Indexed keywords

ADOLESCENT; ADULT; ARTICLE; COMPARATIVE STUDY; FEMALE; HUMAN; INFORMATION PROCESSING; INTERNET; MALE; METHODOLOGY; MIDDLE AGED; PSYCHOMETRY; QUESTIONNAIRE; SELF REPORT; STANDARD; UTILIZATION REVIEW; YOUNG ADULT;

ADOLESCENT; ADULT; DATA COLLECTION; FEMALE; HUMANS; INTERNET; MALE; MIDDLE AGED; PSYCHOMETRICS; QUESTIONNAIRES; SELF REPORT; YOUNG ADULT;

EID: 84883006620     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0031607     Document Type: Article

References (95)

1. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
2. Bäckström, M., Björklund, F., & Larsson, M. R. (2009). Five-factor inventories have a major general factor related to social desirability which can be reduced by framing items neutrally. Journal of Research in Personality, 43, 335-344. doi:10.1016/j.jrp.2008.12.013
3. Bandilla, W., Bosnjak, M., & Altdorfer, P. (2003). Survey administration effects? A comparison of Web-based and traditional written selfadministered surveys using the ISSP environment module. Social Science Computer Review, 21, 235-243. doi:10.1177/0894439303021002009
4. Barak, A., & Cohen, L. (2002). Empirical investigation of an online version of the Self-Directed Search. Journal of Career Assessment, 10, 387-400. doi:10.1177/1069072702238402
5. Barbeite, F. G., & Weiss, E. M. (2004). Computer self-efficacy and anxiety scales for an Internet sample: Testing measurement equivalence of existing measures and development of new scales. Computers in Human Behavior, 20, 1-15. doi:10.1016/S0747-5632(03)00049-9
6. Barger, S. D. (2002). The Marlowe-Crowne affair: Short forms, psychometric structure, and social desirability. Journal of Personality Assessment, 79, 286-305. doi:10.1207/S15327752JPA7902_11
7. Berends, M. (2006). Survey methods in educational research. In J. L. Green, G. Camilli, & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 623-640). Mahwah, NJ: Erlbaum.
8. Bethlehem, J. (2010). Selection bias in web surveys. International Statistical Review, 78, 161-188. doi:10.1111/j.1751-5823.2010.00112.x
9. Boyer, K. K., Olson, J. R., Calantone, R. J., & Jackson, E. C. (2002). Print versus electronic surveys: A comparison of two data collection methodologies. Journal of Operations Management, 20, 357-373. doi:10.1016/S0272-6963(02)00004-9
10. Buchanan, T. (2002). Online assessment: Desirable or dangerous? Professional Psychology: Research and Practice, 33, 148-154. doi:10.1037/0735-7028.33.2.148
11. Buchanan, T. (2003). Internet-based questionnaire assessment: Appropriate use in clinical contexts. Cognitive Behaviour Therapy, 32, 100-109. doi:10.108016506070310000957
12. Buchanan, T. (2007). Personality testing on the Internet: What we know, and what we do not. In A. Joinson, K. McKenna, T. Postmes, & U.-D. Reips (Eds.), Oxford handbook of Internet psychology (pp. 447-459). Oxford, England: Oxford University Press.
13. Buchanan, T., Ali, T., Heffernan, T. M., Ling, J., Parrott, A. C., Rodgers, J., & Scholey, A. B. (2005). Nonequivalence of on-line and paper-andpencil psychological tests: The case of the prospective memory questionnaire. Behavior Research Methods, 37, 148-154. doi:10.3758/BF03206409
14. Buchanan, T., Johnson, J. A., & Goldberg, L. R. (2005). Implementing a five-factor personality inventory for use on the Internet. European Journal of Psychological Assessment, 21, 115-127. doi:10.1027/1015-5759.21.2.115
15. Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144. doi:10.1348/000712699161189
16. Cantrell, M. A., & Lupinacci, P. (2007). Methodological issues in online data collection. Journal of Advanced Nursing, 60, 544-549. doi: 10.1111/j.1365-2648.2007.04448.x
17. Carini, R. M., Hayek, J. C., Kuh, G. D., Kennedy, J. M., & Ouimet, J. A. (2003). College student responses to Web and paper surveys: Does mode matter? Research in Higher Education, 44, 1-19. doi:10.1023/A: 1021363527731
18. Church, A. H. (2001). Is there a method to our madness? The impact of data collection methodology on organizational survey results. Personnel Psychology, 54, 937-969. doi:10.1111/j.1744-6570.2001.tb00238.x
19. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159. doi:10.1037/0033-2909.112.1.155
20. Cohen, J. (1994). The earth is round (p <. 05). American Psychologist, 49, 997-1003. doi:10.1037/0003-066X.49.12.997
21. Cole, M. S., Bedeian, A. G., & Feild, H. S. (2006). The measurement equivalence of web-based and paper-and-pencil measures of transformational leadership: A multinational test. Organizational Research Methods, 9, 339-368. doi:10.1177/1094428106287434
22. Compeau, D. R., & Higgins, C. A. (1995). Computer self-efficacy: Development of a measure and initial test. MIS Quarterly, 19, 189-211. doi:10.2307/249688
23. Costa, P. T., Jr., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO-PI-qR) and NEO Five-Factor Inventory (NEO-FFI): Professional manual. Odessa, FL: Psychological Assessment Resources.
24. Cribbie, R. A., Gruman, J. A., & Arpin-Cribbie, C. A. (2004). Recommendations for applying tests of equivalence. Journal of Clinical Psychology, 60, 1-10. doi:10.1002/jclp.10217
25. Cronk, B. C., & West, J. L. (2002). Personality research on the Internet: A comparison of web-based and traditional instruments in take-home and in-class settings. Behavior Research Methods, Instruments & Computers, 34, 177-180. doi:10.3758/BF03195440
26. Davis, R. N. (1999). Web-based administration of a personality questionnaire: Comparison with traditional methods. Behavior Research Methods, Instruments & Computers, 31, 572-577. doi:10.3758/BF03200737
27. Diener, E. (2009). Positive psychology: Past, present, and future. In S. J. Lopez & C. R. Snyder (Eds.), Oxford handbook of positive psychology (pp. 7-11). New York, NY: Oxford University Press.
28. Dolnicar, S., Laesser, C., & Matus, K. (2009). Online versus paper: Format effects in tourism surveys. Journal of Travel Research, 47, 295-316. doi:10.1177/0047287508326506
29. Drummond, R. J., & Jones, K. D. (2010). Assessment procedures for counselors and helping professionals (7th ed.). Boston, MA: Pearson.
30. Epstein, J., Klinkenberg, W. D., Wiley, D., & McKinley, L. (2001). Insuring sample equivalence across Internet and paper-and-pencil assessments. Computers in Human Behavior, 17, 339-346. doi:10.1016/S0747-5632(01)00002-4
31. Evans, D. C., Garcia, D. J., Garcia, D. M., & Baron, R. S. (2003). In the privacy of their own homes: Using the Internet to assess racial bias. Personality and Social Psychology Bulletin, 29, 273-284. doi:10.1177/0146167202239052
32. Fischer, D. G., & Fick, C. (1993). Measuring social desirability: Short forms of the Marlowe-Crowne Social Desirability Scale. Educational and Psychological Measurement, 53, 417-424. doi:10.1177/0013164493053002011
33. Fouladi, R. T., McCarthy, C. J., & Moller, N. P. (2002). Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment. Assessment, 9, 204-215. doi:10.1177/10791102009002011
34. Fraley, R. C., & Marks, M. J. (2007). The null hypothesis significancetesting debate and its implications for personality research. In R. W. Robins, R. C. Fraley, & R. F. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 149-169). New York, NY: Guilford Press.
35. Goldberg, L. R. (1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe, (Vol. 7, pp. 7-28). Tilburg, the Netherlands: Tilburg University Press.
36. Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006). The International Personality Item Pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84-96. doi:10.1016/j.jrp.2005.08.007
37. Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust Web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 93-104. doi: 10.1037/0003-066X.59.2.93
38. Harrison, A. W., & Rainer, R. K. (1992). An examination of the factor structures and concurrent validities for the Computer Attitude Scale, the Computer Anxiety Rating Scale, and the Computer Self-Efficacy Scale. Educational and Psychological Measurement, 52, 735-745. doi: 10.1177/0013164492052003024
39. Heath, N. M., Lawyer, S. R., & Rasmussen, E. B. (2007). Web-based versus paper-and-pencil course evaluations. Teaching of Psychology, 34, 259-261. doi:10.1080/00986280701700433
40. Herrero, J., & Meneses, J. (2006). Short Web-based versions of the Perceived Stress (PSS) and Center for Epidemiological Studies-Depression (CES-D) Scales: A comparison to pencil and paper responses among Internet users. Computers in Human Behavior, 22, 830-846. doi:10.1016/j.chb.2004.03.007
41. Hewson, C., & Charlton, J. P. (2005). Measuring health beliefs on the Internet: A comparison of paper and Internet administrations of the Multidimensional Health Locus of Control Scale. Behavior Research Methods, 37, 691-702. doi:10.3758/BF03192742
42. Huang, H.-M. (2006). Do print and web surveys provide the same results? Computers in Human Behavior, 22, 334-350. doi:10.1016/j.chb.2004.09.012
43. International Personality Item Pool. (2011). A scientific collaboratory for the development of advanced measures of personality and other individual differences. Retrieved from http://ipip.ori.org/
44. Joinson, A. (1999). Social desirability, anonymity, and Internet-based questionnaires. Behavior Research Methods, Instruments & Computers, 31, 433-438. doi:10.3758/BF03200723
45. Khorrami-Arani, O. (2001). Researching computer self-efficacy. International Education Journal, 2, 17-25.
46. Kiernan, N. E., Kiernan, M., Oyler, M. A., & Gilles, C. (2005). Is a web survey as effective as a mail survey? A field experiment among computer users. American Journal of Evaluation, 26, 245-252. doi:10.1177/1098214005275826
47. Kramer, G. P., Bernstein, D. A., & Phares, V. (2009). Introduction to clinical psychology (7th ed.). Upper Saddle River, NJ: Pearson.
48. Lewis, I. M., Watson, B. C., & White, K. M. (2009). Internet versus paper-and-pencil survey methods in psychological experiments: Equivalence testing of participant responses to health-related messages. Australian Journal of Psychology, 61, 107-116. doi:10.1080/00049530802105865
49. McCabe, S. E. (2004). Comparison of web and mail surveys in collecting illicit drug use data: A randomized experiment. Journal of Drug Education, 34, 61-72. doi:10.2190/4HEY-VWXL-DVR3-HAKV
50. McCoy, C. (2010). Perceived self-efficacy and technology proficiency in undergraduate college students. Computers & Education, 55, 1614-1617. doi:10.1016/j.compedu.2010.07.003
51. McDonald, H., & Adam, S. (2003). A comparison of online and postal data collection methods in marketing research. Marketing Intelligence & Planning, 21, 85-95. doi:10.1108/02634500310465399
52. McGrath, R. E. (2011). Quantitative models in psychology. Washington, DC: American Psychological Association. doi:10.1037/12316-000
53. McGraw, K. O., Tew, M. D., & Williams, J. E. (2000). The integrity of web-delivered experiments: Can you trust the data? Psychological Science, 11, 502-506. doi:10.1111/1467-9280.00296
54. Meyerson, P., & Tryon, W. W. (2003). Validating Internet research: A test of the psychometric equivalence of Internet and in-person samples. Behavior Research Methods, Instruments & Computers, 35, 614-620. doi:10.3758/BF03195541
55. Miyazaki, A. D., & Taylor, K. A. (2008). Researcher interaction biases and business ethics research: Respondent reactions to researcher characteristics. Journal of Business Ethics, 81, 779-795. doi:10.1007/s10551-007-9547-5
56. Murphy, C. A., Coover, D., & Owen, S. V. (1989). Development and validation of the Computer Self-Efficacy Scale. Educational and Psychological Measurement, 49, 893-899. doi:10.1177/001316448904900412
57. Naus, M. J., Philipp, L. M., & Samsi, M. (2009). From paper to pixels: A comparison of paper and computer formats in psychological assessment. Computers in Human Behavior, 25, 1-7. doi:10.1016/j.chb.2008.05.012
58. Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241-301. doi:I0.1037//1082-989X.S.2.241
59. Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002). E-research: Ethics, security, design, and control in psychological research on the Internet. Journal of Social Issues, 58, 161-176. doi:10.1111/1540-4560.00254
60. Patterson, S. D., & Jones, B. (2006). Bioequivalence: A review of study design and statistical analysis for orally administered products. International Journal of Pharmaceutical Medicine, 20, 243-250. doi:10.2165/00124363-200620040-00004
61. Pettit, F. A. (2002). A comparison of World-Wide Web and paper-andpencil personality questionnaires. Behavior Research Methods, Instruments & Computers, 34, 50-54. doi:10.3758/BF03195423
62. Ployhart, R. E., Weekley, J. A., Holtz, B. C., & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733-752. doi:10.1111/j.1744-6570.2003.tb00757.x
63. Preckel, F., & Thiemann, H. (2003). Online-versus paper-pencil-version of a high potential intelligence test. Swiss Journal of Psychology, 62, 131-138. doi:10.1024//1421-0185.62.2.131
64. Reynolds, W. M. (1982). Development of reliable and valid short forms of the Marlowe-Crowne Social Desirability Scale. Journal of Clinical Psychology, 38, 119-125. doi:10.1002/1097-4679(198201)38:1<119:: AID-JCLP2270380118>3.0.CO;2-I
65. Richman, W. L., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computeradministered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754-775. doi:10.1037/0021-9010.84.5.754
66. Robitschek, C., Ashton, M. W., Spering, C. C., Geiger, N., Byers, D., Schotts, C., & Thoen, M. A. (2012). Development and psychometric evaluation of the Personal Growth Initiative Scale-II. Journal of Counseling Psychology, 59, 274-287. doi:10.1037/a0027310
67. Rogers, J. L., Howard, K. I., & Vessey, J. T. (1993). Using significance tests to evaluate equivalence between two experimental groups. Psychological Bulletin, 113, 553-565. doi:10.1037/0033-2909.113.3.553
68. Rosenthal, R. (1966). Experimenter effects in behavioral research. New York, NY: Appleton-Century-Crofts.
69. Rusticus, S. A., & Lovato, C. Y. (2011). Applying tests of equivalence for multiple group comparisons: Demonstration of the confidence interval approach. Practical Assessment, Research & Evaluation, 16, 1-6.
70. Schuirmann, D. J. (1987). Acomparison of the two one-sided tests procedure and the power approach for assessing equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15, 657-680.
71. Schulenberg, S. E., & Yutrzenka, B. A. (1999). The equivalence of computerized and paper-and-pencil psychological instruments: Implications for measures of negative affect. Behavior Research Methods, Instruments & Computers, 31, 315-321. doi:10.3758/BF03207726
72. Schulenberg, S. E., & Yutrzenka, B. A. (2001). Equivalence of computerized and conventional versions of the Beck Depression Inventory-II (BDI-II). Current Psychology, 20, 216-230. doi:10.1007/s12144-001-1008-1
73. Seaman, M. A., & Serlin, R. C. (1998). Equivalence confidence intervals for two-group comparisons of means. Psychological Methods, 3, 403-411. doi:10.1037/1082-989X.3.4.403
74. Seligman, M. E. P., Steen, T. A., Park, N., & Peterson, C. (2005). Positive psychology progress: Empirical validation of interventions. American Psychologist, 60, 410-421. doi:10.1037/0003-066X.60.5.410
75. Sethuraman, R., Kerin, R. A., & Cron, W. L. (2005). A field study comparing online and offline data collection methods for identifying product attribute preferences using conjoint analysis. Journal of Business Research, 58, 602-610. doi:10.1016/j.jbusres.2003.09.009
76. Shih, T.-H., & Fan, X. (2008). Comparing response rates from Web and mail surveys: A meta-analysis. Field Methods, 20, 249-271. doi: 10.1177/1525822X08317085
77. Smith, B., Smith, T. C., Gray, G. C., & Ryan, M. A. K. (2007). When epidemiology meets the Internet: Web-based surveys in the Millennium Cohort Study. American Journal of Epidemiology, 166, 1345-1354. doi:10.1093/aje/kwm212
78. Smith, M. A., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments & Computers, 29, 496-505. doi: 10.3758/BF03210601
79. Smither, J. W., Walker, A. G., & Yap, M. K. T. (2004). An examination of the equivalence of Web-based versus paper-and-pencil upward feedback ratings: Rater-and ratee-level analyses. Educational and Psychological Measurement, 64, 40-61. doi:10.1177/0013164403258429
80. Stanton, J. M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725. doi:10.1111/j.1744-6570.1998.tb00259.x
81. Steele, R. M., Mummery, W. K., & Dwyer, T. (2009). A comparison of face-to-face or Internet-delivered physical activity intervention on targeted determinants. Health Education & Behavior, 36, 1051-1064. doi:10.1177/1090198109335802
82. Stegner, B. L., Bostrom, A. G., & Greenfield, T. K. (1996). Equivalence testing for use in psychosocial and services research: An introduction with examples. Evaluation and Program Planning, 19, 193-198. doi: http://dx.doi.org/10.1016/0149-7189(96)00011-0
83. Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386. doi:10.1037/1082-989X.6.4.371
84. Tryon, W. W., & Lewis, C. (2008). An inferential confidence interval method of establishing statistical equivalence that corrects Tryon's (2001). reduction factor. Psychological Methods, 13, 272-277. doi: 10.1037/a0013158
85. Vallejo, M. A., Mañanes, G., Comeche, M. A., & Díaz, M. I. (2008). Comparison between administration via Internet and paper-and-pencil administration of two clinical instruments: SCL-90-R and GHQ-28. Journal of Behavior Therapy and Experimental Psychiatry, 39, 201-208. doi:10.1016/j.jbtep.2007.04.001
86. van der Linden, D., te Nijenhuis, J., & Bakker, A. B. (2010). The General Factor of Personality: A meta-analysis of Big Five intercorrelations and a criterion-related validity study. Journal of Research in Personality, 44, 315-327. doi:10.1016/j.jrp.2010.03.003
87. Van de Vijver, F. J. R., & Harsveld, M. (1994). The incomplete equivalence of the paper-and-pencil and computerized versions of the General Aptitude Test Battery. Journal of Applied Psychology, 79, 852-859. doi:10.1037/0021-9010.79.6.852
88. Voiskounsky, A. E. (2008). Flow experience in cyberspace: Current studies and perspectives. In A. Barak (Ed.), Psychological aspects of cyberspace: Theory, research, applications (pp. 70-101). New York, NY: Cambridge University Press.
89. Westlake, W. J. (1976). Symmetrical confidence intervals for bioequivalence trials. Biometrics, 32, 741-744. doi:10.2307/2529259
90. Whitaker, B. G. (2007). Internet-based attitude assessment: Does gender affect measurement equivalence? Computers in Human Behavior, 23, 1183-1194. doi:10.1016/j.chb.2004.11.016
91. Whittier, D. K., Seeley, S., & St. Lawrence, J. S. (2004). A comparison of Web-with paper-based surveys of gay and bisexual men who vacationed in a gay resort community. AIDS Education and Prevention, 16, 476-485. doi:10.1521/aeap.16.5.476.48735
92. Wiechmann, D., & Ryan, A. M. (2003). Reactions to computerized testing in selection contexts. International Journal of Selection and Assessment, 11, 215-229. doi:10.1111/1468-2389.00245
93. Wood, E., Nosko, A., Desmarais, S., Ross, C., & Irvine, C. (2006). Online and traditional paper-and-pencil survey administration: Examining experimenter presence, sensitive material and long surveys. Canadian Journal of Human Sexuality, 15, 147-155.
94. Zhang, Y. (2000). Using the Internet for survey research: A case study. Journal of the American Society for Information Science, 51, 57-68. doi:10.1002/(SICI)1097-4571(2000)51:1<57::AID-ASI9>3.0.CO;2-W
95. Zook, A., & Sipps, G. J. (1985). Cross-validation of a short form of the Marlowe-Crowne Social Desirability Scale. Journal of Clinical Psychology, 41, 236-238. doi:10.1002/1097-4679(198503)41:2<236::AIDJCLP2270410217 >3.0.CO;2-H