Comparing the use of open and closed questions for Web-based measures of the continued-influence effect

Behavior Research Methods

Volumn 51, Issue 3, 2019, Pages 1426-1440

  Connor Desai, Saoirse ^a   Reimers, Stian ^a  

^a CITY UNIVERSITY   (United Kingdom)

Author keywords

Closed ended; Continued influence effect; Misinformation; Open ended; Response formats; Web based

Indexed keywords

ADULT; ANIMAL EXPERIMENT; ARTICLE; BEHAVIORAL SCIENCE; CONTROLLED STUDY; FEMALE; MALE; MOUSE; NONHUMAN; COMPARATIVE STUDY; HUMAN; INTERNET; INTERPERSONAL COMMUNICATION; MIDDLE AGED; STATISTICAL BIAS; WRITING; YOUNG ADULT;

ADULT; BIAS; COMMUNICATION; FEMALE; HUMANS; INTERNET; MALE; MIDDLE AGED; WRITING; YOUNG ADULT;

EID: 85049000949     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-018-1066-z     Document Type: Article

References (47)

1. Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800–813. doi:10.3758/s13428-011-0081-0
2. Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014). Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science, 58, 739–753. doi:10.1111/ajps.12081
3. Cappella, J. N., Ophir, Y., & Sutton, J. (2018). The importance of measuring knowledge in the age of misinformation and challenges in the tobacco domain. In B. G. Southwell, E. A. Thorson, & L. Sheble (Eds.), Misinformation and mass audiences (pp. 51–70). Austin, TX: University of Texas Press.
4. Chan, M. S., Jones, C. R., Hall Jamieson, K., & Albarracín, D. (2017). Debunking: A meta-analysis of the psychological efficacy of messages countering misinformation. Psychological Science, 28, 1531–1546. doi:10.1177/0956797617714579
5. Connor Desai, S., & Reimers, S. (2017). But where’s the evidence? The effect of explanatory corrections on inferences about false information. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. Davelaar (Eds.), Proceedings of the 39th Annual Conference of the Cognitive Science Society (pp. 1824–1829). Austin, TX: Cognitive Science Society.
6. Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8, e57410. doi:10.1371/journal.pone.0057410
7. Ecker, U. K. H., Hogan, J. L., & Lewandowsky, S. (2017). Reminders and repetition of misinformation: Helping or hindering its retraction? Journal of Applied Research in Memory and Cognition, 6, 185–192.
8. Ecker, U. K. H., Lewandowsky, S., & Apai, J. (2011b). Terrorists brought down the plane!—No, actually it was a technical fault: processing corrections of emotive information. Quarterly Journal of Experimental Psychology, 64, 283–310. doi:10.1080/17470218.2010.497927
9. Ecker, U. K. H., Lewandowsky, S., Swire, B., & Chang, D. (2011a). Correcting false information in memory: Manipulating the strength of misinformation encoding and its retraction. Psychonomic Bulletin & Review, 18, 570–578. doi:10.3758/s13423-011-0065-1
10. Ecker, U. K. H., Lewandowsky, S., & Tang, D. T. W. (2010). Explicit warnings reduce but do not eliminate the continued influence of misinformation. Memory & Cognition, 38, 1087–1100. doi:10.3758/MC.38.8.1087
11. Fazio, L. K., Brashier, N. M., Payne, B. K., & Marsh, E. J. (2015). Knowledge does not protect against illusory truth. Journal of Experimental Psychology: General, 144, 993–1002. doi:10.1037/xge0000098
12. Fisher, R. P., Brewer, N., & Mitchell, G. (2009). The relation between consistency and accuracy of eyewitness testimony: Legal versus cognitive explanations. In R. Bull, T. Valentine, & T. Williamson (Eds.), Handbook of psychology of investigative interviewing: Current developments and future directions (pp. 121–136). Hoboken, NJ: Wiley. doi:10.1002/9780470747599.ch8
13. Frew, E. J., Whynes, D. K., & Wolstenholme, J. L. (2003). Eliciting willingness to pay: Comparing closed-ended with open-ended and payment scale formats. Medical Decision Making, 23, 150–159. doi:10.1177/0272989X03251245
14. Frew, E. J., Wolstenholme, J. L., & Whynes, D. K. (2004). Comparing willingness-to-pay: Bidding game format versus open-ended and payment scale formats. Health Policy, 68, 289–298.
15. Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012). Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review, 19, 847–857. doi:10.3758/s13423-012-0296-9
16. Gordon, A., Brooks, J. C. W., Quadflieg, S., Ecker, U. K. H., & Lewandowsky, S. (2017). Exploring the neural substrates of misinformation processing. Neuropsychologia, 106, 216–224. doi:10.1016/j.neuropsychologia.2017.10.003
17. Graesser, A., Ozuru, Y., & Sullins, J. (2010). What is a good question? In M. McKeown & G. Kucan (Eds.), Bringing reading research to life (pp. 112–141). New York, NY: Guilford.
18. Grysman, A. (2015). Collecting narrative data on Amazon’s Mechanical Turk. Applied Cognitive Psychology, 29, 573–583.
19. Guillory, J. J., & Geraci, L. (2013). Correcting erroneous inferences in memory: The role of source credibility. Journal of Applied Research in Memory and Cognition, 2, 201–209. doi:10.1016/j.jarmac.2013.10.001
20. Guillory, J. J., & Geraci, L. (2016). The persistence of erroneous information in memory: The effect of valence on the acceptance of corrected information. Applied Cognitive Psychology, 30, 282–288. doi:10.1002/acp.3183
21. Jacoby, L. L. (1996). Dissociating automatic and consciously controlled effects of study/test compatibility. Journal of Memory and Language, 35, 32–52.
22. Johnson, H. M., & Seifert, C. M. (1994). Sources of the continued influence effect: When misinformation in memory affects later inferences. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 1420–1436. doi:10.1037/0278-7393.20.6.1420
23. Johnson-Laird, P. N. (1980). Mental models in cognitive science. Cognitive Science, 4, 71–115. doi:10.1207/s15516709cog0401_4
24. Khoe, W., Kroll, N. E., Yonelinas, A. P., Dobbins, I. G., & Knight, R. T. (2000). The contribution of recollection and familiarity to yes–no and forced-choice recognition tests in healthy subjects and amnesics. Neuropsychologia, 38, 1333–1341.
25. Krippendorff, K. (2012). Content analysis: An introduction to its methodology. New York, NY: Sage.
26. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.
27. Lee, M. D., & Wagenmakers, E.-J. (2014). Bayesian cognitive modeling: A practical course. Cambridge, UK: Cambridge University Press.
28. Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012). Misinformation and its correction: Continued influence and successful debiasing. Psychological Science in the Public Interest, 13, 106–131. doi:10.1177/1529100612451018
29. Moons, W. G., Mackie, D. M., & Garcia-Marques, T. (2009). The impact of repetition-induced familiarity on agreement with weak and strong arguments. Journal of Personality and Social Psychology, 96, 32–44. doi:10.1037/a0013461
30. Morey, R. D., & Rouder, J. N. (2015). BayesFactor: Computation of Bayes factors for common designs. Retrieved from https://cran.r-project.org/package=BayesFactor
31. Ozuru, Y., Briner, S., Kurby, C. A., & McNamara, D. S. (2013). Comparing comprehension measured by multiple-choice and open-ended questions. Canadian Journal of Experimental Psychology, 67, 215–227. doi:10.1037/a0032918
32. Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163.
33. Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031. doi:10.3758/s13428-013-0434-y
34. Reimers, S., & Stewart, N. (2007). Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities. Behavior Research Methods, 39, 365–370. doi:10.3758/BF03193004
35. Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 47, 309–327. doi:10.3758/s13428-014-0471-1
36. Reja, U., Manfreda, K. L., Hlebec, V., & Vehovar, V. (2003). Open-ended vs. close-ended questions in Web questionnaires. Developments in Applied Statistics, 19, 159–177. Retrieved from www.websm.org/uploadi/editor/Reja_2003_open_vs._close-ended_questions.pdf
37. Rich, P. R., & Zaragoza, M. S. (2016). The continued influence of implied and explicitly stated misinformation in news reports. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42, 62–74. doi:10.1037/xlm0000155
38. Schwarz, N., Hippler, H. J., Deutsch, B., & Strack, F. (1985). Response scales—Effects of category range on reported behavior and comparative judgments. Public Opinion Quarterly, 49, 388–395. doi:10.1086/268936
39. Schwarz, N., Knauper, B., Hippler, H. J., Neumann, B., & Clark, L. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570–582.
40. Schwarz, N., Strack, F., Müller, G., & Chassein, B. (1988). The range of response alternatives may determine the meaning of the question: Further evidence on informative functions of response alternatives. Social Cognition, 6, 107–117.
41. Shapiro, L. R. (2006). The effects of question type and eyewitness temperament on accuracy and quantity of recall for a simulated misdemeanor crime. Emporia State Research Studies, 43, 1–7.
42. Simcox, T., & Fiez, J. A. (2014). Collecting response times using amazon mechanical turk and adobe flash. Behavior Research Methods, 46, 95–111. doi:10.3758/s13428-013-0345-y
43. Stewart, N., Chandler, J., & Paolacci, G. (2017). Crowdsourcing samples in cognitive science. Trends in Cognitive Sciences, 21, 736–748. doi:10.1016/j.tics.2017.06.007
44. Wilkes, A. L., & Leatherbarrow, M. (1988). Editing episodic memory following the identification of error. Quarterly Journal of Experimental Psychology, 40A, 361–387. doi:10.1080/02724988843000168
45. Wolfe, C. R. (2017). Twenty years of Internet-based research at SCiP: A discussion of surviving concepts and new methodologies. Behavior Research Methods, 49, 1615–1620. doi:10.3758/s13428-017-0858-x
46. Yonelinas, A. P. (2002). The nature of recollection and familiarity: A review of 30 years of research. Journal of Memory and Language, 46, 441–517. doi:10.1006/jmla.2002.2864
47. Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (2017). Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review. 10.3758/s13423-017-1348-y