It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks

SAGE Open

Volumn 5, Issue 2, 2015, Pages

  Hauser, David J ^a   Schwarz, Norbert ^b  

^a UNIVERSITY OF MICHIGAN   (United States)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

instructional manipulation checks; research methods; social cognition; systematic thinking

Indexed keywords

EID: 84962188358     PISSN: None     EISSN: 21582440     Source Type: Journal    
DOI: 10.1177/2158244015584617     Document Type: Article

References (21)

1. Berinsky A. J., Margolis M. F., Sances M. W., (2014). Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science, 58, 739-753.
2. Chandler J., Mueller P., Paolacci G., (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112-130.
3. Clark H. H., Schober M. F., (1992). Asking questions and influencing answers. In Tanur J. M., (Ed.), Questions about questions: Inquiries into the cognitive bases of surveys (pp. 15-48). New York, NY: Russell Sage.
4. Denes-Raj V., Epstein S., (1994). Conflict between intuitive and rational processing: When people behave against their better judgment. Journal of Personality and Social Psychology, 66, 819-829.
5. Frederick S., (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25-42.
6. Goodman J. K., Cryder C. E., Cheema A., (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213-224.
7. Hauser D. J., Schwarz N., (in press). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods. Advance online publication.doi:10.3758/s13428-015-0578-z.
8. Krosnick J. A., (1999). Survey research. Annual Review of Psychology, 50, 537-567.
9. Nauts S., Langner O., Huijsmans I., Vonk R., Wigboldus D. H., (2014). Forming impressions of personality. Social Psychology, 45, 153-163.
10. Oppenheimer D. M., Meyvis T., Davidenko N., (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867-872.
11. Paolacci G., Chandler J., Ipeirotis P., (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.
12. Paxton J. M., Bruni T., Greene J. D., (2014). Are “counter-intuitive” deontological judgments really counter-intuitive? An empirical reply to Kahane et al. (2012). Social Cognitive and Affective Neuroscience, 9, 1368-1371.
13. Paxton J. M., Ungar L., Greene J. D., (2012). Reflection and reasoning in moral judgment. Cognitive Science, 36, 163-177.
14. Peer E., Vosgerau J., Acquisti A., (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023-1031.
15. Pinillos N. Á., Smith N., Nair G. S., Marchetto P., Mun C., (2011). Philosophy’s new challenge: Experiments and intentional action. Mind & Language, 26, 115-139.
16. Reis H. T., Gosling S. D., (2010). Social psychological methods outside the laboratory. In Fiske S. T., Gilbert D. T., Lindzey G., (Eds.), Handbook of social psychology (5th ed., Vol. 1, pp. 82-114). New York, NY: John Wiley.
17. Schwarz N., (1994). Judgment in a social context: Biases, shortcomings, and the logic of conversation. In Zanna M., (Ed.), Advances in experimental social psychology (Vol. 26, pp. 123-162). San Diego, CA: Academic Press.
18. Schwarz N., (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93-105.
19. Toplak M. E., West R. F., Stanovich K. E., (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39, 1275-1289.
20. West R. F., Meserve R. J., Stanovich K. E., (2012). Cognitive sophistication does not attenuate the bias blind spot. Journal of Personality and Social Psychology, 103, 506-519.
21. Wolf W., Levordashka A., Ruff J. R., Kraaijeveld S., Lueckmann J. M., Williams K. D., (2014). Ostracism Online: A social media ostracism paradigm. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-014-0475-x