Message pretesting using assessments of expected or perceived persuasiveness: Evidence about diagnosticity of relative actual persuasiveness

Journal of Communication

Volumn 68, Issue 1, 2018, Pages 120-142

  O’Keefe, Daniel J ^a  

^a NORTHWESTERN UNIVERSITY   (United States)

Author keywords

Formative Research; Message Pretesting.; Perceived Effectiveness (PE); Perceived Message Effectiveness (PME); Persuasion

Indexed keywords

ACCURACY ASSESSMENT; ASSESSMENT METHOD; COMMUNICATION; PERCEPTION; RESEARCH;

EID: 85046083383     PISSN: 00219916     EISSN: 14602466     Source Type: Journal    
DOI: 10.1093/joc/jqx009     Document Type: Article

References (88)

1. Abraham, C., & Kools, M. (Eds.). (2012). Writing health communication: An evidence-based guide. Los Angeles, CA: Sage.
2. Alhadreti, O., & Mayhew, P. (2017). To intervene or not to intervene: An investigation of three think-aloud protocols in usability testing. Journal of Usability Studies, 12, 111–132.
3. Andsager, J. L., Austin, E. W., & Pinkleton, B. E. (2001). Questioning the value of realism: Young adults’ processing of messages in alcohol-related public service announcements and advertising. Journal of Communication, 51, 121–142. doi:10.1111/j.1460-2466.2001. tb02875.x
4. Atkin, C. K., & Freimuth, V. (2013). Guidelines for formative evaluation research in campaign design. In R. E. Rice & C. K. Atkin (Eds.), Public communication campaigns (4th ed., pp. 53–68). Los Angeles, CA: Sage.
5. Barnes, A. J., Hanoch, Y., Miron-Shatz, T., & Ozanne, E. M. (2016). Tailoring risk communication to improve comprehension: Do patient preferences help or hurt? Health Psychology, 35, 1007–1016. doi:10.1037/hea0000367
6. *Barnett, M. A., Klassen, M., McMinimy, V., & Schwarz, L. (1987). The role of self- and other-oriented motivation in the organ donation decision. Advances in Consumer Research, 14, 335–337.
7. Bartlett, Y. K., Webb, T. L., & Hawley, M. S. (2017). Using persuasive technology to increase physical activity in people with chronic obstructive pulmonary disease by encouraging regular walking: A mixed-methods study exploring opinions and preferences. Journal of Medical Internet Research, 19, e124. doi:10.2196/jmir.6616
8. *Bhattacharjee, A., Berger, J., & Menon, G. (2014). When identity marketing backfires: Consumer agency in identity expression. Journal of Consumer Research, 41, 294–309. doi:10.1086/676125
9. Bigsby, E., Cappella, J. N., & Seitz, H. H. (2013). Efficiently and effectively evaluating public service announcements: Additional evidence for the utility of perceived effectiveness. Communication Monographs, 80, 1–23. doi:10.1080/03637751.2012.739706
10. Bogale, G. W., Boer, H., & Seydel, E. R. (2010). Reaching the hearts and minds of illiterate women in the Amhara highland of Ethiopia: Development and pre-testing of oral HIV/ AIDS prevention messages. SAHARA-J: Journal of Social Aspects of HIV/AIDS, 7(1), 2–9. doi:10.1080/17290376.2010.9724949
11. Booth-Butterfield, S., Welbourne, J., Williams, C., & Lewis, V. (2007). Formative field experiments of a NIOSH alert to reduce the risks to firefighters from structural collapse: Applying the cascade framework. Health Communication, 22, 79–88. doi:10.1080/ 10410230701310331
12. Borenstein, M., & Rothstein, H. (2005). Comprehensive meta-analysis (Version 2.2.023) [Computer software]. Englewood, NJ: Biostat.
13. Brennan, E., Durkin, S. J., Wakefield, M. A., & Kashima, Y. (2014). Assessing the effectiveness of antismoking television advertisements: Do audience ratings of perceived effectiveness predict changes in quitting intentions and smoking behaviours? Tobacco Control, 23, 412–418. doi:10.1136/tobaccocontrol-2012-050949
14. *Byrne, S., Katz, S. J., Mathios, A., & Niederdeppe, J. (2015). Do the ends justify the means? A test of alternatives to the FDA proposed cigarette warning labels. Health Communication, 30, 680–693. doi:10.1080/10410236.2014.895282
15. Cacioppo, J. T., Petty, R. E., & Morris, K. J. (1983). Effects of need for cognition on message evaluation, recall, and persuasion. Journal of Personality and Social Psychology, 45, 805–818. doi:10.1037/0022-3514.45.4.805
16. Chen, M., McGlone, M. S., & Bell, R. A. (2015). Persuasive effects of linguistic agency assignments and point of view in narrative health messages about colon cancer. Journal of Health Communication, 20, 977–988. doi:10.1080/10810730.2015.1018625
17. Choi, J., & Cho, H. (2016). Perceived message effectiveness, attitude toward messages, and perceived realism. In D. K. Kim & J. W. Dearing (Eds.), Health communication research measures (pp. 139–152). New York: Peter Lang.
18. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
19. Davis, K. C., Nonnemaker, J., Duke, J., & Farrelly, M. C. (2013). Perceived effectiveness of cessation advertisements: The importance of audience reactions and practical implications for media campaign planning. Health Communication, 28, 461–472. doi:10. 1080/10410236.2012.696535
20. Davis, K. C., Uhrig, J., Bann, C., Rupert, D., & Fraze, J. (2011). Exploring African American women’s perceptions of a social marketing campaign to promote HIV testing. Social Marketing Quarterly, 17(3), 39–60. doi:10.1080/15245004.2011.595536
21. Dillard, J. P. (2013). The effects of prior behavior on judgments of perceived message effectiveness: Evaluating HPV vaccine messages. Journal of Vaccines & Vaccination, 4, 193. doi:10.4172/2157-7560.1000193
22. Dillard, J. P., & Ha, Y. (2016). Interpreting perceived effectiveness: Understanding and addressing the problem of mean validity. Journal of Health Communication, 21, 1016–1022. doi:10.1080/10810730.2016.1204379
23. Dillard, J. P., Shen, L., & Vail, R. G. (2007). Does perceived message effectiveness cause persuasion or vice versa? 17 consistent answers. Human Communication Research, 33, 467–488. doi:10.1111/j.1468-2958.2007.00308.x
24. Dillard, J. P., Weber, K. M., & Vail, R. G. (2007). The relationship between the perceived and actual effectiveness of persuasive messages: A meta-analysis with implications for formative campaign research. Journal of Communication, 57, 613–631. doi:10.1111/j. 1460-2466.2007.00360.x
25. Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010). Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction, 17, article 18. doi:10.1145/ 1879831.1879836
26. Druckman, J. N., Peterson, E., & Slothuus, R. (2013). How elite partisan polarization affects public opinion formation. American Political Science Review, 107, 57–79. doi:10.1017/ S0003055412000500
27. Easterday, M. W., Rees Lewis, D., & Gerber, E. M. (2017). Designing crowdcritique systems for formative feedback. International Journal of Artificial Intelligence in Education, 27, 623–663. doi:10.1007/s40593-016-0125-9
28. *Falk, E. B., Berkman, E. T., & Lieberman, M. D. (2012). From neural responses to population behavior: Neural focus group predicts population-level media effects. Psychological Science, 23, 439–445. doi:10.1177/0956797611434964
29. *Falk, E. B., O’Donnell, M. B., Tompson, S., Gonzalez, R., Dal Cin, S., Strecher, V., … An, L. (2016). Functional brain imaging predicts public health campaign success. Social Cognitive and Affective Neuroscience, 11, 204–214. doi:10.1093/scan/nsv108
30. *Faulkner, M., & Kennedy, R. (2008). A new tool for pre-testing direct mail. International Journal of Market Research, 50, 469–490.
31. *Ganzach, Y., Weber, Y., & Ben-Or, P. (1997). Message framing and buying behavior: On the difference between artificial and natural environment. Journal of Business Research, 40, 91–95. doi:10.1016/S0148-2963(96)00208-1
32. Glynn, S. A., Williams, A. E., Nass, C. C., Bethel, J., Kessler, D., Scott, E. P., … Schreiber, G. B. (2003). Attitudes toward blood donation incentives in the United States: Implications for donor recruitment. Transfusion, 43, 7–16. doi:10.1046/j.1537-2995.2003.00252.x
33. *Goldsmith, K., & Dhar, R. (2013). Negativity bias and task motivation: Testing the effectiveness of positively versus negatively framed incentives. Journal of Experimental Psychology: Applied, 19, 358–366. doi:10.1037/a0034415
34. *Gollust, S. E., Niederdeppe, J., & Barry, C. L. (2013). Framing the consequences of childhood obesity to increase public support for obesity prevention policy. American Journal of Public Health, 103, e96–e102. doi:10.2105/AJPH.2013.301271
35. Haley, R. I., & Baldinger, A. L. (1991). The ARF Copy Research Validity Project. Journal of Advertising Research, 31(2), 11–32.
36. *Hansmann, R., Loukopoulos, P., & Scholz, R. W. (2009). Characteristics of effective battery recycling slogans: A Swiss field study. Resources, Conservation and Recycling, 53, 218–230. doi:10.1016/j.resconrec.2008.12.003
37. Healey, B., & Hoek, J. (2016). Young adult smokers’ and prior-smokers’ evaluations of novel tobacco warning images. Nicotine & Tobacco Research, 18, 93–97. doi:10.1093/ntr/ntv041
38. Hernandez, C., Wang, S., Abraham, I., Angulo, M. I., Kim, H., Meza, J. R., … Uddin, S. (2014). Evaluation of educational videos to increase skin cancer risk awareness and sun-safe behaviors among adult Hispanics. Journal of Cancer Education, 29, 563–569. doi:10.1007/s13187-014-0624-z
39. Hood, K. B., Shook, N. J., & Belgrave, F. Z. (2017). “Jimmy cap before you tap”: Developing condom use messages for African American women. The Journal of Sex Research, 54, 651–664. doi:10.1080/00224499.2016.1168351
40. *Hornikx, J. (2008). Comparing the actual and expected persuasiveness of evidence types: How good are lay people at selecting persuasive evidence? Argumentation, 22, 555–569. doi:10.1007/s10503-007-9067-6
41. *Hullett, C. R. (2000). The mediating role of values in value-expressive attitudes. Doctoral dissertation, ProQuest 9985400, Michigan State University, East Lansing, MI.
42. Hullett, C. R. (2002). Charting the process underlying the change of value-expressive attitudes: The importance of value-relevance in predicting the matching effect. Communication Monographs, 69, 158–178. doi:10.1080/714041711
43. Hullett, C. R. (2004). Using functional theory to promote sexually transmitted disease (STD) testing: The impact of value-expressive messages and guilt. Communication Research, 31, 363–396. doi:10.1177/0093650204266103
44. Jasek, J. P., Johns, M., Mbamalu, I., Auer, K., Kilgore, E. A., & Kansagra, S. M. (2015). One cigarette is one too many: Evaluating a light smoker-targeted media campaign. Tobacco Control, 24, 362–368. doi:10.1136/tobaccocontrol-2013-051348
45. Latimer, A. E., Krishnan-Sarin, S., Cavallo, D. A., Duhig, A., Salovey, P., & O’Malley, S. A. (2012). Targeted smoking cessation messages for adolescents. Journal of Adolescent Health, 50, 47–53. doi:10.1016/j.jadohealth.2011.04.013
46. *Leoniak, K. J., & Maj, K. (2016). A slice of hygiene: Justification and consequence in the persuasiveness of prescriptive and proscriptive signs. Social Influence, 11, 271–283. doi:10.1080/15534510.2016.1267663
47. *Lockwood, P., Marshall, T. C., & Sadler, P. (2005). Promoting success or preventing failure: Cultural differences in motivation by positive and negative role models. Personality and Social Psychology Bulletin, 31, 379–392. doi:10.1177/0146167204271598
48. Maddock, J. E., Silbanuz, A., & Reger-Nash, B. (2008). Formative research to develop a mass media campaign to increase physical activity and nutrition in a multiethnic state. Journal of Health Communication, 13, 208–216. doi:10.1080/10810730701807225
49. Malo, T. L., Gilkey, M. B., Hall, M. E., Shah, P. D., & Brewer, N. T. (2016). Messages to motivate human papillomavirus vaccination: National studies of parents and physicians. Cancer Epidemiology, Biomarkers & Prevention, 25, 1383–1391. doi:10.1158/1055-9965.EPI-16-0224
50. *McIntyre, P., Barnett, M. A., Harris, R. J., Shanteau, J., Skowronski, J. J., & Klassen, M. (1987). Psychological factors influencing decisions to donate organs. Advances in Consumer Research, 14, 331–334.
51. McLean, S. A., Paxton, S. J., Massey, R., Hay, P. J., Mond, J. M., & Rodgers, B. (2016). Identifying persuasive public health messages to change community knowledge and attitudes about bulimia nervosa. Journal of Health Communication, 21, 178–187. doi:10. 1080/10810730.2015.1049309
52. Mendez, R. D. R., Rodrigues, R. C., Spana, T. M., Cornélio, M. E., Gallani, M. C., & Pérez-Nebra, A. R. (2012). Validation of persuasive messages for the promotion of physical activity among people with coronary heart disease. Revista Latino-Americana de Enfermagem, 20, 1015–1023. doi:10.1590/S0104-11692012000600002
53. Morgan, M. G., Fischhoff, B., Bostrom, A., & Atman, C. J. (2002). Risk communication: A mental models approach. New York: Cambridge University Press.
54. Mowbray, F., Marcu, A., Godinho, C. A., Michie, S., & Yardley, L. (2016). Communicating to increase public uptake of pandemic flu vaccination in the UK: Which messages work? Vaccine, 34, 3268–3274. doi:10.1016/j.vaccine.2016.05.006
55. Noar, S. M., Hall, M. G., Francis, D. B., Ribisl, K. M., Pepper, J. K., & Brewer, N. T. (2016). Pictorial cigarette pack warnings: A meta-analysis of experimental studies. Tobacco Control, 25, 341–354. doi:10.1136/tobaccocontrol-2014-051978
56. Noar, S. M., Palmgreen, P., Zimmerman, R. S., Lustria, M. L. A., & Li, H.-Y. (2010). Assessing the relationship between perceived message sensation value and perceived message effectiveness: Analysis of PSAs from an effective campaign. Communication Studies, 61, 21–45. doi:10.1080/10510970903396477
57. *Nolan, J. M., Kenefick, J., & Schultz, P. W. (2011). Normative messages promoting energy conservation will be underestimated by experts … unless you show them the data. Social Influence, 6, 169–180. doi:10.1080/15534510.2011.584786
58. *Nolan, J. M., Schultz, P. W., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2008). Normative social influence is underdetected. Personality and Social Psychology Bulletin, 34, 913–923. doi:10.1177/0146167208316691
59. O’Keefe, D. J. (2002). Persuasion: Theory and research (2nd ed.). Thousand Oaks, CA: Sage.
60. O’Keefe, D. J. (2013). The relative persuasiveness of different message types does not vary as a function of the persuasive outcome assessed: Evidence from 29 meta-analyses of 2,062 effect sizes for 13 message variations. Annals of the International Communication Association, 37, 221–249. doi:10.1080/23808985.2013.11679151
61. *Paul, C. L., Redman, S., & Sanson-Fisher, R. W. (1997). The development of a checklist of content and design characteristics for printed health education materials. Health Promotion Journal of Australia, 7, 153–159.
62. *Paul, C. L., Redman, S., & Sanson-Fisher, R. W. (2003). Print material content and design: Is it relevant to effectiveness? Health Education Research, 18, 181–190. doi:10.1093/her/18.2.181
63. *Pechmann, C., Zhao, G. Z., Goldberg, M. E., & Reibling, E. T. (2003). What to convey in antismoking advertisements for adolescents: The use of protection motivation theory to identify effective message themes. Journal of Marketing, 67(2), 1–18. doi:10.1509/jmkg. 67.2.1.18607
64. *Pettigrew, S., Jongenelis, M., Chikritzhs, T., Slevin, T., Pratt, I. S., Glance, D., & Liang, W. B. (2014). Developing cancer warning statements for alcoholic beverages. BMC Public Health, 14, article no. 786. doi:10.1186/1471-2458-14-786
65. *Pettigrew, S., Jongenelis, M. I., Glance, D., Chikritzhs, T., Pratt, I. S., Slevin, T., … Wakefield, M. (2016). The effect of cancer warning statements on alcohol consumption intentions. Health Education Research, 31, 60–69. doi:10.1093/her/cyv067
66. *Piccolino, E. B. (1966). Depicted threat, realism, and specificity: Variables governing safety poster effectiveness. Doctoral dissertation, UMI no. 68-4473, Illinois Institute of Technology, Chicago, IL.
67. Pollard, C. M., Howat, P. A., Pratt, I. S., Boushey, C. J., Delp, E. J., & Kerr, D. A. (2016). Preferred tone of nutrition text messages for young adults: Focus group testing. JMIR mHealth and uHealth, 4, e1. doi:10.2196/mhealth.4764
68. Popova, L., Neilands, T. B., & Ling, P. M. (2014). Testing messages to reduce smokers’ openness to using novel smokeless tobacco products. Tobacco Control, 23, 313–321. doi:10.1136/tobaccocontrol-2012-050723
69. Riker, C. A., Butler, K. M., Ricks, J. M., Record, R. A., Begley, K., Anderson, D. G., & Hahn, E. J. (2015). Creating effective media messaging for rural smoke-free policy. Public Health Nursing, 32, 613–624. doi:10.1111/phn.12188
70. Rossiter, J. R., & Eagleson, G. (1994). Conclusions from the ARF’s copy research validity project. Journal of Advertising Research, 34(3), 19–32.
71. Santa, A. F., & Cochran, B. N. (2008). Does the impact of anti-drinking and driving public service announcements differ based on message type and viewer characteristics? Journal of Drug Education, 38, 109–129. doi:10.2190/DE.38.2.b
72. Siegel, J. T., Lienemann, B. A., & Rosenberg, B. D. (2017). Resistance, reactance, and misinterpretation: Highlighting the challenge of persuading people with depression to seek help. Social and Personality Psychology Compass, 11, e12322. doi:10.1111/spc3.12322
73. *Tal-Or, N., Shilo, S., & Meister, T. (2009). Third-person perception and purchase behavior in response to various selling methods. Psychology and Marketing, 26, 1091–1107. doi:10.1002/mar.20314
74. Taylor, R. E. (2015). The role of message strategy in improving hand hygiene compliance rates. American Journal of Infection Control, 43, 1166–1170. doi:10.1016/j.ajic.2015.06.015
75. *Thornton, B., Kirchner, G., & Jacobs, J. (1991). Influence of a photograph on a charitable appeal: A picture may be worth a thousand words when it has to speak for itself. Journal of Applied Social Psychology, 21, 433–445. doi:10.1111/j.1559-1816.1991.tb00529.x
76. *Trawalter, S., Driskell, S., & Davidson, M. N. (2016). What is good isn’t always fair: On the unintended effects of framing diversity as good. Analyses of Social Issues and Public Policy, 16, 69–99. doi:10.1111/asap.12103
77. Truong, K. N., Hayes, G. R., & Abowd, G. D. (2006). Storyboarding: An empirical determination of best practices and effective guidelines. In DIS’06: Proceedings of the 6th conference on designing interactive systems (pp. 12–21).
78. *Turner, M. M., Banas, J. A., Rains, S. A., Jang, S. A., Moore, J. L., & Morrison, D. (2010). The effects of altercasting and counterattitudinal behavior on compliance: A lost letter technique investigation. Communication Reports, 23, 1–13. doi:10.1080/ 08934211003598759
79. *Turner, M. M., Banas, J. A., Rains, S., Moore, J., & Jang, S. A. (2005, November). Testing the effects of altercasting, source status, and target liking on compliance using the lost letter technique. Paper presented at the annual conference of the National Communication Association, Boston, MA.
80. Volk, R. J., Kinsman, G. T., Le, Y.-C. L., Swank, P., Blumenthal-Barby, J., McFall, S. L., … Cantor, S. B. (2015). Designing normative messages about active surveillance for men with localized prostate cancer. Journal of Health Communication, 20, 1014–1020. doi:10. 1080/10810730.2015.1018618
81. *Weaver, K., Hock, S. J., & Garcia, S. M. (2016). “Top 10” reasons: When adding persuasive arguments reduces persuasion. Marketing Letters, 27, 27–38. doi:10.1007/s11002-014-9286-1
82. Webb, O. J., & Eves, F. F. (2007). Promoting stair climbing: Effects of message specificity and validation. Health Education Research, 22, 49–57. doi:10.1093/her/cyl045
83. Whittingham, J., Ruiter, R. A. C., Zimbile, F., & Kok, G. (2008). Experimental pretesting of public health campaigns: A case study. Journal of Health Communication, 13, 216–230. doi:10.1080/10810730701854045
84. Willoughby, J. F., & Furberg, R. (2015). Underdeveloped or underreported? Coverage of pretesting practices and recommendations for design of text message-based health behavior change interventions. Journal of Health Communication, 20, 472–478. doi:10. 1080/10810730.2014.977468
85. *Wogalter, M. S., Begley, P. B., Scancorelli, L. F., & Brelsford, J. W. (1997). Effectiveness of elevator service signs: Measurement of perceived understandability, willingness to comply and behaviour. Applied Ergonomics, 28, 181–187. doi:10.1016/S0003-6870(96) 00063-4
86. Yardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015). The person-based approach to intervention development: Application to digital health-related behavior change interventions. Journal of Medical Internet Research, 17, e30. doi:10.2196/jmir.4055
87. Yzer, M., LoRusso, S., & Nagler, R. H. (2015). On the conceptual ambiguity surrounding perceived message effectiveness. Health Communication, 30, 125–134. doi:10.1080/ 10410236.2014.974131
88. Zhao, X., Strasser, A., Cappella, J. N., Lerman, C., & Fishbein, M. (2011). A measure of perceived argument strength: Reliability and validity. Communication Methods and Measures, 5, 48–75. doi:10.1080/19312458.2010.547822