Evaluating public diplomacy programmes

The Hague Journal of Diplomacy

Volumn 2, Issue 3, 2007, Pages 255-281

  Pahlavi, Pierre C ^a,b  

^a Canadian Forces College   (Canada)

^b Department of Defense Studies ^*  (Canada)

Author keywords

Evaluation; Foreign policy; New information and communication technologies (NICT); Public diplomacy; Public opinions; Soft power; Strategic communications

Indexed keywords

EID: 35548959645     PISSN: 18711901     EISSN: 1871191X     Source Type: Journal    
DOI: 10.1163/187119007X240523     Document Type: Article

References (97)

1. Joseph Nye, The Paradox of American Power: Why the World's Only Superpower Can't Go It Alone (Oxford: Oxford. University Press, 2002).
2. US State Department, Under-Secretary for Public Diplomacy, ECA, Annual Report 2002 (Washington DC: Wilson Editions, 2002).
3. US BBG, Annual Report 2002 (Washington DC: BBG, 2003).
4. US General Accountability Office, International Broadcasting: New Strategic Approach Focuses on Reaching Large Audiences but Lacks Measurable Program Objectives, report to the House of Representatives' Committee on Internadonal Relations, GAO report: GAO-03-772 (Washington DC: GAO, July 2003).
5. US White House Office of Management and Budget, 2006 Report, released at the White House, ExpectMore.gov, 2006.
6. US General Accountability Office, US Public Diplomacy: Interagency Coordination Efforts Hampered by the Lack of a National Communication Strategy, GAO-05-323, April 2005;
7. US Advisory Commission on Public Diplomacy, 2004 Report, 28 September 2004.
8. US State Department, US Department of State Strategic Plan 2000, released by the Office of Management Policy and Planning, US Department of State, 25 October 2000.
9. US Bureau of Resource Management, 'Strategic Goal 11: Public Diplomacy', in FY2003: Performance and Accountability Report, December 2003.
10. 'Marrying the Mission to die Market' summarizes the fundamental strategy of US international broadcasting; see US BBG, Strategic Plan 2004, at http://www.bbg.gov/bbg_plan.cfm, accessed March 2004.
11. US Bureau of Resource Management, 'Strategic Goal 11: Public Diplomacy'.
12. France's MAE, DGCID, 'Série Évaluation 2003', in Bilan 2002 et Perspectives (Paris: MAE, DGCID, 2003).
13. South Africa's Department of Foreign Affairs, 'Priority Six: Public Diplomacy", in Strategic Plan 2003-2005 (Pretoria: DFA, 2003), p. 55.
14. Australia's DFAT, Annual Report 2002-2003 (Canberra: DFAT, 2003), p. 9.
15. Pierre Pahlavi, 'Canada's Public Diplomacy: Turning Point or Rhetoric?', Canadian Foreign Policy Journal, vol. 14, no. 1 (forthcoming 2007).
16. Similarly to the other programmes mentioned, 'The Beeb', as the BBC is affectionately known among the British, provides an excellent example of media, that find themselves playing a role somewhere between that of the traditional information service and that of an instrument of foreign policy. One should note that although it manages British audiovisual diplomacy with an editorial independence reinforced by the 2002 Broadcasting Agreement, the BBC is nevertheless constrained by the need to respect the strategic lines drawn by the British. Foreign and Commonwealth Office (FCO), which approves most of its decisions and tactical choices. Under the current regime, the FCO provides 94 per cent of the financing of this public service dirough a licensing fee that was indexed to inflation and fixed until 2006; see UK Department of National Heritage Broadcasting, Royal Charter of the British Broadcasting Corporation (London: HMSO and the Queen's Printer for Scotland, 1996).
17. US BBG, Annual Report 2002.
18. See Hugh Miles, Al-Jazeera: The Inside Story of the Arab News Channel that is Challenging the West (New York: Grove Press, 2005).
19. To obtain online audience figures, both the number of people logging onto websites and the number of page views in an average month are counted. The BBC counts a monthly average of 13 million visitors and. 77 million page views. It is interesting to note that Aljazeera.net is not far behind with 14 million visitors and 67 million page views.
20. US House of Representatives, CIR, Freedom Promotion Act of 2002 (introduced by US Representative HJ. Hyde (Chairman), 14 March 2001, reported by HIRC, 25 April 2001).
21. UK FCO, chapter 8 entitled 'Influence Worldwide', in FCO Annual Report 2003, p. 80, available at http://www.fco.gov.uk/Files/KFile/ 08_InfluenceWorldwide.pdf, p. 87.
22. US State Department, Under-Secretary for Public Diplomacy Charlotte Beers, 'Statement at Hearing: Public Diplomacy', House International Relations Committee, Washington DC, 10 October 2001, released by the Bureau of Public Affairs.
23. 'Among editors and opinion leaders, credibility is the crucial resource and an important source of soft power'; see Joseph S. Nye, 'Public Diplomacy in the 21st Century', The Globalist, 10 May 2004,
24. an extract from Joseph Nye's Soft Power. The Means to Success in World Politics (New York: Public Affairs, 2004).
25. On this question, the results contrast markedly. On the one hand, Al-Irakya and Al-Hurra channels draw great scepticism from Arab populations. On the other hand, a survey by global media ratings giant AC Nielson revealed that BBG s Al-Sawa is considered trustworthy by 42 per cent of 15-29-year-olds in the Middle East. Among the station's targets in Amman, Jordan (individuals between the ages of 17 and 28), when asked 'what station do you listen to most for news?', 33 per cent said that they listened to AlSawa. In Morocco, Radio Sawa is now the number one station among the coveted youth audience in Rabat and Casablanca; see Middle East Online, 7 April 2004.
26. In Bangladesh and in Indonesia the level reaches 85 per cent and 39 per cent versus 47 per cent and 16 percent for VOA; in Poland 32 per cent of the audience trusts the BBC versus 17 per cent for Deutsche Welle; in Egypt the BBC's credibility falls to 17 per cent, leading RFI-RMC with 9 per cent.
27. France's TV5, 'Le Centre du Monde est Partout', in Rapport 2003 (Paris : Dossier de Presse, 2003). 26) GAO-03-772, p. 17.
28. Charlotte Beers noted: 'The recall of these messages was higher than a soft drink can achieve in six months of advertising. It broke the bank in terms of recall. In terms of message retention, every single person who recognized it came back and said, "They're talking about the way they live in the United. States. I had no idea". A woman said, "I didn't know you could, wear scarves safely in that country". Another said, "Do you mean they're free to pray openly?'". see US State Department, Under-Secretary for Public Diplomacy C. Beers, American Public Diplomacy and Islam', testimony before the Senate Foreign Relations Committee, Washington DC, 27 February 2003, released by ECA.
29. BBC, Annual Review 2003.
30. 'Publics do not make foreign policy in the United States or anywhere else', observes Donna. Oglesby, 'but they do, in democratic countries in particular, set the parameters within which policy is made and carried out'. Remarks by Donna Marie Oglesby, Diplomat in Residence at Eckerd College, 'Public Diplomacy: Politics, Propaganda or Publicity', Tampa Bay Area Committee on Foreign Relations, 21 March 2006.
31. The poll of 39,43.5 people was conducted, for the BBC World Service by the international polling firm GlobeScan, together widi the Program on International Policy Attitudes (PIPA) at the University of Maryland.
32. A number of articles and commentary collected by Adele Lan Chen for die University of Southern California's Center on Public Diplomacy provide a useful perspective to evaluate both how the world views China and how China views the rest of the world; see Adele Lan Chen, 'World Perception Series, Part I: China', USCPublic Diplomacy Center, April 2005.
33. These analyses are currently being extended by an ambitious programme of research, publications, forums and public discussions on China that has been launched by the London-based Foreign Policy Centre (FPC); see Foreign Policy Centre, China and Globalization: Mapping Chinese Visions of their Future in the World, November 2004, at http://fpc.org.uk/fsblob/343.pdf [PDF].
34. Zogby International released a poll in April 2002 that concluded that Arabs and Muslims generally hold a favourable view of American culture, movies, television, science, technology and education; see Zogby International, The Ten Nation Impressions of America Poll (New York: Zogby International, 11 April 2002).
35. Delinda Hanley, 'Secretary's Open Forum Examines Public Diplomacy', Washington Report on Middle East Affairs, January/February 2004, vol. 23, pp. 75-76.
36. US State Department, J.F. Hoge Jr., 'The Benefits of Public Diplomacy and Exchange Programs', National Council for International Visitors, 12 March 2003.
37. David Hoffman, 'Beyond Public Diplomacy', Foreign Affairs, vol. 81, no. 2, 2002, p. 1.
38. R.S. Zaharna, American Public Diplomacy in the Arab World', Policy Brief, vol. 8, no. 2, 2003.
39. US White House Office of Management, 2006 Report.
40. US State Department, Minutes of the US Advisory Commission on Public Diplomacy January 2006 Official Meeting (Washington DC: US Advisory Commission on Public Diplomacy, 18 January 2006).
41. US General Accounting Office, US Public Diplomacy State Department Expands Efforts but Faces Significant Challenges, report to the House of Representatives, Committee on International Relations, Washington DC, GAO-03-951, September 2003, p. 27.
42. Anthony J. Blinken, 'Winning the War of Ideas', Washington Quarterly, vol. 25, no. 2, spring 2002, p. 289.
43. US State Department, Advisory Commission on Public Diplomacy, H.C. Pachios (Chairman), 'Hearing to Examine US Understanding of Arab Social and Political Thought', testimony before the Committee on Government Reform, Subcommittee on National Security, Veterans Affairs and International Relations, Washington DC, 8 October 2002.
44. Mark Leonard, and. C. Stead, Public Diplomacy (London: Foreign Policy Centre, 2002), p. 90.
45. The results revealed by the UK FCO's 2003 report are as follows: (i) teaching centre users 44 per cent (2001-02); (ii) library centre users 63 per cent (2001-02); (iii) international networking events 69 per cent (April 2002-January 2003); (iv) study fellows 74 per cent (April 2002-February 2003); (v) organized visits 86 per cent (April 2002-February 2003); see UK FCO, 'Influence Worldwide', FCO Annual Report 2003, p. 82.
46. US Department of State, ECA, The Visiting Fulbright Scholar Program Outcome Assessment (Washington DC: Bureau of Educational and Cultural Affairs, October 2005). This evaluation, conducted, by SRI International, is based on a 2003 survey conducted with. 1,894 Fulbright alumni from sixteen countries.
47. The objective of the outcome assessment was to determine the impact of the IVLP exchange experience on die professional lives of alumni, their affiliated organizations and home countries, and to ascertain -in quantitative and qualitative terms -the effectiveness of the programme in achieving its legislative mandate of increasing mutual understanding between the people of die United States and die people of other countries. US State Department, Educational and Cultural Affairs, International Visitor Leadership Program Outcome Assessment, released, by the Office of Policy and Evaluation, 15 February 2006.
48. After exposure, alumni return home with a better perception of US social institutions and values. These results have been interpreted quite (over)enthusiastically by PD decision-makers. 'We do have long-tested proofs that we can engage successfully', declared the head of die US public diplomacy programmes. 'When we bring people in on our educational and cultural exchanges, they are literally transformed from being hostile and suspicious to friends of the United States'; see US State Department, Under-Secretary for Public Diplomacy Charlotte Beers, 'American Public Diplomacy and Islam'.
49. M.A. Gentzkow and J.M. Shapiro, Media, Education, and Anti-Americanism in the Muslim World, available online from Harvard University's SSRN, 15 September 2003).
50. Erik C. Nisbet et al., 'Public Diplomacy, Television News, and. Muslim Opinion', Harvard International Journal of Press/Politics, vol. 9, no. 2, 2004, pp. 11-37.
51. Of course, these results raise obvious concerns about reverse causality: it may be that those with relatively pro-US attitudes are more likely to watch Western media than Al-Jazeera, and vice versa. One way to deal with this is to interrogate audiences directly on their impressions of the programmes' effects on their perception. It is also indispensable to conduct pre- and post-programme surveys that directly test and compare participant attitudes. Although it would not solve all the problems, it is a simple technique employed by public relations firms (and the BBC already uses such polls); see UK FCO, 'Influence Worldwide', FCO Annual Report 2003, p. 82.
52. Among others, Stuart Soroka examines the importance of mass media, in the relationship between public opinion and foreign policy; see Stuart N. Soroka, 'Media, Public Opinion, and Foreign Policy', Harvard International Journal of Press/Politics, vol. 8, no.1, 2003, pp. 27-48.
53. Susan Raines presents tentative findings from current exchange evaluation efforts; see Susan Raines, 'Evaluating Cross-Cultural Exchanges for Peace-Building: How Much Bang for the Buck?', paper presented, at the annual meeting of the International Studies Association, Montreal, 17 March 2003. In a similar piece, G. Spencer tested the positive impact of television news on the Northern Ireland peace process; see G. Spencer, 'The Impact of Television News on the Northern Ireland Peace Process', Media, Culture and Society, vol. 26, 2004, pp. 602-623.
54. In 'The Perfect War', Nicholas J. Cull compares the performance of US PD broadcasting during 'Desert Shield' and 'Desert Storm' to the current 'hearts and minds' efforts of George Bush's 'war on terror'. Kai Hafez asks whether Arab satellite broadcasting has taken over some of the roles usually played by political parties in 'Democracy without Political Parties'. Philip Seib suggests new media technologies like satellite TV, instant messaging, mobile phones and the internet are changing Middle Eastern politics, while S. Abdallah Schleifer expounds on 'The Impact of Arab Satellite Television on the Prospects for Democracy in the Arab World'. See Transnational Broadcasting Studies, no. 15, autumn 2005 (Oxford: Middle East Centre at St Antony's College, University of Oxford; and Cairo: Adham Center for Electronic Journalism at the American University In Cairo).
55. Wilton Park, Changing Perceptions: Review of Public Diplomacy (Steyning, West Sussex: Wilton Park, March 2002);
56. see also US CFR, Public Diplomacy: A Strategy for Reform, report of the Independent Task Force on Public Diplomacy, sponsored by the CFR, 30 July 2002.
57. For instance, the US State Department contracted with Aquirre International to conduct an independent evaluation of its Internet Access and Training Program in Eurasia (IATP); see US Department of State, Internet Access and Training Program (IATP): Evaluation in Eurasia, September 2005.
58. Hans N. Tuch, Communicating with the World: US Public Diplomacy Overseas (New York: St Martin's Press, 1990), p. 3.
59. M.E. Price, 'Public Diplomacy and the Transformation of International Broadcasting', Comparative Media Law Journal, vol. 1, 2003, p. 78.
60. In 'Inciting Genocide', Susan Bensch argues that music and political speech intended to compel others to commit genocide have played a central role; see Susan Bensch. 'Inciting Genocide', World Policy Journal, summer 2004, pp. 62-69.
61. See also US House of Representatives, Hearing before the House Subcommittee on the Middle East and South Asia, Words Have Consequences: The Impact of Incitement and Anti-American and Anti-Semitic Propaganda on American Interests in the Middle East (Washington DC: House of Representatives, 18 April 2002).
62. Michael Nelson, Wars of the Black Heavens: The Battles of Western Broadcasting in the Cold War (Syracuse NY: Syracuse University Press, 1997);
63. see also M.E. Price, 'Journeys in Media Space: Global Media and National Controls -Rethinking the Role of the State', paper presented at the 2001 Spry Memorial Lecture, at http://www.fas.umontreal.ca/COM/spry/spry-mp-e.html (accessed, June 2004).
64. See Kenneth A. Osgood, 'Hearts and Minds: The Unconventional Cold War', Journal of Cold War Studies, vol. 4, no. 2, spring 2002, pp. 85-107;
65. Daniel C. Thomas, 'Human Rights Ideas, the Demise of Communism and the End of the Cold War', Journal of Cold War Studies, vol. 7, no. 2, spring 2005, pp. 110-1,41;
66. Nancy E. Bernhard, US Television News and Cold War Propaganda, 1947-1960 (Cambridge: Cambridge University Press, 1999).
67. Cynthia P. Schneider, 'Culture Communicates: US Diplomacy that Works', in Jan Melissen (ed.), The New PtMc Diplomacy: Sofi Power in International Relations (Basingstoke: Palgrave-Macmillan, 2005), p. 151.
68. R. W. McChesney, 'Graham Spry and the Future of Public Broadcasting', Canadian Journal of Communication, no. 24, 1999, pp. 25-47.
69. M. Leonard and A. Small, Norway's Public Diplomacy (London: Foreign Policy Centre, 2003), p. 7.
70. For more on Turkey's cultural diplomacy, see P.C. Palilavi, "The Conquest of a Common Cultural Legacy: Turkey and Turkic Asia', in The Cultural Legacy of Conflict in Central and Inner Asia: Toronto Studies in Central and Inner Asia, no. 6, edited by M. Gervers et al (Toronto: Joint Centre for Asia Pacific Studies, 2004).
71. J. Davis, "The Campaign to Ban Landmines: Public Diplomacy, Middle Power Leadership and an Unconventional Negotiating Process', Journal, of Humanitarian Assistance, 2004.
72. Leonard and Stead, Public Diplomacy, p. 58;
73. see also E.H. Potter, Canada and the New Public Diplomacy, Discussion Paper In Diplomacy, published by Spencer Mawby, University of Leicester, Netherlands Institute of International Relations 'Clingendael', 2002, p. 12.
74. In Lloyd Axworthy's words, Canada obtained these results and others 'using soft-power concepts creatively, negotiations rather than coercion, powerful ideas rather than powerful weapons, public diplomacy rather than backroom bargaining'; see Canadian Foreign Affairs Minister Lloyd Axworthy, National Post, 23 January 1999.
75. United States Institute for Peace (USIP), 'Bridging the Divides: Improving Relations Widi and Within the Muslim World', in Grant Solicitation: Public Diplomacy (Washington DC: US Institute of Peace, March 2004).
76. When consulted on this issue, a number of academics specializing in PD and international affairs, as well as private sector officials from US public relations and opinion research firms with international operations, all agreed on the complexity of measuring public diplomacy. A 2002 GAO report on strategies for assessing US government information dissemination efforts recognized that 'establishing a causal link between agency actions and the ultimate impact of such programs is difficult'; see US GAO, Program Evaluation: Strategies for Assessing How Information Dissemination Contributes to Agency Goals, GAO-02-923 (Washington DC: GAO, 30 September 2002).
77. B. Barrett (US Advisory Commission for Public Diplomacy chair), quoted in Carl Weiser, 'Report Lists "Public Diplomacy" Failures', Gannett News Service, USA Today, 15 September 2003.
78. US Department of State, Bureau of International Information Programs, Payoff for Public Diplomacy: A Long-Term Proposition, USINFO, July 2004, available at http://usinfo.state.gov/xarchives/display. html?p=waslifile- english&y=2004&m=july&x=20040723164934sjhtrop0.1559259 (accessed May 2007).
79. 'Can a universal yardstick be used to measure or explain public relations?', asks Zaharan R.S. Zaharna incredulously in 'Intercultural Communication and. International Public Relations: An Integrated. Literature Review and Critique', Communication Quarterly, no. 48, 2000, pp. 85-100.
80. For a general overview of diis field, see Michael B Goodman, "The Role of Business In Public Diplomacy', Journal of Business Strategy, vol. 27, no. 3, 2006, p. 5.
81. Jay Wang, 'Public Diplomacy and Global Business', Journal, of Business Strategy, vol. 27, no. 3, 2006, p. 41.
82. Anna Tiedeman, An Examination Of US Public Diplomacy Efforts After September 11 2001, Master of Arts in Law and Diplomacy Thesis (Medford MA: Tufts University, The Fletcher School, 2005), p. 60.
83. Developed by USAID in 1969, the Logical Framework Approach (LFA) is frequently used by multilateral donor organizations like SIDA, NORAD, DFID and UNDP; for a detailed explanation see C. Alter and M. Egan, 'Logic Modeling: A Tool for Teaching Critical Thinking in Social Work Practice', Journal of Social Work Education, vol. 33, no. 1, 1, 997;
84. and M. Fairchild, The IPR Toolkit: Planning Research and Evaluation for Public Relations Success (London: IPR, 2001), the manual used by the UK FCO s evaluation bureau.
85. Leonard and Stead, Public Diplomacy, p. 91.
86. USGAO-03-951, p. 26.
87. For example, a corporation such as Coca-Cola makes use of this model to gauge the positive effects of its advertising campaigns on its products' consumption patterns on a global and regional scale. In the case of the Coca-Cola Company, short-term outcomes include to target audience knowledge and awareness of ad slogans; intermediate outcomes include positive changes in target opinions; and long-term outcomes include a noticeable increase in terms of consumption.
88. Evaluations of performance can then assess the effects of an integrated set of efforts; see GAO-03-951, p. 28.
89. These efforts are supported by new regulations such as the US Government Performance and Results Act (GPRA) and the German Concept 2000.
90. In certain cases, evaluation bureaus have even perfected techniques for conducting ad hoc evaluations at short notice. See France's DGCID, Bilan des Évaluations 2001 (Paris: MAE, 2002).
91. "We are restructuring the evaluation of all of our programs, based on the successful model used In our educational and cultural affairs division'; US State Department's Under-Secretary for Public Diplomacy Karen Hughes, America's Dialogue with the World', statement before the House International Relations Committee, (Washington DC: US State Department, November 2005).
92. US State Department, Minutes of the US Advisory Commission on Public Diplomacy, January 2006 Official Meeting (Washington DC: US Advisory Commission on Public Diplomacy, 18 January 2006).
93. Interviews with Barry Fulton, George Washington University and former USIA agent; Eytan Gilboa, Bar-Ilan University and Holon Institute of Technology (Israel); Evan H. Pooler, DFAIT (Canada); Robin C. Brown, University of Leeds (UK) (conducted in Montreal, 18 March 2004 during the ISA conference); and interviews realized with DGCID and, TV5 leaders or members, Paris, Quai d'Orsay, July 2003.
94. 'In programs in which agencies do not act directly to achieve their goals, but inform and persuade others to act to achieve a desired outcome, it would seem all the more important to assure decisionmakers that this strategy is credible and likely to succeed'; GAO-02-923.
95. Nye, Soft Power, pp. 124-125.
96. Nye, The Paradox of American Power.
97. Such an evolution is already perceptible in Korea. See John Hemery, 'Training for Public Diplomacy: An Evolutionary Perspective', in Jan Melissen (ed.), The New Public Diplomacy: Soft Power in International Relations (Basingstoke: Palgrave-Macmillan, 2005), pp. 203-207.