Trust, but Verify: The Transparency Revolution and Qualitative International Relations

Security Studies

Volumn 23, Issue 4, 2014, Pages 663-688

  Moravcsik, Andrew ^a  

^a PRINCETON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84916212737     PISSN: 09636412     EISSN: 15561852     Source Type: Journal    
DOI: 10.1080/09636412.2014.970846     Document Type: Article

References (95)

1. Daniel Maliniak, Susan Peterson, and Daniel J. Tierney, TRIP around the World: Teaching, Research and Policy Views of International Relations Faculty in 20 Countries (Williamsburg, Virginia: Institute for the Theory and Practice of International Relations, College of William and Mary, 2012), Charts 28-30, available at http://www.wm.edu/offices/itpir/_documents/trip/trip_around_the_world_2011.pdf.
2. Active Citation 1 in online supplemental material. Instructions for accessing this material can be found at the end of this article.
3. Maliniak M et al., Chart 42. These seventeen scholars are Michael Barnett
4. Barry Buzan, Martha Finnemore, Samuel Huntington, John Ikenberry, Robert Jervis, Peter Katzenstein, Robert Keohane, Stephen Krasner, David Lake, John Mearsheimer, Joseph Nye, John Ruggie, Jack Snyder, Stephen Walt, Kenneth Waltz, and Alexander Wendt.Philip Tetlock, Expert Political Judgment: How Good is It? How Can We Know? (Princeton, NJ: Princeton University Press, 2005) Chapter 3.
5. Maliniak M et al., Chart 57. See Active Citation 2 in online supplemental material.
6. Jon Wiener, Historians in Trouble: Plagiarism, Fraud and Politics in the Ivory Tower (New York: New Press, 2005);
7. Malcolm Wright and J. Scott Armstrong, “The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge?” Interfaces 38, no. 2 (March-April 2008): 125-32
8. David Goodstein, On Fact and Fraud: Cautionary Tales from the Front Lines of Science (Princeton, NJ: Princeton University Press, 2010)
9. Ole Bjørn Rekdal, “Academic Citation Practice: A Sinking Sheep?” Libraries and the Academy 14, no. 4 (2014): 567-85.
10. Gary King, Robert O. Keohane, and Sidney Verba, Designing Social Inquiry: Scientific Inference in Qualitative Research (Princeton, NJ: Princeton University Press, 1994)
11. Steven Van Evera, Guide to Methods for Students of Political Science (Ithaca, NY: Cornell University Press, 1997)
12. Colin Elman and Miriam Fendius Elman, eds., Bridges and Boundaries: Historians, Political Scientists, and the Study of International Relations (Cambridge, MA: MIT Press, 2001)
13. Andrew Bennett, Case Studies and Theory Development in the Social Sciences (Cambridge, MA: MIT Press, 2005)
14. Audie Klotz and Deepa Prakash, eds., Qualitative Methods in International Relations: A Pluralist Guide (Hampshire, UK: Palgrave Macmillan, 2008)
15. Henry Brady and David Collier, Rethinking Social Inquiry: Diverse Tools, Shared Standards (Lanham, MD: Rowman & Littlefield, 2010).
16. This tripartite distinction revises that found in American Political Science Association (APSA), A Guide to Professional Ethics in Political Science, 2nd ed. Washington, DC: APSA, 2012), 9-10, also available at
17. Arthur Lupia and Colin Elman, “Openness in Political Science: Data Access and Research Transparency,” PS: Political Science & Politics 47, no. 1 (January 2014): 19-42, Appendices A and B.
18. Richard P. Feynman, “Cargo Cult Science: Caltech 1974 Commencement Address,” Engineering and Science 37, no. 7 (June 1974): 10-13.
19. R.G. Collingwood, The Idea of History (Oxford: Oxford University Press, 1946), p. 252.
20. James Mann, The Rebellion of Ronald Reagan: A History of the End of the Cold War (New York: Penguin, 2009), 65, 100, 267, 273.
21. Bennett Case Studies and Theory Development; David Collier, “Understanding Process Tracing,” PS: Political Science & Politics 44, no. 4 (2011): 823-30
22. Peter A. Hall, “Systematic Process Analysis: When and How to Use It,” European Management Review 3 (2006): 24-31.
23. APSA, A Guide to Professional Ethics.
24. Brady and Collier, Rethinking Social Inquiry, 252-53
25. James Mahoney and Gary Goertz, A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences (Princeton, NJ: Princeton University Press, 2012), 230-31
26. Van Evera, Guide to Methods for Students of Political Science.
27. APSA, Guidelines for Qualitative Transparency (2013), see Footnote 7.
28. Qualitative Data Repository (QDR), Center for Qualitative and Multi-Method Inquiry, Syracuse University, http://www.qdr.org.
29. For guidance on qualitative methods, see ibid. This is increasingly commonplace outside of academia. For a creative use of multi-media active footnotes in a museum website, see Metropolitan Museum of Art, www.metmuseum.org.
30. Andrew Moravcsik, “Example of an Active Citation about Steven Spielberg’s ‘Lincoln,”’ Methodological Memo, Princeton University, 2014, http://www.princeton.edu/amoravcs/library/ Stevens-ExampleActiveCitation.pdf
31. QDR, http://www.qdr.org.
32. Andrew Moravcsik, “Active Citation: A Precondition for Replicable Qualitative Research,” PS: Political Science & Politics 43, no. 1 (January 2010): 29-35;
33. Moravcsik, “Active Citation and Qualitative Political Science,” Qualitative & Multi-Method Research 10, no.1 (Spring 2012)
34. Moravcsik, “Transparency: The Revolution in Qualitative Political Science,” PS: Political Science & Politics 47, no. 1 (January 2014): 48-53.
35. APSA, Guidelines for Qualitative Transparency.
36. Moravcsik, ; Moravcsik, “Transparency.”
37. In addition to APSA serving a lead host, the workshop is co-sponsored by Syracuse University’s Center for Qualitative and Multi-Method Inquiry and University of Michigan’s Center for Political Studies and the Inter-university Consortium for Political and Social Research, University of Michigan, Ann Arbor.
38. International Security provides admirably detailed traditional footnotes, Though it does not consistently provide either annotations or extended quotations. For similar book citations
39. Thomas J. Christensen, Worse Than a Monolith: Alliance Politics and Problems of Coercive Diplomacy in Asia (Princeton, NJ: Princeton University Press, 2011)
40. Jack L. Snyder, The Ideology of the Offensive: Military Decision Making and the Disasters of 1914 (Ithaca, NY: Cornell University Press, 1984).
41. Linked documentation is the principle underlying the software Zotero, Supported by the Mellon and Sloan Foundations, https://www.zotero.org/blog/building-a-sustainable-zotero-project/.
42. For an adventurous example, See the Yale Law Journal Online, http://www.yalelawjournal.org/.
43. US Department of State, Foreign Relations of the United States, 2009.
44. QDR, http://www.qdr.org
45. Henry Murray Data Archive at Harvard University, http://www.murray.harvard.edu/
46. Colin Elman, Diana Kapiszewski, and Lorena Vinuela, “Qualitative Data Archiving: Rewards and Challenges,” PS: Political Science & Politics 43, no. 1 (January 2010): 23-27
47. QDR, “A Guide to Sharing Qualitative Data,” Qualitative Data Repository Project, funded by the National Science Foundation, December 2012.
48. Evan S. Lieberman, “Bridging the Qualitative-Quantitative Divide: Best Practices in the Development of Historically Oriented Replication Databases,” Annual Review of Political Science 13 (2010): 37-59.
49. For a fine example, See the work of the Research Network on Gender Politics and the State, http://libarts.wsu.edu/pppa/rngs/.
50. Mark Bauderlein, “We Must Stop the Avalanche of Low-Quality Research,” The Chronicle of Higher Education (13 June 2010).
51. Ian S. Lustick, “History, Historiography, and Political Science: Multiple Historical Records and the Problem of Selection Bias,” American Political Science Review 90, no. 3 (September 1996): 605-18.
52. K. S. Larsson, “The Dissemination of False Data through Inadequate Citation,” Journal of Internal Medicine 238 (1995): 445-50
53. James K. Wetterer, “Quotation Error, Citation Copying, and Ant Extinctions in Madeira,” Scientometrics 6, no. 7(2006): 351-72
54. Wright and Armstrong, “The Ombudsman,” 125-32.
55. It would be unprofessional on my part to criticize specific works without extensive documentation and an opportunity for rebuttal by the author, both of which go beyond the scope and space limits of this article. However, one recent work, Sebastian Rosato’s book, Europe United: Power Politics and the Making of the European Community (Ithaca, NY: Cornell University Press, 2011), does make claims that more than one scholar has charged—and in some cases Rosato has admitted—are based on erroneous data, data analysis, and biased methodology
56. Andrew Moravcsik, “Did Power Politics Cause European Integration? Realist Theory Meets Qualitative Methods,” Security Studies 22, no. 4 (December 2013): 773-90
57. Robert H. Lieshout, “Review of Europe United,” Journal of Cold War Studies 14, no. 4 (Fall 2012): 234-37
58. Sebastian Rosato, “Theory and Evidence in Europe United: A Response to My Critics,” Security Studies 22, no. 4 (December 2013): 802-20.
59. For an example of a debate over whether verbatim quotations were edited so as to change their meaning radically, which in two cases the author concedes
60. Moravcsik, “Did Power Politics Cause European Integration?” 786-88. For Rosato’s admissions
61. Rosato, “Theory and Evidence in Europe United,” 808n13, 807n12, 808n15, respectively.
62. Jack Levy, Thomas J. Christensen, and Marc Trachtenberg, “, Correspondence: Mobilization and Inadvertence in the July Crisis,” International Security 16, no. 1 (Summer 1991): 189-204
63. Christensen, Worse Than a Monolith; Snyder, The Ideology of the Offensive; Elizabeth Saunders, Leaders at War: How Presidents Shape Military Interventions (Ithaca, NY: Cornell University Press, 2011)
64. Keir A. Lieber, “The New History of World War I and What It Means for International Relations Theory,” International Security 32, no. 2 (Fall 2007): 155-91
65. Keren Yarhi-Milo, Knowing Thy Adversary: Leaders, Intelligence, and Assessments of the Adversary’s Intentions (Princeton, NJ: Princeton University Press, 2014).
66. Marc Trachtenberg, The Craft of International History: A Guide to Method (Princeton, NJ: Princeton University Press, 2009)
67. Layna Mosley, ed., Interview Research in Political Science (Ithaca, NY: Cornell University Press, 2013)
68. Lisa Wedeen, “Reflections on Ethnographic Work in Political Science,” Annual Review of Political Science 13 (2010): 255-72
69. Diana Kapiszewski et al. Field Research in Political Science: Practices and Principles (New York: Cambridge University Press, forthcoming).
70. Elman, Kapiszewski, and Vinuela, “Qualitative Data Archiving,” 23.
71. For an example of how a major theoretical point can turn on contextual interpretation of a document
72. Moravcsik, “Transparency,” 782; Rosato, “Theory and Evidence in Europe United,” 815-16
73. Documents Diplomatiques Franc¸ais, no. 297, tome 1, 1955. I have activated the final citation to the original source, and my annotation addresses the debate. See Active Citation 3 in online supplemental material.
74. Yuen Foong Khong, Analogies at War: Korea, Munich, Dien Bien Phu, and the Vietnam Decisions of 1965 (Princeton, NJ: Princeton University Press, 1992); examples in note 38
75. Derek Beach and R. B. Pedersen, Process-Tracing Methods: Foundations and Guidelines (Ann Arbor: University of Michigan Press, 2013)
76. James D. Fearon, “Counterfactuals and Hypothesis Testing in Political Science,” World Politics 43, no. 2 (January 1991): 169-95
77. Levy et al., “Correspondence”; Bennett, Case Studies and Theory Development
78. Mahoney and Goertz, A Tale of Two Cultures; James Mahoney and Rachel Sweet Vanderpoell, “Set Diagrams and Qualitative Research,” Comparative Political Studies (forthcoming).
79. Lustick M “History, Historiography, and Political Science.” Some scholars maintain that sampling is unnecessary. Challenged on the lack of an explicit strategy and potential biases in sampling secondary and primary material, one scholar has argued, “Rather than relying on some kind of sample, my claims rest on an informed reading of the secondary literature.” Yet the same scholar also concludes, “The existing historiography leans toward my view.” Rosato, “Theory and Evidence in Europe United,” 811. This begs the basic methodological issue: How can we know what direction the historical literature leans unless we have sampled it fairly?
80. Robert Siebers, “Data Inconsistencies in Abstracts of Articles in Clinical Chemistry,” Clinical Chemistry 47, no.1 (January 2001):149.
81. Moravcsik “Transparency,” 779-80
82. Rosato, “Theory and Evidence in Europe United,” 812-831.
83. Robert H. Bates et al., Analytic Narratives (Princeton, NJ: Princeton University Press, 1998).
84. H-DIPLO: H-Net Network on Diplomatic History and International Affairs, https://networks.h-.org/h-diplo (accessed 1 July 2014).
85. APSA and various publishers are considering proposals for replication journals or sections of journals on replication. Qualitative replication is increasingly taught at the graduate level. Collier, “Understanding Process Tracing.”
86. Marc Trachtenberg, “Audience Costs in 1954,” 2014, http://www.h-net.org/diplo/ISSF/PDF/ISSF-Forum-1.pdf
87. Jack Snyder and Erica D. Borghard, “The Cost of Empty Threats: A Penny, Not a Pound,” American Political Science Review 105, no. 3 (August 2011): 437-56
88. Francis Gavin et al. “Forum on ‘What We Talk About When We Talk About Nuclear Weapons’,” H-Diplo/ISSF Forum (15 June 2014), available at http://issforum.org/ISSF/PDF/ISSF-Forum-2.pdf
89. Tommaso Pavone, “Constitutional Constraints, Organizational Preferences, and Qualitative Methods: A Replication and Reassessment of North and Weingast” (Unpublished manuscript, Department of Politics, Princeton University, May 2014).
90. Harry Eckstein, “Case Study and Theory in Political Science,” In Handbook of Political Science, ed. Fred Greenstein and Nelson Polsby (Reading, MA: Addison-Wesley, 1975), 79-117
91. Jon Elster, “A Plea for Mechanisms” In Social Mechanisms, ed. P. Hedström and R. Swedborg (Cambridge, Cambridge University Press, 1998), 1-31.
92. Gary King, “A Hard Unsolved Problem? Post-Treatment Bias in Big Social Science Questions,” Presentation at Hard Problems in Social Science Symposium, Institute for Quantitative Social Science, Harvard University, 10 April 2010, http://GKing.Harvard.edu.
93. Bent Flyvbjerg, “Case Study,” In The SAGE Handbook of Qualitative Research, 4th ed., ed. Norman Denzin and Yvonna Lincoln (Los Angeles: Sage, 2011), 301-16
94. Peter J. Katzenstein and Rudra Sil, “Analytic Eclecticism in the Study of World Politics: Reconfiguring Problems and Mechanisms across Research Traditions,” Perspectives on Politics 8, no. 2 (2010): 411-31.
95. Francis Gavin, “Panel Comments” 54th Annual Convention of the International Studies Association, San Francisco, CA, 5 April 2013.