Contextualizing the twin concepts of systematicity and transparency in information systems literature reviews

European Journal of Information Systems

Volumn 25, Issue 6, 2016, Pages 493-508

  Paré, Guy ^a   Tate, Mary ^b   Johnstone, David ^b   Kitsiou, Spyros ^c  

^a HEC MONTRÉAL   (Canada)

^b VICTORIA UNIVERSITY OF WELLINGTON   (New Zealand)

^c UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

Author keywords

literature review; rigour; systematicity; transparency; trustworthiness

Indexed keywords

INFORMATION USE; ITERATIVE METHODS; TRANSPARENCY;

EMPIRICAL STUDIES; HIGH STANDARDS; INTERPRETIVIST; LITERATURE REVIEWS; REVIEW PAPERS; RIGOR; SYSTEMATICITY; TRUSTWORTHINESS;

INFORMATION SYSTEMS;

EID: 85014343759     PISSN: 0960085X     EISSN: 14769344     Source Type: Journal    
DOI: 10.1057/S41303-016-0020-3     Document Type: Article

References (103)

1. ARKSEY H and O’MALLEY L (2005) Scoping studies: towards a methodological framework. International Journal of Social Research Methodology 8(1), 19–32.
2. AVEYARD H (2014) Doing a Literature Review in Health and Social Care: A Practical Guide. McGraw-Hill Education, UK.
3. AYTUG ZG, ROTHSTEIN HR, ZHOU W and KERN MC (2012) Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods 15(1), 103–133.
4. BAKER MJ (2000) Writing a literature review. Marketing Review 1(2), 219–247.
5. BANDARA W, FURTMULLER E, GORBACHEVA E, MISKON S and BEEKHUYZEN J (2015) Achieving rigour in literature reviews: insights from qualitative data analysis and tool-support. Communications of the Association for Information Systems 37, 8.
6. BARKI H and HARTWICK J (1989) Rethinking the concept of user involvement. MIS Quarterly 13(1), 53–63
7. BAUMEISTER RF (2013) Writing a literature review. In The Portable Mentor, pp 119–132. Springer, New York.
8. BENBASAT I and WEBER R (1996) Research commentary: Rethinking ‘‘diversity’’ in information systems research. Information Systems Research 7(4), 389–399.
9. BOELL SK and CECEZ-KECMANOVIC D (2015a) On being ‘systematic’ in literature reviews in IS. Journal of Information Technology 30(2), 161–173.
10. BOELL S and CECEZ-KECMANOVIC D (2015b) Debating systematic literature reviews (SLR) and their ramifications for IS: a rejoinder to Mike Chiasson, Briony Oates, Ulrike Schultze, and Richard Watson. Journal of Information Technology 30(2), 188–193.
11. BOOTH A, PAPAIOANNOU D and SUTTON A (2012) Systematic Approaches to a Successful Literature Review. Sage, London.
12. BORKO H, LISTON D and WHITCOMB JA (2007) Genres of empirical research in teacher education. Journal of Teacher Education 58(1), 3.
13. CAMPBELL DT (1974) Evolutionary epistemology. In The Philosophy of Karl Popper (Schilpp PA, Ed), vol. 14, pp 413–463. Open Court, LaSalle, IL.
14. CAMPBELL R, POUND P, MORGAN M, DAKER-WHITE G, BRITTEN N, PILL R, YARDLEY L, POPE C and DONOVAN J (2011) Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research. Health Technology Assessment 15(43), 1–164.
15. CHEUNG C and THADANI D (2012) The impact of electronic word-of-mouth communication: a literature analysis and integrative model. Decision Support Systems 54(1), 461–470.
16. CHIASSON M (2015) Avoiding methodological overdose: a declaration for independent ends. Journal of Information Technology 30(2), 174–176.
17. CHUA WF (1986) Radical developments in accounting thought. The Accounting Review 61(4), 601–632.
18. CONN VS, ISARAMALAI SA, RATH S, JANTARAKUPT P, WADHAWAN R and DASH Y (2003) Beyond MEDLINE for literature searches. Journal of Nursing Scholarship 35(2), 177–182.
19. COOPER HM (1988) Organizing knowledge syntheses: a taxonomy of literature reviews. Knowledge, Technology & Policy 1(1), 104–126.
20. COOPER HM (2010) Research Synthesis and Meta-Analysis: A Step-by-Step Approach, 4th edn. Sage, London.
21. COOPER HM and HEDGES L (2009) Research synthesis as a scientific process. In The Handbook of Research Synthesis and Meta-analysis (Cooper H, Hedges L and Valentine L, Eds), pp 3–16. Russell Sage Foundation, London.
22. CURRAN PJ (2009) The seemingly quixotic pursuit of a cumulative psychological science. Psychological Methods 14, 77–80.
23. D’ARCY J and HERATH T (2011) A review and analysis of deterrence theory in the IS security literature: making sense of the disparate findings. European Journal of Information Systems 20, 643–658.
24. DAUDT HM, VAN MOSSEL C and SCOTT SJ (2013). Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. BMC Medical Research Methodology 13, 48.
25. DEEKS JJ, DINNES J, DAMICO R, SOWDEN AJ, SAKAROVITCH C, SONG F, PETTICREW M and ALTMAN DJ (2003) Evaluating non-randomised intervention studies. Health Technology Assessment 7(27), 1–179.
26. DIXON-WOODS M, BONAS S, BOOTH A, JONES DR, MILLER T, SUTTON AJ, SHAW RL, SMITH JA and YOUNG B (2006) How can systematic reviews incorporate qualitative research? A critical perspective. Qualitative Research 6(1), 27–44.
27. DUBÉ L and PARÉ G (2003) Rigor in information systems positivist case research: current practices, trends, and recommendations. MIS Quarterly 27(4), 597–636.
28. EDWARDS A, ELWYN G, HOOD K and ROLLNICK S (2000) Judging the ‘weighting of evidence’ in systematic reviews: introducing the rigour into the qualitative overview stage by assessing signal and noise. Journal of Evaluation in Clinical Practice 6, 177–184.
29. FISHBEIN M and AJZEN I (1975) Belief, Attitude, Intention and Behaviour: An Introduction to Theory and Research. Addison-Wesley, Reading, MA.
30. GARVIN DA (1984) What does ‘‘product quality’’ really mean? Sloan Management Review 26(1), 25–43.
31. GARVIN DA (1987) Competing on the eight dimensions of quality. Harvard Business Review 65(6), 101–109.
32. GOUGH D, THOMAS J, OLIVER S (2012) Clarifying differences between review designs and methods. Systematic Reviews 1(1), 28–36.
33. GREENHALGH T and PEACOCK R (2005) Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. British Medical Journal 331(7524), 1064–1065.
34. GREENHALGH T, ROBERT G, MACFARLANE F, BATE P, KYRIAKIDOU O and PEACOCK R (2005) Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Social Science & Medicine 61(2), 417–430.
35. GUYATT G, OXMAN AD, AKL EA, KUNZ R, VIST G, BROZEK J et al (2011) GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology 64(4), 383–394.
36. HART C (2001) Doing a Literature Search: A Comprehensive Guide for the Social Sciences. Sage, Thousand Oaks, London.
37. HIGGINS JPT and ALTMAN DG (2008) Assessing risk of bias. In Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series (Higgins JPT, Green S, Eds), pp 187–241. Wiley, Chichester, UK.
38. HIGGINS J and GREEN S (2008) Cochrane Handbook for Systematic Reviews of Interventions. Wiley-Blackwell, Chichester.
39. HIRSCHHEIM RA and KLEIN HK (2006) Crisis in the IS field? A critical reflection on the state of the discipline. In Information Systems: The State of the Field (King JL and Lyytinen K, Eds), Chapter 5, pp 71–146.: Wiley, Chichester, UK.
40. HOON C (2013) Meta-synthesis of qualitative case studies: an approach to theory building. Organizational Research Methods 16(4), 522–556.
41. HOUY C, FETTKE P and LOOS P (2015) Stylized facts as an instrument for literature review and cumulative information systems research. Communications of the Association for Information Systems 37, 10.
42. JARVENPAA SL, DICKSON GW and DESANCTIS G (1985) Methodological issues in experimental IS research: experiences and recommendations. MIS Quarterly 9(2), 141–156.
43. KEUTEL M, MICHALIK B and RICHTER J (2014) Towards mindful case study research in IS: a critical analysis of the past ten years. European Journal of Information Systems 23, 256–272.
44. KING WR and HE J (2005) Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems 16(1), 665–686.
45. KITCHENHAM B, BRERETON OP, BUDGEN D, TURNER M, BAILEY J and LINKMAN S (2009). Systematic literature reviews in software engineering – a systematic literature review. Information and Software Technology 51(1), 7–15.
46. KITCHENHAM B and CHARTERS S (2007) Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report. http://www.dur.ac.uk/ebse/resources/guidelines/Systematic-reviews-5-8.pdf
47. KITSIOU S, PARÉ G and JAANA M (2013) Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research 15(7), e150.
48. KLEIN HK (2003) Crisis in the IS field? A critical reflection on the state of the discipline. Journal of the Association for Information Systems 4, 10.
49. KLEIN HK and MYERS MD (1999) A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly 23(1), 67–93.
50. KUHN TS (1962) The Structure of Scientific Revolutions. University of Chicago, Chicago, IL.
51. LEE RP, HART RI, WATSON RM and RAPLEY T (2015) Qualitative synthesis in practice: some pragmatics of meta-ethnography. Qualitative Research 15(3), 334–350.
52. LIBERATI A, ALTMAN DG, TETZLAFF J, MULROW C, GØTZSCHE PC, IOANNIDIS JP et al (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Annals of Internal Medicine 151(4), W-65.
53. LINCOLN YS and GUBA EA (1985) Naturalistic Inquiry. Sage, Beverly Hills, CA.
54. LONG, AF and GODFREY M (2004) An evaluation tool to assess the quality of qualitative research studies. International Journal of Social Research Methodology 7(2), 181–196.
55. MARCH, ST and SMITH, GF (1995) Design and natural science research on information technology. Decision Support Systems 15(4), 251–266.
56. MINGERS J, MUTCH A and WILLCOCKS L (2013) Critical realism in information systems research. MIS Quarterly 37(3), 795–802.
57. MOHER D, SHAMSEER L, CLARKE M, GHERSI D, LIBERATI A, PETTICREW M et al (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews 4(1), 1.
58. NEWBERT SL, DAVID RJ and HAN SK (2014) Rarely pure and never simple: assessing cumulative evidence in strategic management. Strategic Organization 12(2), 142–154.
59. NOBLIT GW and HARE RD (1988) Meta-ethnography: Synthesizing Qualitative Studies (vol. 11). Sage, London.
60. OATES B (2011) Evidence-based information systems: a decade later. In Proceedings of the European Conference on Information Systems, Paper no. 222. Helsinki, Finland.
61. OATES B (2015) On systematic reviews for evidence-based practice. Journal of Information Technology 30(2), 177–179.
62. OKOLI C (2015a) A guide to conducting a standalone systematic literature review. Communications of the Association for Information Systems 37, 43.
63. OKOLI C (2015b) The view from giants’ shoulders: developing theory with theory-mining systematic literature reviews. Working Paper. http://ssrn. com/abstract=2699362.
64. OXMAN AD and GUYATT GH (1991) Validation of an index of the quality of review articles. Journal of Clinical Epidemiology 44(11), 1271–1278.
65. PARÉ G, CAMERON AF, POBA-NZAOU P and TEMPLIER M (2013). A systematic assessment of rigor in information systems ranking-type Delphi studies. Information & Management 50(5), 207–217.
66. PARÉ G, TRUDEL MC, JAANA M and KITSIOU S (2015) Synthesizing information systems knowledge: a typology of literature reviews. Information & Management 52(2), 183–199.
67. PAWSON R, GREENHALGH T, HARVEY G and WALSHE K (2005) Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy 10(suppl 1), 21–34.
68. PENTLand BT (1995) Information systems and organizational learning: the social epistemology of organizational knowledge systems. Accounting, Management and Information Technologies 5(1), 1–21.
69. PETTER S, STRAUB D and RAI A (2007) Specifying formative constructs in information systems research. MIS Quarterly 31(4), 623–656.
70. PETTICREW M and MCCARTNEY G (2011) Using systematic reviews to separate scientific from policy debate relevant to climate change. American Journal of Preventive Medicine 40(5), 576–578.
71. PETTICREW M and ROBERTS H (2006) Why do we need systematic reviews? In: Systematic Reviews in the Social Sciences: A Practical Guide, chapter 1, pp 1–26. Blackwell Publishing Co, Malden, MA
72. PINSONNEAULT A and KRAEMER KL (1993) Survey research methodology in management information systems: an assessment. Journal of Management Information Systems 10(2), 75–105.
73. RAGIN C (1987) The Comparative Method. Moving Beyond Qualitative and Quantitative Strategies. University of California Press, Berkeley.
74. RAVICHANDRAN T, LERTWONGSATIEN C and LERTWONGSATIEN C (2005) Effect of information systems resources and capabilities on firm performance: a resource-based perspective. Journal of Management Information Systems 21(4), 237–276.
75. REEVES C and BEDNAR D (1994). Defining quality: alternatives and implications. The Academy of Management Review 19(3), 419–445.
76. RIVARD S and LAPOINTE L (2012) Information technology implementers’ responses to user resistance: nature and effects. MIS Quarterly 36(3), 897–920.
77. ROUSSEAU DM, MANNING J and DENYER D (2008) Evidence in management and organizational science: assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals 2(1), 475–515.
78. ROWE F (2014) What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems 23(3), 241–255.
79. SANDELOWSKI M and BARROSO J (2003) Creating metasummaries of qualitative findings. Nursing Research 52(4), 226–233.
80. SANDELOWSKI M, VOILS CI, LEEMAN J and CRANDELL JL (2012) Mapping the mixed methods–mixed research synthesis terrain. Journal of Mixed Methods Research 6(4), 317–331.
81. SCHRYEN G (2015) Writing qualitative IS literature reviews – guidelines for synthesis, interpretation, and guidance of research. Communications of the Association for Information Systems 37, 12.
82. SCHULTZE U (2015) Skirting SLR’s language trap: reframing the ‘systematic’ vs ‘traditional’ Literature review opposition as a continuum. Journal of Information Technology 30(2), 180–184.
83. SHEA BJ, HAMEL C, WELLS GA, BOUTER LM, KRISTJANSSON E, GRIMSHAW J and BOERS M (2009) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology 62(10), 1013–1020.
84. SHORT J (2009) The art of writing a review article. Journal of Management 35(6), 1312–1317.
85. SIVO SA, SAUNDERS C, CHANG Q and JIANG JJ (2006) How low should you go? Low response rates and the validity of inference in IS questionnaire research. Journal of the Association for Information Systems 7(6), 17.
86. STRAUB DW (1989) Validating instruments in MIS research. MIS Quarterly 13(2), 147–169.
87. STRAUB DW, BOUDREAU MC and GEFEN D (2004) Validation guidelines for IS positivist research. Communications of the Association for Information Systems 13, 63.
88. SYLVESTER A, TATE M and JOHNSTONE D (2013) Beyond synthesis: representing heterogeneous research literature. Behaviour & Information Technology 32(12), 1199–1215.
89. TATE M, FURTMUELLER E, EVERMANN J and BANDARA W (2015) Introduction to the special issue: the literature review in information systems. Communications of the Association for Information Systems 37, 5.
90. TEMPLIER M and PARÉ G (2015) A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems 37, 6.
91. TORRACO RJ (2005) Writing integrative literature reviews: guidelines and examples. Human Resource Development Review 4(3), 356–367.
92. TURNER M, KITCHENHAM B, BRERETON P, CHARTERS S and BUDGEN D (2010) Does the technology acceptance model predict actual use? A systematic literature review. Information and Software Technology 52(5), 463–479.
93. VALANIDES N and CHAROULA A (2008) An exploratory study about the role of epistemological beliefs and dispositions on learners’ thinking about an iII-defined issue in solo and duo problem-solving contexts. In Knowing, Knowledge and Beliefs, pp 197–218. Springer, Netherlands.
94. VICKERS M (1999) Information technology development methodologies towards a non-positivist, developmental paradigm. The Journal of Management Development 18(3), 255–272.
95. VOM BROCKE J, SIMONS A, NIEHAVES B, RIEMER K, PLATTFAUT R and CLEVEN A (2009) Reconstructing the giant: on the importance of rigour in documenting the literature search process. Proceedings of the European Conference on Information Systems 9, 2206–2217.
96. VOM BROCKE J, SIMONS A, RIEMER K, NIEHAVES B, PLATTFAUT R and CLEVEN A (2015) Standing on the shoulders of giants: challenges and recommendations of literature search in information systems research, Communications of the Association for Information Systems 37, 9.
97. WALKER LO and AVANT KC (2010) Strategies for Theory Construction in Nursing. Prentice Hall, San Francisco, CA.
98. WALL JD, STAHL BC and SALAM AF (2015) Critical discourse analysis as a review methodology: an empirical example, Communications of the Association for Information Systems 37, 11.
99. WARD S and READ L (1983) ‘‘Introduction’’ In Knowledge Structure and Use: Implications for Synthesis and Interpretation (Ward S and Read L, Eds), pp1–18. Temple University Press, Philadelphia.
100. WATSON R (2015) Beyond being systematic in literature reviews in IS. Journal of Information Technology 30(2), 185–187.
101. WEBSTER J and WATSON RT (2002) Analyzing the past to prepare for the future: writing a literature review. MIS Quarterly 26(2), xiii–xxiii.
102. WOLFSWINKEL JF, FURTMUELLER E and WILDEROM CP (2013) Using grounded theory as a method for rigorously reviewing literature. European Journal of Information Systems 22(1), 45–55.
103. WONG G, GREENHALGH T, WESTHORP G, BUCKINGHAM J and PAWSON R (2013) RAMESES publication standards: realist syntheses. BMC Medicine 11(1), 1.