A Taxonomy of Evaluation Methods for Information Systems Artifacts

Journal of Management Information Systems

Volumn 32, Issue 3, 2015, Pages 229-267

  Prat, Nicolas ^a   Comyn Wattiau, Isabelle ^b   Akoka, Jacky ^b  

^a ESSEC BUSINESS SCHOOL   (France)

^b CEDRIC CNAM   (France)

Author keywords

artifact evaluation; content analysis; design evaluation; design science; system design; taxonomy

Indexed keywords

COMPUTATIONAL COMPLEXITY; DESIGN; ECONOMIC AND SOCIAL EFFECTS; PARALLEL PROCESSING SYSTEMS; SYSTEMS ANALYSIS; TAXONOMIES;

ALGORITHMIC COMPLEXITY; ARTIFACT EVALUATION; CONTENT ANALYSIS; DESIGN EVALUATION; DESIGN SCIENCE; DESIGN SCIENCE RESEARCHES (DSR); EVALUATION METHODS; ORGANIZATIONAL IMPACTS;

INFORMATION SYSTEMS;

EID: 84951112173     PISSN: 07421222     EISSN: 1557928X     Source Type: Journal    
DOI: 10.1080/07421222.2015.1099390     Document Type: Article

References (69)

1. Abbasi, A.; Zahedi, F.M.; Zeng, D.; Chen, Y.; Chen, H.; and Nunamaker, J.F. Enhancing predictive analytics for anti-phishing by exploiting website genre information. Journal of Management Information Systems, 31, 4 (Spring 2015), 109-157.
2. Ackoff, R.L. Towards a system of systems concepts. Management Science, 17, 11 (1971), 661-671.
3. Aier, S., and Fischer, C. Criteria of progress for information systems design theories. Information Systems and e-Business Management, 9, 1 (2011), 133-172.
4. Arnott, D., and Pervan, G. Design science in decision support systems research: An assessment using the Hevner, March, Park, and Ram guidelines. Journal of the Association for Information Systems, 13, 11 (2012), 923-949.
5. Bergman, M.; Lyytinen, K.; and Mark, G. Boundary objects in design: An ecological view of design artifacts. Journal of the Association for Information Systems, 8, 11 (2007), 546-568.
6. Bondi, A.B. Characteristics of scalability and their impact on performance. Proceedings of the Second International Workshop on Software and Performance (WOSP 2000). Ottawa: ACM, 2000, pp. 195-203.
7. Bowler, T. General Systems Thinking: Its Scope and Applicability. New York: North Holland, 1981.
8. Checkland, P., and Scholes, J. Soft Systems Methodology in Action. Chichester:Wiley, 1990.
9. Chen, H. Editorial: Design science, grand challenges, and societal impacts. ACM Transactions on Management Information Systems, 2, 1 (2011), 1:1-1:10.
10. Churchman, C.W. The Design of Inquiring Systems: Basic Concepts of Systems and Organization. New York: Basic Books, 1971.
11. Cleven, A.; Gubler, P.; and Hüner, K.M. Design alternatives for the evaluation of design science research artifacts. In V. Vaishnavi and S. Purao (eds.), Proceedings of the Fourth International Conference on Design Science Research in Information Systems and Technology (DESRIST 2009). Philadelphia, PA: ACM, 2009, pp. 1-8.
12. Coners, A., and Matthies, B. A content analysis of content analyses in IS research: Purposes, data sources, and methodological characteristics. In K. Siau, Q. Li, and X. Guo (eds.), Proceedings of the Eighteenth Pacific Asia Conference on Information Systems (PACIS 2014)/ Chengdu, China: Association for Information Systems, 2014, pp. 1-16.
13. Creighton, C., and Hanash, S. Mining gene expression databases for association rules. Bioinformatics, 19, 1 (2003), 79-86.
14. Davis, F.D. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 3 (1989), 319-340.
15. DeLone, W.H., and McLean, E.R. Information systems success: The quest for the dependent variable. Information Systems Research, 3, 1 (1992), 60-95.
16. Fischer, C. The information systems design science research body of knowledge: A citation analysis in recent top-journal publications. In P.B. Seddon and S. Gregor (eds.), Proceedings of the Fifteenth Pacific Asia Conference on Information Systems (PACIS 2011)/ Brisbane: Association for Information Systems, 2011, pp. 1-12.
17. Gill, T.G., and Hevner, A.R. A fitness-utility model for design science research. ACM Transactions on Management Information Systems, 4, 2 (2013), 1-24.
18. Glinz, M. On non-functional requirements. In A. Sutcliffe and P. Jalote (eds.), Proceedings of the Fifteenth IEEE International Requirements Engineering Conference (RE 2007). New Delhi: IEEE, 2007, pp. 21-26.
19. Gregor, S. The nature of theory in information systems. MIS Quarterly, 30, 3 (2006), 611-642.
20. Gregor, S., and Hevner, A.R. Positioning and presenting design science research for maximum impact. MIS Quarterly, 37, 2 (2013), 337-355.
21. Gregor, S., and Iivari, J. Designing for mutability in information systems artifacts. In D. Hart and S. Gregor (eds.), Information Systems Foundations: Theory, Representation and Reality. Canberra: Australian National University Press, 2007, pp. 3-24.
22. Gregor, S., and Jones, D. The anatomy of a design theory. Journal of the Association for Information Systems, 8, 5 (2007), 312-335.
23. Hahsler, M.; Buchta, C.; Grün, B.; and Hornik, K. arules: Mining association rules and frequent itemsets, R package version 1.1-6, http://CRAN.R-project.org/package=arules, 2014.
24. Hair, J.; Black, W.; Babin, B.; and Anderson, R. Multivariate Data Analysis: A Global Perspective. 7th ed. Upper Saddle River, NJ: Hall, 2010.
25. Henderson, J.C., and Venkatraman, N. Strategic alignment: Leveraging information technology for transforming organizations. IBM Systems Journal, 32, 1 (1993), 4-16.
26. Hevner, A.R.; March, S.T.; Park, J.; and Ram, S. Design science in information systems research. MIS Quarterly, 28, 1 (2004), 75-105.
27. Huang, C.-Y., and Kuo, S.-Y. Analysis of incorporating logistic testing-effort function into software reliability modeling. IEEE Transactions on Reliability, 51, 3 (2002), 261-270.
28. Huysmans, P., and De Bruyn, P. A mixed methods approach to combining behavioral and design research methods in information systems research. Proceedings of the Twenty-First European Conference on Information Systems (ECIS 2013). Utrecht, the Netherlands: Association for Information Systems, 2013, pp. 1-12.
29. ISO/IEC/IEEE, Systems and Software Engineering: Vocabulary, ISO/IEC/IEEE 24765:2010(E), December 2010, pp. 1-418.
30. Le Moigne, J.-L. Modeling for reasoning socio-economic behaviors. Cybernetics and Human Knowing, 13, 3-4 (2006), 9-26.
31. Liu, F., and Myers, M.D. An analysis of the AIS basket of top journals. Journal of Systems and Information Technology, 13, 1 (2011), 5-24.
32. Lu, M.-T., and Yeung, W.-L. A framework for effective commercial web application development. Internet Research, 8, 2 (1998), 166-173.
33. Lukyanenko, R., and Parsons, J. Reconciling theories with design choices in design science research. In J. vom Brocke, R. Hekkala, S. Ram, and M. Rossi (eds.), Proceedings of the Eighth International Conference on Design Science Research in Information Systems and Technology (DESRIST 2013). Helsinki: Springer Verlag, 2013, pp. 165-180.
34. March, S.T., and Smith, G.F. Design and natural science research on information technology. Decision Support Systems, 15, 4 (1995), 251-266.
35. Mathiassen, L.; Chiasson, M.; and Germonprez, M., Compositional styles in action research: A critical analysis of leading information systems journals. Sprouts: Working Papers on Information Systems, 9, 35 (2009), Lancaster University, UK. http://sprouts.aisnet.org/9-35/.
36. Moore, G., and Benbasat, I. Development of an instrument to measure the perceptions of adopting an information technology innovation. Information Systems Research, 2, 3 (1991), 192-222.
37. Myers, M.D., and Venable, J.R. A set of ethical principles for design science research in information systems. Information and Management, 51, 6 (2014), 801-809.
38. Nickerson, R.C.; Varshney, U.; and Muntermann, J. A method for taxonomy development and its application in information systems. European Journal of Information Systems, 22, 3 (2013), 336-359.
39. Nunamaker, J.F.; Chen, M.; and Purdin, T.D.M. Systems development in information systems research. Journal of Management Information Systems, 7, 3 (1990-91), 89-106.
40. Nunamaker, J.F.; Derrick, D.C.; Elkins, A.C.; Burgoon, J.K.; and Patton, M.W. Embodied conversational agent-based kiosk for automated interviewing. Journal of Management Information Systems, 28, 1 (2011), 17-48.
41. Offermann, P.; Blom, S.; Schönherr, M.; and Bub, U. Artifact types in information systems design science: A literature review. In R. Winter, J.L. Zhao, and S. Aier (eds.), Proceedings of the Fifth International Conference on Design Science Research in Information Systems and Technology (DESRIST 2010). St. Gallen: Springer Verlag, 2010, pp. 77-92.
42. Orlikowski, W.J., and Iacono, C.S. Research commentary: Desperately seeking the "IT" in IT research: A call to theorizing the IT artifact. Information Systems Research, 12, 2 (2001), 121-134.
43. Peffers, K.; Rothenberger, M.; Tuunanen, T.; and Vaezi, R. Design science research evaluation. In K. Peffers, M. Rothenberger, and B. Kuechler (eds.), Proceedings of the Seventh International Conference on Design Science Research in Information Systems and Technology (DESRIST 2012). Las Vegas: Springer Verlag, 2012, pp. 398-410.
44. Peffers, K.; Tuunanen, T.; Rothenberger, M.A.; and Chatterjee, S. A design science research methodology for information systems research. Journal of Management Information Systems, 24, 3 (2007-8), 45-77.
45. R Core Team, R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing, 2015. http://www.R-project.org.
46. Sangupamba Mwilu, O.; Prat, N.; and Comyn-Wattiau, I. Business intelligence and big data in the cloud: oppportunities for design-science researchers. In M. Indulska and S. Purao (eds.), Proceedings of ER 2014 Workshops. Atlanta, GA: Springer International, 2014, pp. 75-84.
47. Siau, K., and Rossi, M. Evaluation techniques for systems analysis and design modelling methods: A review and comparative analysis. Information Systems Journal, 21, 3 (2011), 249-268.
48. Simon, H. The Sciences of the Artificial. 3rd ed. Cambridge, MA: MIT Press, 1996.
49. Skyttner, L. General Systems Theory: Problems, Perspectives, Practice. 2nd ed. Singapore: World Scientific, 2005.
50. Sonnenberg, C., and vom Brocke, J. Evaluations in the science of the artificial: Reconsidering the build-evaluate pattern in design science research. In K. Peffers, M. Rothenberger, and B. Kuechler (eds.), Proceedings of the Seventh International Conference on Design Science Research in Information Systems and Technology (DESRIST 2012). Las Vegas: Springer Verlag, 2012, pp. 381-397.
51. Spanoudakis, G., and Constantopoulos, P. Elaborating analogies from conceptual models. International Journal of Intelligent Systems, 11, 11 (1996), 917-974.
52. Sun, Y., and Kantor, P.B. Cross-evaluation: A new model for information system evaluation. Journal of the American Society for Information Science and Technology, 57, 5 (2006), 614-628.
53. Tremblay, M.C.; Hevner, A.R.; and Berndt, D.J. Focus groups for artifact refinement and evaluation in design research. Communications of the Association for Information Systems, 26, 1 (2010), 599-618.
54. Twyman, N.W.; Lowry, P.B.; Burgoon, J.K.; and Nunamaker, J.F. Autonomous scientifically controlled screening systems for detecting information purposely concealed by individuals. Journal of Management Information Systems, 31, 3 (2014), 106-137.
55. Vaishnavi, V., and Kuechler, B., Design Science Research in Information Systems. http://desrist.org/design-research-in-information-systems, January 2004 (last updated October 2013), pp. 1-45.
56. Venable, J. Rethinking design theory in information systems. In J. vom Brocke, R. Hekkala, S. Ram, and M. Rossi (eds.), Proceedings of the Eighth International Conference on Design Science Research in Information Systems and Technology (DESRIST 2013). Helsinki: Springer Verlag, 2013, pp. 136-149.
57. Venable, J.; Pries-Heje, J.; and Baskerville, R. A comprehensive framework for evaluation in design science research. In K. Peffers, M. Rothenberger, and B. Kuechler (eds.), Proceedings of the Seventh International Conference on Design Science Research in Information Systems and Technology (DESRIST 2012). Las Vegas: Springer Verlag, 2012, pp. 423-438.
58. Venable, J.; Pries-Heje, J.; and Baskerville, R. FEDS: A framework for evaluation in design science research. European Journal of Information Systems (advance online publication, November 2014), doi:10.1057/ejis.2014.36.
59. von Bertalanffy, L. General System Theory: Foundations, Development, Applications. Rev. ed. New York: George Braziller, 1969.
60. Walls, J.G.; Widmeyer, G.R.; and El Sawy, O.A. Building an information system design theory for vigilant EIS. Information Systems Research, 3, 1 (1992), 36-59.
61. Wand, Y., and Weber, R. On the ontological expressiveness of information systems analysis and design grammars. Information Systems Journal, 3, 4 (1993), 217-237.
62. Wang, R., and Strong, D. Beyond accuracy: what data quality means to data consumers. Journal of Management Information Systems, 12, 4 (1996), 5-33.
63. Wang, S., and Wang, H. Towards innovative design research in information systems. Journal of Computer Information Systems, 51, 1 (2010), 11-18.
64. Weber, R.P. Basic Content Analysis. 2nd ed. Newbury Park, CA: Sage, 1990.
65. Weinberg, G. An Introduction to General Systems Thinking. New York: Wiley, 1975.
66. Wieringa, R. Tutorial #2: Empirical validation research methods. Communication at the Sixth International Conference on Research Challenges in Information Science (RCIS 2012). Valencia, Spain, May 2012.
67. Williams, K.; Chatterjee, S.; and Rossi, M. Design of emerging digital services: A taxonomy. European Journal of Information Systems, 17, 5 (2008), 505-517.
68. Winter, R. Design science research in Europe. European Journal of Information Systems, 17, 5 (2008), 470-475.
69. Zlotkin, G., and Rosenschein, J.S. Cooperation and conflict resolution via negotiation among autonomous agents in non-cooperative domains. IEEE Transactions on Systems, Man and Cybernetics, 21, 6 (1991), 1317-1324.