Applying the criteria indicators and metrics evaluation method on ICT research: The HUMBOLDT project

Research Evaluation

Volumn 22, Issue 1, 2013, Pages 30-40

  Khan, Zaheer ^a   Ludlow, David ^a   Rix, Joachim ^b  

^a UNIVERSITY OF THE WEST OF ENGLAND   (United Kingdom)

^b FRAUNHOFER IGD   (Germany)

Author keywords

collaborative IT projects; criteria indicators and metrics; evaluation stakeholders; IT research and development; IT research project evaluation; user based evaluation methodology

Indexed keywords

EID: 84872255117     PISSN: 09582029     EISSN: 14715449     Source Type: Journal    
DOI: 10.1093/reseval/rvs037     Document Type: Article

References (43)

1. Andras, P. (2011) 'Research: Metrics, Quality, and Management Implications', Research Evaluation, 20/2: 90-106.
2. Babar, M. A., Zhu, L. and Jeffery, R. 'A framework for classifying and comparing software architecture evaluation methods', Proceedings of the 15th Australian Software Engineering Conference (ASWEC), pp. 309-19. Melbourne Australia: IEEE Computer Society.
3. Basili, V., Caldiera, G. and Rombach, D. (1994) Goal Question Metric Approach, Encyclopedia of Software Engineering, 2 vols, pp. 528-32. Chichester: John Wiley & Sons.
4. Basili, V., Donzelli, P. and Asgari, S. (2004) 'A Unified Model of Dependability: Capturing Dependability in Context', IEEE Software, 21/6: 19-25.
5. Briand, L., Morasca, S. and Basili, V. (2002) 'An Operational Process for Goal-Driven Definition of Measures', IEEE Transactions on Software Engineering, 28/12: 1106-25.
6. Brooke, J. (1996) 'SUS-A quick and Dirty Usability Scale'. In: Jordan, P., Thomas, B. and Weerdmeester, B. (eds) Usability Evaluation in Industry, pp. 189-94. London: Taylor and Francis.
7. Bryson, J., Patton, M. and Bowman, R. (2011) 'Working with Evaluation Stakeholders: A Rationale, Step-Wise Approach and Toolkit', Evaluation and Program Planning, 34/1: 1-12.
8. CASCADOSS Consortium (ed.). (2007) Evaluation Criteria for Open Source GIS and RS Software, , accessed 4 Nov 2011.
9. de Vries, M., di Donato, P. and Penninga, F. (2007) A7.1-D1 Concept of Application-Specific Harmonised Data Models, (Darmstadt: HUMBOLDT Consortium) accessed 4 Oct 2012.
10. Dobrica, L. and Niemela, E. (2002) 'A Survey on Software Architecture Analysis Methods', IEEE Transactions on Software Engineering, 28/7: 638-53.
11. Falessi, D. et al. (2009) 'Applying Empirical Software Engineering to Software Architecture: Challenges and Lessons Learned', Empirical Software Engineering, 15/3: 250-76.
12. Fichtinger, A. et al. (2011) 'Data Harmonisation Put into Practice by the HUMBOLDT Project', International Journal of Spatial Data Infrastructures Research, 6: 234-60.
13. Finstad, K. (2010) 'The Usability Metric for User Experience', Interacting with Computers, 22: 232-327.
14. GMES. (2012) 'Global Monitoring for Environment and Security', accessed 4 Oct 2012.
15. Gustedt, J., Jeannot, E. and Quinson, M. (2009) 'Experimental validation in large-scale systems: a survey of methodologies', INRIA NANCY-GRAND EST Report: inria-00364180, version 1, p.16.
16. HALE. (2011) 'HUMBOLDT Alignment Editor. Wiki',
17. 'User Documentation', accessed 5 Oct 2012.
18. HUDSON. (2010) 'HUDSON: Extensible continuous integration server', accessed 15 Nov 2011.
19. HUMBOLDT. (2010a) 'Framework Tools', accessed 25 Oct 2012.
20. -. (2010b) 'Handbook of Standards', accessed 25 Oct 2012.
21. -. (2010c) 'Scenarios', accessed 25 Oct 2012.
22. -. (2010d) 'Technical Evaluation', accessed 25 Oct 2012.
23. INSPIRE. (2007) accessed 4 Oct 2012.
24. ISO/IEC 9126-1. (2001) Software Engineering-Product Quality-Part 1: Quality Model.
25. Jong, S. et al. (2011) 'Evaluation of Research in Context: An Approach and Two Cases', Research Evaluation, 20/1: 61-72.
26. Juristo, N. (1997) 'Structuring the knowledge-based systems evaluation process', Proceedings of Intelligent Information Systems '97, Grand Bahama Island, 8-10 December 1997, pp. 266-70.
27. Juristo, N. and Morant, J. (1998) 'Common Framework for the Evaluation Process of KBS and Conventional Software', Elsevier Knowledge-Based Systems, 11/2: 145-59. (Pubitemid 128413124)
28. Khan, Z., Odeh, M. and McClatchey, R. (2011) 'Bridging the gap between business process models and service-oriented architectures with reference to the grid environment', International Journal of Grid and Utility Computing, 2/4: 253-83.
29. Kotonya, G. and Sommerville, I. (2002) Requirements Engineering: Processes and Techniques. Chichester: John Wiley & Sons.
30. Lindvall, M. et al. (2007) 'Experimenting with Software Testbeds for Evaluating New Technologies', Empirical Software Engineering, 12: 417-44. (Pubitemid 47115403)
31. OASIS Consortium (ed.). (2006) 'POS 1 Evaluation Report', Version 1.0.
32. OGC. (2012) 'Open Geospatial Consortium'. (2012), accessed 4 Oct 2012.
33. Ostreika, A. (2010) Evaluation of Benefit Analysis-The HUMBOLDT Project Deliverable A10.5, accessed 25 Oct 2012.
34. RESPECT Consortium (ed.). (1998) User-Centred Requirements Handbook. Version 3.3, accessed 25 Oct 2012.
35. RISE (ed.). (2007) Report on the Operation of the RISE Testing System, accessed 25 Oct 2012.
36. Smite, D. et al. (2010) 'Empirical Evidence in Global Software Engineering: A Systematic Review', Empirical Software Engineering, 15/1: 91-118.
37. Sommerville, I. (2006) Software Engineering. Wokingham, UK: Addison-Wesley.
38. Sonar: Code Analysis Server. (2010) accessed 25 Oct 2012.
39. The HUMBOLDT Project. (2006-2011) Sixth Framework Programme, European Commission, accessed 25 Oct 2012.
40. The HUMBOLDT Project Community. (2006-2011), accessed 25 Oct 2012.
41. Tiits, K. (2003) Usability of Geographic Information Systems in Internet. A Case Study of Journey Planners, (Master Thesis at Tartu University, Institute of Geography) accessed 25 Oct 2012.
42. Villa, P. et al. (2012) 'Harmonisation Requirements and Capabilities Towards a European Spatial Data Infrastructure (ESDI): The HUMBOLDT Protected Areas Scenario', International Journal of Digital Earth, 5/5: 417-38.
43. WebPark Consortium (ed.). (2002) Usability Testing Template, D2.2.3, V05, accessed 4 Oct 2012.