Evolutionary development: A model for the design, implementation, and evaluation of ICT for education programmes

Journal of Computer Assisted Learning

Volumn 28, Issue 2, 2012, Pages 81-98

  Rodriguez P ^a   Nussbaum, M ^b   Dombrovskaia, L ^c  

^a UNIVERSITY OF CHILE   (Chile)

^b PONTIFICIA UNIVERSIDAD CATÓLICA DE CHILE   (Chile)

^c DEPARTAMENTO DE FÍSICA   (Chile)

Author keywords

Assessment; Cost effectiveness; Design research; Monitoring and evaluation; Primary and secondary education; Scaling up

Indexed keywords

EID: 84858283733     PISSN: 02664909     EISSN: 13652729     Source Type: Journal    
DOI: 10.1111/j.1365-2729.2011.00419.x     Document Type: Article

References (100)

1. Agodini R., Dynarski M., Honey M. & Levin D. (2003) The effectiveness of educational technology: issues and recommendations for the national study. Mathematica Policy Research, Inc. Available at: (last accessed 17 November 2010).
2. Anderson R.E. & Plomp T. (2009) Introduction. In Cross-National Information and Communication Technology Policies and Practices in Education, 2nd edition (eds T. Plomp, R.E. Anderson, N. Law & A. Quayle ), pp. 3-17. IAP-Information Age Publishing, Greenwich, CT.
3. Balanskat A., Blamire R. & Kefala S. (2006) The ICT impact report: a review of studies of ICT impact on schools in Europe. Available at: (last accessed 17 November 2010).
4. Barrera-Osorio F. & Linden L. (2009) The use and misuse of computers in education: evidence from a randomized controlled trial of a language arts program1. Available at: (last accessed 17 November 2010).
5. Bielaczyc K. (2006) Designing social infrastructure: critical issues in creating learning environments with technology. Journal of the Learning Sciences 15, 301-329.
6. Bottino R., Chiappini G., Forcheri P., Lemut E. & Molfino M. (1999) Activity theory: a framework for design and reporting on research projects based on ICT. Education and Information Technologies 4, 279-293.
7. Burns T. & Ungerleider C. (2003) Information and communication technologies in elementary and secondary education. International Journal Educational Policy, Research and Practice 3, 27-54.
8. Bustos H. & Nussbaum M. (2009) An experimental study of the inclusion of technology in higher education. Computer Applications in Engineering Education 17, 100-107.
9. Campuzano L., Dynarski M., Agodini R. & Rall K. (2009) Effectiveness of reading and mathematics software products: findings from two student cohorts. Available at: (last accessed 17 November 2010).
10. Carter R.A., Antón A.I., Williams L. & Dagnino A. (2001) Evolving beyond requirements creep: a risk-based evolutionary prototyping model. In Requirements Engineering, 5th IEEE International Symposium, pp. 94-101, Toronto, ON, Canada, 27-31 August 2001.
11. Castro C. (2004) Are new technologies better technologies? For whom? In Adapting Technology for School Improvement: A Global Perspective (eds D.W. Chapman & L.O. Mählck ), pp. 39-54. International Institute for Educational Planning - Unesco, Paris.
12. Chatterji M. (2005) Evidence on 'what works': an argument for Extended-Term Mixed-Method (ETMM) evaluation designs. Educational Researcher 34, 14-24.
13. Coburn C.E. (2003) Rethinking scale: moving beyond numbers to deep and lasting change. Educational Researcher 32, 3-12.
14. Conlon T. & Simpson M. (2003) Silicon Valley versus Silicon Glen: the impact of computers upon teaching and learning: a comparative study. British Journal of Educational Technology 34, 137-150.
15. Cortez C., Nussbaum M., López X., Rodríguez P., Santelices R., Rosas R. & Marianov V. (2005) Teachers' support with ad-hoc collaborative networks. Journal of Computer Assisted Learning 21, 171-180.
16. Cox M. (2008) Research IT in education. In International Handbook of Information Technology in Primary and Secondary Education (eds J. Voogt & G. Knezek ), pp. 965-981. Springer, New York, NY.
17. Cox M. & Abbot C. (2004) ICT and attainment: a review of the research literature. ICT in Schools Research and Evaluation Series - No. 17, DfES-Becta. Available at: (last accessed 17 November 2010).
18. Cox M. & Marshall G. (2007) Effects of ICT: do we know what we should know? Education and Information Technologies 12, 59-70.
19. Cox M. & Webb M. (2004) ICT and pedagogy: a review of the research literature. ICT in Schools Research and Evaluation Series - No. 18, DfES-Becta. Available at: (last accessed 17 November 2010).
20. Culp K.M., Honey M. & Mandinach E. (2003) A retrospective on twenty years of education technology policy. U.S. Department of Education, Office of Educational Technology. Available at: (last accessed 17 November 2010).
21. Davis A., Bersoff E. & Comer E. (1988) A strategy for comparing alternative software development life cycle models. IEEE Transactions on Software Engineering 14, 1453-1461.
22. Dede C. (2006) Scaling up: evolving innovations beyond ideal settings to challenging contexts of practice. In Cambridge Handbook of the Learning Sciences (ed. R.K. Sawyer ), pp. 551-566. Cambridge University Press, Cambridge, UK.
23. Dillenbourg P. (1999) Collaborative Learning: Cognitive and Computational Approaches. Advances in Learning and Instruction Series. Elsevier Science, New York, NY.
24. Dillenbourg P. (2008) Integrating technologies into educational ecosystems. Distance Education 29, 127-140.
25. Dynarski M., Agodini R., Heaviside S., Novak T., Carey N., Campuzano L., Means B., Murphy R, Penuel W., Javitz H., Emery D. & Sussex W. (2007) Effectiveness of reading and mathematics software products: findings from the first student cohort. U.S. Department of Education, Institute of Education Sciences. Available at: (last accessed 17 November 2010).
26. Earle R.S. (2002) The integration of instructional technology into public education: promises and challenges. Educational Technology 42, 5-13.
27. Fan X. (2001) Statistical significance and effect size in education research: two sides of a coin. The Journal of Educational Research 94, 275-282.
28. Feinstein O.N. & Picciotto R. (2001) Evaluation and Poverty Reduction. World Bank Series on Evaluation and Development (Vol. 3). Transaction Publishers. New Brunswick, NJ.
29. Gilb T. (1981) Evolutionary development. ACM SIGSOFT Software Engineering Notes 6, 17-17.
30. Harris D.N. (2009) Toward policy-relevant benchmarks for interpreting effect sizes: combining effects with costs. Educational Evaluation and Policy Analysis 31, 3-29.
31. Harrison C., Comber C., Fischer T., Haw K., Lewin C., Lunzer E., McFarlane A., Mavers D., Scrimshaw P., Somekh B. & Watling R. (2002) ImpaCT2: the impact of information and communication technologies on pupil learning and attainment. BECTA. Available at: (last accessed 17 November 2010).
32. Hayes D.N.A. (2007) ICT and learning: lessons from Australian classrooms. Computers & Education 49, 385-395.
33. Hennessy S., Ruthven K. & Brindley S. (2005) Teacher perspectives on integrating ICT into subject teaching: commitment, constraints, caution, and change. Journal of Curriculum Studies 37, 155-192.
34. Hinostroza E., Labbé C. & Claro M. (2005) ICT in Chilean schools: students' and teachers' access and use of ICT. Human Technology 1, 246-264.
35. Infante C. & Nussbaum M. (2010) Third order barriers to the integration of technology in the classroom. Working paper.
36. Infante C., Hidalgo P., Nussbaum M., Alarcón R. & Gottlieb A. (2009) Multiple mice based collaborative one-to-one learning. Computers & Education 53, 393-401.
37. Infante C., Weitz J., Reyes T., Nussbaum M., Gómez F. & Radovic D. (2010) Co-located collaborative learning video game with single display groupware. Interactive Learning Environments 18, 177-195.
38. Kirkman C. (2000) A model for the effective management of information and communications technology development in schools derived from six contrasting case studies. Technology, Pedagogy and Education 9, 37-52.
39. Kozma R.B. (1991) Learning with media. Review of Educational Research 61, 179-211.
40. Kozma R.B. (2003) Technology and classroom practices: an international study. Journal of Research on Technology in Education 36, 1-14.
41. Kulik J. (2003) Effects of using instructional technology in elementary and secondary schools: what controlled evaluation studies say. Technical Report, SRI International.
42. Lai K. (2008) ICT supporting the learning process: the premise, reality, and promise. In International Handbook of Information Technology in Primary and Secondary Education (eds J. Voogt & G. Knezek ), pp. 215-230. Springer, New York, NY.
43. Larman C. & Basili V. (2003) Iterative and incremental developments. A brief history. Computer 36, 47-56.
44. Laurillard D. (2007) Modelling benefits-oriented costs for technology enhanced learning. Higher Education 54, 21-39.
45. Law N. (2009) Curriculum and staff development for ICT In education. In Cross-National Information and Communication Technology Policies and Practices in Education, 2nd edition (eds T. Plomp, R.E. Anderson, N. Law & A. Quayle ), pp. 19-39. IAP-Information Age Publishing, Greenwich, CT.
46. Levin H. (1983) Cost-Effectiveness: A Primer. New Perspectives on Evaluation. Sage Publications, Thousand Oaks, CA.
47. Liao Y. & Hao Y. (2008) Large-scale studies and quantitative methods. In International Handbook of Information Technology in Primary and Secondary Education (eds J. Voogt & G. Knezek ), pp. 1019-1035. Springer, New York, NY.
48. Lim C.P. (2002) A theoretical framework for the study of ICT in schools: a proposal. British Journal of Educational Technology 33, 411-421.
49. Lipponen L., Rahikainen M., Lallimo J. & Hakkarainen K. (2003) Patterns of participation and discourse in elementary students' computer-supported collaborative learning. Learning and Instruction 13, 487-509.
50. Luckin R. (2008) The learner centric ecology of resources: a framework for using technology to scaffold learning. Computers & Education 50, 449-462.
51. Luckin R. (2010) The ecology of resources model of context: a unifying representation for the development of learning-oriented technologies. In Mobile, Hybrid, and On-Line Learning, International Conference on (eds A.M. Gadomski, B. Krämer, C.Y. Lester & M. Popescu ), pp. 47-52. IEEE Computer Society, Los Alamitos, CA.
52. McDonald S., Keesler V.A., Kauffman N.J. & Schneider B. (2006) Scaling-up exemplary interventions. Educational Researcher 35, 15-24.
53. McFarlane A. (2001) Perspectives on the relationships between ICT and assessment. Journal of Computer Assisted Learning 17, 227-234.
54. Margolis J., Nussbaum M., Rodríguez P. & Rosas R. (2006) Methodology for evaluating a novel education technology: a case study of handheld video games in Chile. Computers & Education 46, 174-191.
55. Marshall G. & Cox M. (2008) Research methods: their design, applicability and reliability. In International Handbook of Information Technology in Primary and Secondary Education (eds J. Voogt & G. Knezek ), pp. 983-1002. Springer, New York, NY.
56. Mioduser D., Nachmias R., Tubin D. & Forkosh-Baruch A. (2003) Analysis schema for the study of domains and levels of pedagogical innovation in schools using ICT. Education and Information Technologies 8, 23-36.
57. Mooij T. & Smeets E. (2001) Modelling and supporting ICT implementation in secondary schools. Computers & Education 36, 265-281.
58. Moonen J. (2008) Policy from a global perspective. In International Handbook of Information Technology in Primary and Secondary Education (eds J. Voogt & G. Knezek ), pp. 1171-1178. Springer, New York, NY.
59. Moyle K. (2008) Total cost of ownership and total value of ownership. In International Handbook of Information Technology in Primary and Secondary Education (eds J. Voogt & G. Knezek ), pp. 615-631. Springer, New York, NY.
60. Nachmias R., Mioduser D., Cohen A., Tubin D. & Forkosh-Baruch A. (2004) Factors involved in the implementation of pedagogical innovations using technology. Education and Information Technologies 9, 291-308.
61. Nussbaum M., Alvarez C., McFarlane A., Gomez F., Claro S. & Radovic D. (2009) Technology as small group face-to-face collaborative scaffolding. Computers & Education 52, 147-153.
62. Nussbaum M. & Rosas R. (2004) Enlaces Móvil: Mobile Technology in the Classroom. Final Report. Enlaces, Center for Educational Technology, Ministry of Education, Santiago, Chile.
63. OECD (2006) Are students ready for a technology-rich world? What PISA studies tell us. OECD Publications, Paris. Available at: (last accessed 17 November 2010).
64. OECD (2010) Are the new millennium learners making the grade?: technology use and educational performance in PISA 2006, Centre for Educational Research and Innovation, OECD Publishing. Paris.
65. Pawar U.S., Pal J. & Toyama K. (2006) Multiple mice for computers in education in developing countries. In Information and Communication Technologies and Development, 2006. ICTD ′06, (pp. 64-71). Presented at the Information and Communication Technologies and Development, 2006, ICTD ′06.
66. Penuel W.R. (2005) Recommendations for evaluation research within educational innovations. Center for Technology in learning, SRI International. Available at: (last accessed 17 November 2010).
67. Pilkington R.M. (2008) Measuring the impact of information technology on student's learning. In International Handbook of Information Technology in Primary and Secondary Education (eds J. Voogt & G. Knezek ), pp. 1003-1018. Springer, New York, NY.
68. Plomp T., Pelgrum W. & Law N. (2007) SITES2006-international comparative survey of pedagogical practices and ICT in education. Education and Information Technologies 12, 83-92.
69. Raudenbush S.W. (2008) Advancing educational policy by advancing research on instruction. American Educational Research Journal 45, 206-230.
70. Reeves T. (1994) Evaluating what really matters in computer-based education. In Computer Education: New Perspectives (eds M. Wild & D. Kirkpatrick ), pp. 219-246. MASTEC, Perth, Australia. Available at: (last accessed 17 November 2010).
71. Reeves T. (2006) Design research from a technology perspective. In Educational Design Research (eds J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen ), pp. 52-66. Routledge, London.
72. Reeves T. (2008) Evaluation of the design and development of IT tools in education. In International Handbook of Information Technology in Primary and Secondary Education (eds J. Voogt & G. Knezek ), pp. 1037-1051. Springer, New York, NY.
73. Reynolds D., Treharne D. & Tripp H. (2003) ICT - the hopes and the reality. British Journal of Educational Technology 34, 151-167.
74. Robertson J. (2002) The ambiguous embrace: twenty years of IT (ICT) in UK primary schools. British Journal of Educational Technology 33, 403-409.
75. Roblyer M. (2005) Educational technology research that makes a difference: series introduction. Contemporary Issues in Technology and Teacher Education 5, 192-201.
76. Rodríguez P. (2008) Standard for An ICT for Education Programme. Enlaces, Center for Technological Education, Ministry of Education, Santiago, Chile. Available at: (last accessed 17 November 2010).
77. Rodríguez P. (2010) A methodology for replication, scaling-up and transference of ICT supported pedagogical models. PhD in Engineering Thesis. Department of Computer Science, College of Engineering, Pontificia Universidad Católica de Chile.
78. Rodríguez P. & Gómez F. (2007) Mobile technology in the classroom: development of a world class product and standard for transforming pedagogical practices. Final report. CORFO Innova Chile 05CT5IP-02.
79. Rodríguez P., Nussbaum M., López X. & Sepúlveda M. (2010) A monitoring and evaluation scheme for an ICT-supported education program in schools. Journal of Educational Technology & Society 13, 166-179.
80. Roschelle J., Rafanan K., Bhanot R., Estrella G., Penuel B., Nussbaum M. & Claro S. (2010) Scaffolding group explanation and feedback with handheld technology: impact on students' mathematics learning. Educational Technology Research and Development 58, 399-419.
81. Sánchez J. & Salinas A. (2008) ICT & learning in Chilean schools: lessons learned. Computers & Education 51, 1621-1633.
82. Slavin R.E. (2008) Perspectives on evidence-based research in education - what works? Issues in synthesizing educational program evaluations. Educational Researcher 37, 5-14.
83. Slavin R. & Smith D. (2009) The relationship between sample sizes and effect sizes in systematic reviews in education. Educational Evaluation and Policy Analysis 31, 500-506.
84. Squires D. & Preece J. (1999) Predicting quality in educational software: evaluating for learning, usability and the synergy between them. Interacting with Computers 11, 467-483.
85. Tearle P. (2003) ICT implementation: what makes the difference? British Journal of Educational Technology 34, 567-583.
86. Tearle P. (2004) A theoretical and instrumental framework for implementing change in ICT in education. Cambridge Journal of Education 34, 331-351.
87. Tondeur J., Braak J.V. & Valcke M. (2007) Curricula and the use of ICT in education: two worlds apart? British Journal of Educational Technology 38, 962-976.
88. Trucano M. (2005) Knowledge Maps: ICTs in Education. ICT and Education Series. infoDev/World Bank, Washington, DC. Available at: (last accessed 17 November 2010).
89. Valdivia R. & Nussbaum M. (2007) Face-to-face collaborative learning in computer science classes. International Journal of Engineering Education 23, 434-440.
90. Valdivia R. & Nussbaum M. (2009) Using multiple choice questions as a pedagogic model for face-to-face CSCL. Computer Applications in Engineering Education 17, 89-99.
91. Van den Akker J. (2003) Curriculum perspectives: an introduction. In Curriculum Landscapes and Trends (eds J. van den Akker, U. Hameyer & W. Kuiper ), pp. 1-10. Kluwer, Dordrecht.
92. Van den Akker J., Gravemeijer K., McKenney S. & Nieveen N. (2006) Educational Design Research. Routledge, London.
93. Villanueva C. (2003) Measuring ICT use in education in Asia and the Pacific through performance indicators. Presented at the Joint UNECE/UNCTAD/UNESCO/ITU/ OECD/Eurostat Statistical Workshop on Monitoring the Information Society: Data, Measurement and Methods (Geneva, 8-9 December 2003). Available at: (last accessed 17 November 2010).
94. Wagner D.A., Day B., James T., Kozma R.B., Miller J. & Unwin T. (2005) Monitoring and Evaluation of ICT in Education Projects: A Handbook for Developing Countries. ICT and Education Series. infoDev/World Bank, Washington, DC. Available at: (last accessed 17 November 2010).
95. Wainer J., Dwyer T., Dutra R.S., Covic A., Magalhães V.B., Ferreira L.R.R., Pimenta V.A. & Claudio K. (2008) Too much computer and Internet use is bad for your grades, especially if you are young and poor: results from the 2001 Brazilian SAEB. Computers & Education 51, 1417-1429.
96. Watson D.M. (2001) Pedagogy before technology: re-thinking the relationship between ICT and teaching. Education and Information Technologies 6, 251-266.
97. Zurita G. & Nussbaum M. (2004a) Computer supported collaborative learning using wirelessly interconnected handheld computers. Computers & Education 42, 289-314.
98. Zurita G. & Nussbaum M. (2004b) A constructivist mobile learning environment supported by a wireless handheld network. Journal of Computer Assisted Learning 20, 235-243.
99. Zurita G. & Nussbaum M. (2007) A conceptual framework based on activity theory for mobile CSCL. British Journal of Educational Technology 38, 211-235.
100. Zurita G., Nussbaum M. & Salinas R. (2005) Dynamic grouping in collaborative learning supported by wireless handhelds. Educational Technology & Society 8, 149-161.