Designing Crowdcritique Systems for Formative Feedback

International Journal of Artificial Intelligence in Education

Volumn 27, Issue 3, 2017, Pages 623-663

  Easterday, Matthew W ^a   Rees Lewis, Daniel ^a   Gerber, Elizabeth M ^a  

^a NORTHWESTERN UNIVERSITY   (United States)

Author keywords

Complex problems; Critique; Crowdsourcing; Crowdwork; Design principles; Formative feedback; Project based learning; Social technical systems

Indexed keywords

COMPUTER AIDED INSTRUCTION; CROWDSOURCING; EXPERT SYSTEMS; SCAFFOLDS;

COMPLEX PROBLEMS; CRITIQUE; CROWDWORK; DESIGN PRINCIPLES; FORMATIVE FEEDBACKS; PROJECT BASED LEARNING; TECHNICAL SYSTEMS;

EDUCATION;

EID: 85025069894     PISSN: 15604292     EISSN: 15604306     Source Type: Journal    
DOI: 10.1007/s40593-016-0125-9     Document Type: Article

References (93)

1. Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: 7 research-based principles for smart teaching. San Francisco: Jossey-Bass.
2. Bannan-Ritland, B. (2003). The role of design in research: The integrative learning design framework. Educational Researcher, 32(1), 21–24. Retrieved from JSTOR.org: http://www.jstor.org/stable/3699931.
3. Barrett, T. (2000). Studio critiques of student art: as they are, as they could be with mentoring. Theory Into Practice, 39(1), 29–35.
4. Bēhance. (n.d.). Bēhance. [Web page] Retrieved from http://www.behance.net/
5. Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010). Soylent: A word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User Interface Software and Technology (pp. 313–322).
6. Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating project-based learning: sustaining the doing, supporting the learning. Educational Psychologist, 26(3-4), 369–398.
7. Blumenfeld, P., Fishman, B. J., Krajcik, J., Marx, R. W., & Soloway, E. (2000). Creating usable innovations in systemic reform: scaling up technology-embedded project-based science in urban schools. Educational Psychologist, 35(3), 149–164.
8. Blumer, H. (1954). What is wrong with social theory? American Sociological Review, 19(1), 3–10.
9. Brennan, J., & Williams, R. (2004). Collecting and using student feedback: A guide to good practice. Learning and Teaching Support Network.
10. Brown, A. L. (1992). Design experiments: theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141–178.
11. Brown, A. L., & Campione, J. C. (1994). Guided discovery in a community of learners. The MIT Press.
12. Buxton, B. (2007). Sketching user experiences: Getting the design right and the right design. San Francisco: Morgan Kaufmann.
13. Cho, K., & MacArthur, C. (2010). Student revision with peer and expert reviewing. Learning and Instruction, 20(4), 328–338.
14. Cho, K., & Schunn, C. D. (2007). Scaffolded writing and rewriting in the discipline: a web-based reciprocal peer review system. Computers & Education, 48(3), 409–426.
15. Cho, K., Schunn, C. D., & Charney, D. (2006a). Commenting on writing typology and perceived helpfulness of comments from novice peer reviewers and subject matter experts. Written Communication, 23(3), 260–294.
16. Cho, K., Schunn, C. D., & Wilson, R. W. (2006b). Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. Journal of Educational Psychology, 98(4), 891–901.
17. Cho, K., Chung, T. R., King, W. R., & Schunn, C. (2008). Peer-based computer-supported knowledge refinement: an empirical investigation. Communications of the ACM, 51(3), 83–88.
18. Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. Perspectives on Socially Shared Cognition, 13(1991), 127–149.
19. Collins, A. (1992). Toward a design science of education. In E. Scanlon & T. O’Shea (Eds.), New directions in educational technology (pp. 15–22). Berlin Heidelberg: Springer. doi:10.1007/978-3-642-77750-9_2.
20. Collins, J., & Bielaczyc. (2004). Design research: theoretical and methodological issues. The Journal of the Learning Sciences, 13(1), 15–42.
21. Connolly, T., Jessup, L. M., & Valacich, J. S. (1990). Effects of anonymity and evaluative tone on idea generation in computer-mediated groups. Management Science, 36(6), 689–703.
22. Corbett, A. T., & Anderson, J. R. (2001). Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement and attitudes. In J. Jacko, A. Sears, M. Beaudouin-Lafon, & R. Jacob (Eds.), Proceedings of the ACM CHI’2001 conference on human factors in computing systems (pp. 245–252). New York: ACM Press. Retrieved from http://portal.acm.org/citation.cfm?id=365111.
23. Corbin, J., & Strauss, A. (2007). Basics of qualitative research: Techniques and procedures for developing grounded theory (3rd ed.). Thousand Oaks: Sage.
24. Crinon, J. (2012). The dynamics of writing and peer review at primary school. Journal of Writing Research, 4(2), 121–154.
25. Cross, N. (2006). Designerly ways of knowing. London: Springer.
26. Cross, N. (2011). Design thinking: Understanding how designers think and work. Oxford: Berg.
27. Crowther, P. (2013). Understanding the signature pedagogy of the design studio and the opportunities for its technological enhancement. Journal of Learning Design, 6(3), 18–28.
28. Dannels, D. P., & Martin, K. N. (2008). Critiquing critiques a genre analysis of feedback across novice to expert design studios. Journal of Business and Technical Communication, 22(2), 135–159.
29. Diamond, M. R. (2004). The usefulness of structured mid-term feedback as a catalyst for change in higher education classes. Active Learning in Higher Education, 5(3), 217–231.
30. Dillenbourg, P., & Jermann, P. (2010). Technology for classroom orchestration. In M. S. Khine & I. M. Salhey (Eds.), New science of learning (pp. 525–552). New York: Springer.
31. Dow, S., Gerber, E., & Wong, A. (2013). A pilot study of using crowds in the classroom. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 227–236).
32. Dribbble. (n.d.). Dribbble. [Web page] Retrieved from http://dribbble.com/
33. Easterday, M. W., Rees Lewis, D., & Gerber, E. M. (2014). Design-Based research process: Problems, phases, and applications. In Proceedings of the International Conference of the Learning Sciences (pp. 317–324). June 23-27, 2014, Colorado, USA
34. Easterday, M. W., Rees Lewis, D., & Gerber, E. M. (2015). Crowdsourcing critique. In CSCL crowd 2015, a workshop at the 2015 International Conference on Computer Supported Collaborative Learning. June 7, Gothenburg, Sweden.
35. Edelson, D. C. (2001). Learning-for-use: a framework for the design of technology-supported inquiry activities. Journal of Research in Science Teaching, 38(3), 355–385.
36. Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383. doi:10.2307/2666999.
37. Estellés-Arolas, E., & González-Ladrón-De-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information science, 38(2), 189–200.
38. Gallupe, R. B., Bastianutti, L. M., & Cooper, W. H. (1991). Unblocking brainstorms. The Journal of Applied Psychology, 76(1), 137–142.
39. Gerber, E. M., Marie Olson, J., & Komarek, R. L. D. (2012). Extracurricular design-based learning: preparing students for careers in innovation. International Journal of Engineering Education, 28(2), 317–324.
40. Gero, J. S. (2002). Computational models of creative designing based on situated cognition. In Proceedings of the 4th conference on Creativity and Cognition, (pp. 3–10). New York: ACM.
41. Giles, J. (2005). Internet encyclopaedias go head to head. Nature, 438(7070), 900–901.
42. Goldin, I. M., & Ashley, K. D. (2012). Eliciting formative assessment in peer review. Journal of Writing Research, 4(2), 203–237.
43. Greenberg, M., Gerber, E., & Easterday, M. (2015). Critiki: A scaffolded approach to gathering design feedback from paid crowdworkers. In Proceedings of the 2015 ACM Conference on Creativity and Cognition.
44. Hamer, J., Kell, C., & Spence, F. (2007). Peer assessment using aropä. In Proceedings of the Ninth Australasian Conference on Computing Education (Vol. 66, pp. 43–54).
45. Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York: Routledge.
46. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
47. Hokanson, B. (2012). The design critique as a model for distributed learning. In L. Moller & J. B. Huett (Eds.), The next generation of distance education (pp. 71–83). Springer.
48. Howe, J. (2006). The rise of crowdsourcing. Wired magazine, 14(6), 1–4.
49. Hui, J. S., Gerber, E. M., & Dow, S. P. (2014a). Crowd-based design activities: Helping students connect with users online. In Proceedings of the 2014 Conference on Designing Interactive Systems (pp. 875–884). Retrieved from doi:10.1145/2598510.2598538.
50. Hui, J. S., Greenberg, M. D., & Gerber, E. M. (2014b, February). Understanding the role of community in crowdfunding work. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (pp. 62–74). ACM.
51. Hui, J., Glenn, A., Jue, R., Gerber, E., & Dow, S. (2015, September). Using anonymity and communal efforts to improve quality of crowdsourced feedback. In Third AAAI Conference on Human Computation and Crowdsourcing.
52. Kelley, T., & Littman, J. (2001). The art of innovation: Lessons in creativity from IDEO, America’s Leading Design Firm. New York: Crown Business.
53. Kelly, A. (2004). Design research in education: yes, but is it methodological? The Journal of the Learning Sciences, 13(1), 115–128.
54. Kelly, A. E. (2006). Quality criteria for design research. Educational Design Research, 107–118.
55. Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. J. (2013). The future of crowd work. In Proceedings of the 2013 conference on computer supported cooperative work (pp. 1301–1318).
56. Klebesadel, H., & Kornetsky, L. (2009). Critique as signature pedagogy in the arts. In Exploring signature pedagogies: Approaches to teaching disciplinary habits of mind (pp 99–120).
57. Kolodner, J. L., Crismond, D., Gray, J., Holbrook, J., & Puntambekar, S. (1998). Learning by design from theory to practice. In Proceedings of the Third International Conference of the Learning Sciences, Dec 16-19, 1998, Atlanta, GA (pp. 16–22).
58. Kolodner, J. L., Owensby, J. N., & Guzdial, M. (2004). Case-based learning aids. In Handbook of research for educational communications and technology (Vol. 2, pp. 829–861).
59. Krajcik, J. S., & Shin, N. (2014). Project-based learning. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 275–297). Cambridge University Press.
60. Kraut, R., & Resnick, P. (2012). Encouraging contribution to online communities. In R. Kraut & P. Resnick (Eds.), Building successful online communities: Evidence-based social design (pp. 21–76). Cambridge: MIT Press.
61. Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2015). Peer and self assessment in massive online classes. In H. Platter, C. Meinel, & Leifer (Eds.), Design thinking research (pp. 131–168). New York: Springer International Publishing.
62. Lande, M., & Leifer, L. (2010). Difficulties student engineers face designing the future. International Journal of Engineering Education, 26(2), 271–277.
63. Lawson, B., & Dorst, K. (2009). Design expertise. Oxford: Architectural Press.
64. Lundstrom, K., & Baker, W. (2009). To give is better than to receive: the benefits of peer review to the reviewer’s own writing. Journal of Second Language Writing, 18(1), 30–43.
65. Lynch, C. F., Ashley, K. D., Aleven, V. A., & Pinkwart. (2006). Defining “ill-defined domains”: A literature survey. In V. A. Aleven, K. D. Ashley, C. F. Lynch, & N. Pinkwart (Eds.), Proceedings of the workshop on intelligent tutoring systems for ill-defined domains at the 8th International Conference on Intelligent Tutoring systems (ITS 2006) (pp. 1–10). Johngli: National Central University.
66. McKenney, S., & Reeves, T. C. (2013). Conducting educational design research. New York: Routledge.
67. Miles, M. B., Huberman, A. M., & Saldaña, J. (2013). Qualitative data analysis: A methods sourcebook (3rd ed.). Los Angeles: Sage.
68. Moggridge, B., & Smith, G. C. (2007). Designing interactions (Vol. 17). Cambridge: MIT Press.
69. Moss, D., & Van Duzer, C. (1998). Project-based learning for adult english language learners. ERIC digest.
70. Oh, Y., Ishizaki, S., Gross, M. D., & Do, E. Y.-L. (2013). A theoretical framework of design critiquing in architecture studios. Design Studies, 34(3), 302–325.
71. Patchan, M. M., Charney, D., & Schunn, C. D. (2009). A validation study of students’ end comments: comparing comments by students, a writing instructor, and a content instructor. Journal of Writing Research, 1(2), 124–152.
72. Quinn, A. J., & Bederson, B. B. (2011, May). Human computation: a survey and taxonomy of a growing field. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1403–1412). ACM.
73. Rashid, A. M., Albert, I., Cosley, D., Lam, S. K. (2002). McNee systems. In Proceedings of the 7th International Conference on Intelligent User Interfaces (pp. 127–134).
74. Raymond, E. S. (2001). The cathedral & the bazaar: Musings on Linux and open source by an accidental revolutionary. Sebastopol: O’Reilly Media, Inc.
75. Rogers, E. M. (2003). Diffusion of innovations. New York: Free Press.
76. Sawyer, R. K. (2012). Learning how to create: Toward a learning sciences of art and design. In Proceedings of the International Conference of the Learning Sciences (Vol. 1, pp. 33–39). June 2-6, 2012, Sydney, Australia.
77. Schön, D. A. (1987). Educating the reflective practitioner: toward a new design for teaching and learning in the professions. San Francisco: Jossey-Bass.
78. Schrand, T., & Eliason, J. (2012). Feedback practices and signature pedagogies: what can the liberal arts learn from the design critique? Teaching in Higher Education, 17(1), 51–62.
79. Shulman, L. S. (2005). Signature pedagogies in the professions. Daedalus, 134(3), 52–59.
80. Simon, H. A. (1996). The sciences of the artificial (3rd ed.). Cambridge: MIT Press.
81. Smirnov, N., Easterday, M. W., & Gerber, E. M. (2016). Scaling studio-based learning through social innovation networks. In Proceedings of the International Conference of the Learning Sciences, 2016, Singapore.
82. Spradley, J. P. (1980). Participant observation. Belmont: Wadsworth.
83. Sue-Chan, C., & Latham, G. P. (2004). The relative effectiveness of external, peer, and self-coaches. Applied Psychology, 53(2), 260–278.
84. Surowiecki, J. (2004). The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. New York: Doubleday.
85. Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006). Getting the right design and the design right. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (pp. 1243–1252). New York: ACM.
86. Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249–276.
87. Van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, R. M. Branch, K. Gustafson, N. Nieveen, & T. Plomp (Eds.), Design approaches and tools in education and training (pp. 1–14). Netherlands: Springer.
88. van Gennip, N. A., Segers, M. S., & Tillema, H. H. (2009). Peer assessment for learning from a social perspective: the influence of interpersonal variables and structural features. Educational Research Review, 4(1), 41–54.
89. van Merriënboer, J. J. G., & Kirschner, P. A. (2012). Ten steps to complex learning: A systematic approach to four-component instructional design (2nd ed.). Mahwah: Lawrence Erlbaum Associates.
90. Van Zundert, M., Sluijsmans, D., & Van Merriënboer, J. (2010). Effective peer assessment processes: research findings and future directions. Learning and Instruction, 20(4), 270–279.
91. VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.
92. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. doi:10.1080/00461520.2011.611369.
93. Zurb. (n.d.). Forrst. [Web page] Retrieved from http://forrst.com/posts