Comparing of feedback-collection and think-aloud methods in program comprehension studies

Behaviour and Information Technology

Volumn 28, Issue 2, 2009, Pages 139-164

  Karahasanovic, Amela ^a,b   Hinkel, Unni Nyhamar ^b   Sjberg, Dag I K ^a,b   Thomas, Richard ^c  

^a SIMULA RESEARCH LABORATORY   (Norway)

^b UNIVERSITY OF OSLO   (Norway)

^c UNIVERSITY OF WESTERN AUSTRALIA   (Australia)

Author keywords

Data collection methods; Experience sampling; Program comprehension; Think aloud

Indexed keywords

ANEMOMETERS; DATA ACQUISITION; FUZZY RULES;

COGNITIVE PROCESS; COLLECTION METHODS; DATA COLLECTION METHODS; DATA COLLECTIONS; EXPERIENCE SAMPLING; EXPERIMENTAL COMPARISONS; MEDIUM EFFECTS; PROGRAM COMPREHENSION; SAMPLING METHODS; SOFTWARE APPLICATIONS; THINK-ALOUD; USER LOGS; WRITTEN FEEDBACKS;

FUZZY SYSTEMS;

EID: 61349118100     PISSN: 0144929X     EISSN: 13623001     Source Type: Journal    
DOI: 10.1080/01449290701682761     Document Type: Article

References (79)

1. Anderson, J.R., 1987. Methodologies for studying human knowledge. Behavioural & Brain Science, 10, 467-505.
2. Arisholm, E. and Sjøberg, D., 2004. Evaluating the effect of a delegated versus centralized control style on the maintainability of object-oriented software. IEEE Transactions on Software Engineering, 30 (8), 521-534.
3. Arisholm, E., Sjøberg, D.I.K., Carelius, G., and Lindsjøm, Y., 2002. A Web-based support environment for software engineering experiments. Nordic Journal of Computing, 9, 231-247.
4. Arisholm, E., Sjøberg, D.I.K., and Jørgensen, M., 2001. Assessing the changeability of two object-oriented design alternatives - a controlled experiment. Empirical Software Engineering, 6, 231-277.
5. Arunachalam, V. and Sasso, W., 1996. Cognitive processes in program comprehension: an empirical analysis in the context of software engineering. Journal of Systems & Software, 34, 177-189.
6. Aserinsky, E. and Kleitman, N., 1953. Regularly occuring periods of eye mobility and concomitant phenomena during sleep. Science, 118, 273-374.
7. Bernard, H., Killorth, P., and Sailer, L., 1982. Informant accuracy in social-network data. An experimental attempt to predict actual communication from recall data. Social Science Research, 11, 30-36.
8. Berry, D.C. and Broadbent, D.E., 1990. The role of instruction and verbalization in improving performance on complex search tasks. Behaviour & Information Technology, 9, 175-190.
9. Boren, M.T. and Ramey, J., 2000. Thinking-aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43, 261-278.
10. Borland, 2004. http://www.borland.com/jbuilder/, accessed 2004.
11. Bowers, V.A. and Snyder, H.L., 1990. Concurrent versus retrospective verbal protocol for comparing window usability. In: Human Factors Society - 34th annual meeting, Orlando, Florida, HFES.
12. Chen, H. and Nilan, M., 1998. An exploration of web users' internal experience: application of the experience sampling method to the web environment. In: WebNet 98 world conference, Orlando, Florida.
13. Chi, M.T., 1997. Quantifying qualitative analyses of verbal data: a practical guide. The Journal of the Learning Sciences, 6, 271-315.
14. Cohen, J., 1977. Statistical power analysis for the behavioural sciences. Orlando, FL: Academic Press.
15. Conrath, D.W., Higgins, C.A., and McClean, R.J., 1983. A comparison of questionnaire versus diary data. Social Networks, 5, 315-322.
16. Cortina, J.M. and Nouri, H., 2000. Effect size for ANOV A designers. Thousand Oaks, CA: SAGE Publications, Inc.
17. Czerwinski, M. and Horvitz, E., 2002. Memory for daily computing events. In: X. Faulkner, J. Finlay, and F. Detienne, eds. People and computers XVI, proceedings of HCI.
18. Czerwinski, M., Horvitz, E., and Wilhite, S., 2004. A dairy study of task switching and interruptions. In: Proceedings of the SIGHI conference on human factors in computing systems, Vienna, Austria. New York: ACM Press.
19. D'astous, P., Robillard, P.N., Detienne, F., and Visser, W., 2001. Quantitative measurements of the influence of participants roles during per review meetings. Empirical Software Engineering, 6, 143-159.
20. Davidson, G.V., Shorter, L., Crum, A., and Lane, J., 1996. Children's use of learning strategies and decision making in hypertext computer lesson. In: ED MEDIA 96, conference on educational multimedia and hypermedia. Assoc. Adv. Comp. Educ, Charlottesville, VA, USA.
21. Denning, S., Hoiem, D., Simpson, M., and Sullivan, K., 1990. The value of thinking-aloud protocols in industry: a case study of microsoft. In: Proceedings of the Human Factors Society - 34th annual meeting, The Human Factors Society, Santa Monica, CA.
22. Detienne, F. and Soloway, E., 1990. An empiricallyderived control structure for the process of program understanding. International Journal of Man-Machine Studies, 33, 323-342.
23. Du, H.S. and Wagner, C., 2005. Learning with Weblogs: an empirical investigation. In: Proceedings of the 38th Hawaii international conference on system sciences HICSS-38, IEEE, Big Island, Hawaii.
24. Eldridge, M., Lamming, M., and Flyn, M., 1992. Does a video diary help recall? In: Proceedings HCI 1992 conference, people and computers VII, York, UK.
25. Eldridge, M. and Newman, W., 1996. Agenda benders: modelling the disruptions caused by technology failures in the workplace. In: CHI 96, Proceedings of the conference on human factors in computing systems, Vancouver, British Columbia, Canada. New York: ACM Press.
26. Ericsson, K.A., 2003. Valid and non-reactive verbalisation of thoughts during performance of tasks. Journal of Conciousness Studies, 10, 1-18.
27. Ericsson, K.A., Krampe, R.T., and Tesch-Romer, C., 1993. The role of deliberate practice in acquisition of expert performance. Psychological Review, 100, 346-406.
28. Ericsson, K.A. and Simon, H.A., 1993. Protocol analysis: verbal reports as data. Cambridge, Massachusetts: A Bradford Book, MIT Press.
29. Ericsson, K.A. and Simon, H.A., 1998. How to study thinking in everyday life: contrasting think-aloud protocols with descriptions and explanations of thinking. Mind Culture & Activity, 5, 178-186.
30. Ericsson, K.A., Tesch-Römer, C., and Krampe, R.T., 1990. The role of practice and motivation in the acquisition of expert-level performance in real life: an empirical evaluation of a theoretical framework. In: M.J.A. Howe, ed. Encouraging the development of exceptional skills and talents. Leicester, UK: The British Psychological Society.
31. Eriksson, H.E. and Penker, M., 1998. Case study. UML toolkit. New York: Wiley.
32. Garner, R., 1988. Verbal-report data on cognitive and metacognitive strategies. In: C.E. Weinstein, E.T. Goetz, and P.A. Alexander, eds. Learning and study strategies: issues in assessment, instruction, and evaluation. San Diego, CA: Academic Press, Inc., 63-100.
33. Genest, M. and Turk, D.C, 1981. Think-aloud approaches to cognitive assessment. In: C.R. Merluzzi, C.R. Glass, and M. Genest, eds. Cognitive assessment. New York: Guilford Press.
34. George, S., 2002. Learning and the reflective journal in computer science. In: M. Oudshoorn, ed. Twenty-fifth Australasian Computer Science Conference (ACSC 2002), conferences in research and practice in information technology, Vol. 4, Melbourne, Australia.
35. Grinter, R.E. and Eldridge, M.A., 2001. y do tngrs luv 2 txt msg? In: W. Prinz, M. Jarke, Y. Rogers, K. Schmidt, and V. Wulf, eds. Proceedings of the seventh European conference on Computer-Supported Cooperative Work ECSCW' 01. Bonn, Germany, Dordrecht, Netherlands: Kluwer Academic Publishers.
36. Grissom, R.J. and Kim, J.J., 2005. Effect sizes for researchers: a broad practical approach. Mahwah, NJ: Lawrence Erlbaum Associates.
37. Haak, M.J. and Jong, D.T.M., 2003. Exploring two methods of usability testing: concurrent versus retrospective thinkaloud protocols. In: Proceedings of the International Professional Communication Conference, IPCC 2003. Orlando, Florida: IEEE Computer Society, 285-287.
38. Hale, J.E., Sharpe, S., and Hale, D.P., 1999. An evaluation of the cognitive processes of programmers engaged in software debugging. Journal of Software Maintenance-Research & Practice, 11, 73-91.
39. Hahn, J. and Kim, J., 1999. Why are some diagrams easier to work with? Effects of diagrammatic representation on the cognitive integration process of systems analysis and design. ACM Transactions on Computer-Human Interaction, 6, 181-213.
40. Hedges, L.V. and Olkin, I., 1985. Statistical methods for meta-analysis. San Diego, CA: Academic Press.
41. Hughes, J. and Parkes, S., 2003. Trends in the use of verbal protocol analysis in software engineering research. Behaviour & Information Technology, 22, 127-140.
42. Hurlburt, R.T., 1979. Random sampling of cognitions and behaviour. Journal of Research in Personality, 13, 103-111.
43. Jungk, A., Thull, B., Hoeft, A., and Rau, G., 2000. Evaluation of two new ecological interface approaches for the anesthesia workplace. Journal of Clinical Monitoring & Computing, 16, 243-258.
44. Karahasanovic, A., Anda, B., Arisholm, E., Hove, S.E., Jørgensen, M., Sjøberg, D.I.K., and Welland, R., 2005. Collecting feedback during software engineering experiments. Empirical Software Engineering, 10, 113-147.
45. Karahasanovic, A., Levine, A.K., and Thomas, R., 2007. Comprehension strategies and difficulties in maintaining object-oriented systems: an explorative study. Journal of Systems & Software, 80, 1541-1559.
46. Kersten, M. and Murphy, G.C., 2005. Mylar: A degree-ofinterest model for IDEs. ACM AOSD 05. Chicago, Illinois: ACM Digital Library.
47. Kim, J. and Hahn, J., 1997. Reasoning with multiple diagrams: focusing on the cognitive integration process. In: Proceedings of the 19th annual conference of the Cognitive Society, Mahway, NJ, Lawrence Erlbaum Associates.
48. Kuusela, H. and Paul, P., 2000. A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology, 113, 387-404.
49. Larson, R. and Csikszentmihalyi, M., 1983. The experience sampling method. In: T.R. Harry, ed. Naturalistic approaches to studying social interaction. San Francisco: Jossey-Bass.
50. Letovsky, S., 1986. Cognitive processes in program comprehension. In: E. Soloway and S. Iyengar, eds. Empirical studies of programmers. Norwood, NJ: Ablex.
51. Littman, D.C., Pinto, J., Letovski, S., and Soloway, E., 1986. Mental models and software maintenance. In: E. Soloway and S. Iyengar, eds. Empirical studies of programmers. Norwood, NJ: Ablex.
52. Mark, G., Christiansen, U., and Shafae, M., 2001. A methodology using a microcamera for studying mobile IT usage and person mobility. Presented at the CHI workshop on mobile communications: understanding users, adoption & design, ACM CHI 01 conf. on human factors in computing systems, Seattle, Washington.
53. Mastaglio, T. and Rieman, J., 1991. How experts infer novice programmer expertise: a protocol analysis of LISP code evaluation. In: J. Koenemann-Bellivea, T.G. Moher, and S.P. Robertson, eds. Empirical studies of programmers: fourth workshop. Norwood, NJ: Ablex.
54. Nathan, M.J., 1991. A simple learning environment improves mathematical reasoning. Intelligent Tutoring Media, 2, 101-111.
55. Newell, A. and Simon, H.A., 1972. Human problem solving. Englewood Cliffs, NJ: Prentice Hall.
56. Nielsen, J., 1993. Usability engineering. Amsterdam: Morgan Kaufmann Academic Press.
57. Ohnemus, K.R. and Biers, D.W., 1993. Retrospective versus thinking-aloud in usability testing. In: Human Factors and Ergonomic Society 37th annual meeting, HFES, Seattle, Washington.
58. Palen, L. and Salzman, M., 2002. Voice-mail diaries for naturalistic data capture under mobile conditions. ACM CSCW '02, 87-95.
59. Patel, V.L., Arocha, J.F., Diermeier, M., How, J., and Mottur-Pilson, C., 2001. Cognitive psychological studies of representation and use of clinical practice guidelines. International Journal of Medical Informatics, 63, 147-167.
60. Pennington, N., 1987. Comprehension strategies in programming. In: Empirical studies of programmers, second workshop. Norwood, NJ, USA: Ablex Publishing Corporation.
61. Pennington, N., Nicolich, R., and Rahm, J., 1995. Transfer of training between cognitive subskills: is knowledge use specific? Cognitive Psychology, 28, 175-224.
62. Rieman, J., 1993. The diary study: a workplace-oriented research tool to guide laboratory efforts collecting userinformation for system design. In: ACM INTERCHI'93 conference on human factors in computing systems, Amsterdam.
63. Robillard, P.N., D'astous, P., Detienne, F., and Visser, W., 1998. Measuring cognitive activities in software engineering. In: Proceedings of the 20th international conference on software engineering (ICSE '98). Kyoto, Japan: IEEE CS Press.
64. Russo, J.E., Johnson, E.J., and Stephens, D.L., 1989. The validity of verbal protocols. Memory and Cognition, 17, 759-769.
65. Sanderson, P.M., 1990. Verbal protocol analysis in three experimental domains using SHAPA. In: Proceedings of the Human Factors Society, 34th annual meeting, The Human Factors Society, Santa Monica, CA.
66. Schenk, K.D., Vitalari, N.P., and Davis, K.S., 1998. Differences between novice and expert system analysts: what do we know and what we do? Journal of Management Information Systems, 15, 9-50.
67. Shadish, W.R., Robinson, L., and Lu, C, 1999. ES: a computer program for effect size calculation. St. Puau, MN: Assessment Systems Corporation.
68. Shaft, T.M., 1997. Responses to comprehension questions and verbal protocols as measures of computer program comprehension processes. Behaviour & Information Technology, 16, 320-336.
69. Shaft, T.M. and Vessey, I., 1996. Computer program comprehension processes: the effect of application domain knowledge. In: W. Gray and D. Boehm-Davis, eds. Empirical studies of programmers: sixth workshop. Nordwood, NJ: Ablex Publishing.
70. Sjøberg, D., Anda, B., Arisholm, E., Dybå, T., Jørgensen, M., Karahasanovic, A., and Vokác, M., 2003. Challenges and recommendations when increasing the realism of controlled software engineering experiments. In: R. Conradi and A.I. Wang, eds. Empirical methods and studies in software engineering, ESERNET, 2001-2002, LNCS 2765. Berlin, Heidelberg: Springer-Verlag.
71. Slattery, S., 2003. Research methods for revealing patterns of mediation. SIGDOC '03. San Francisco, California. New York: ACM Press.
72. Taylor, K.L. and Dionne, J.-P., 2000. Accessing problemsolving knowledge: the complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology, 92, 413-425.
73. Thomas, R., Kennedy, G., Draper, S., Mancy, R., Crease, M., Evans, H., and Gray, P., 2003. Generic usage monitoring of programming students. In: ASCILITE 2003 conference, University of Adelaide, Australia.
74. Van Someren, M.W., Barnard, Y.F., and Sandberg, J.A.C., 1994. The thinking aloud method: a practical guide to model cognitive processes. London: Academic Press.
75. Vans, A.M., Von Mayerhanser, A., and Somlo, G., 1999. Program understanding behaviour during corrective maintenance of large-scale software. International Journal of Human-Computer Studies, 51, 31-70.
76. Von Mayrhauser, A. and Lang, S., 1999. A coding scheme to support systematic analysis of software comprehension. IEEE Transactions on Software Engineering, 25, 526-540.
77. Vygotsky, L.S., 1994. Thought and language. Cambridge, MA: MIT Press.
78. Waden, T.A., Berkowitz, R.I., Womble, L.G., Sarwer, D.B., Phelan, S., Cato, R.K., Hesson, L.A., Osei, S.Y., Kaplan, R., and Stunkard, A.J., 2005. Randomized trial of lifestyle modification and pharmacotherapy for obesity. New England Journal of Medicine, 353, 2111-2120.
79. Wright, R.B. and Converse, S.A., 1992. Method bias and concurrent verbal protocol in software usability testing. In: Proceeding of Human Factors Society 36th Annual Meeting, Atlanta, Georgía.