Data Collection and Analysis

Handbook of Research on Educational Communications and Technology, Third Edition

Volumn , Issue , 2008, Pages 763-806

  van Gog, Tamara ^a   Paas, Fred ^a   Savenye, Wilhelmina ^b   Robinson, Rhonda ^c   Niemczyk, Mary ^d   Atkinson, Robert ^b   Johnson, Tristan E ^e   O’Connor, Debra L ^f   Rikers, Remy M J P ^g   Ayres, Paul ^h   Duley, Aaron R ⁱ   Ward, Paul ^e   Hancock, Peter A ^j  

^a OPEN UNIVERSITY OF THE NETHERLANDS   (Netherlands)

^b ARIZONA STATE UNIVERSITY   (United States)

^c NORTHERN ILLINOIS UNIVERSITY   (United States)

^d ARIZONA STATE UNIVERSITY   (United States)

^e FLORIDA STATE UNIVERSITY   (United States)

^f Intelligent Decision Systems Inc ^*  (United States)

^g ERASMUS UNIVERSITY ROTTERDAM   (Netherlands)

^h UNIVERSITY OF NEW SOUTH WALES   (Australia)

ⁱ NASA AMES RESEARCH CENTER   (United States)

^j UNIVERSITY OF CENTRAL FLORIDA   (United States)

more..

Author keywords

Assessment criteria: Describe the aspects of performance that will be assessed.; Assessment of learning: Measuring learning achievement, performance, outcomes, and processes by many means.; Assessment standards: Describe the quality of performance on each of the criteria that can be expected of participants at different stages (e.g., age, grade) based on a participant s past performance (selfreferenced), peer group performance (norm referenced), or an objective standard (criterion referenced).; Collective data collection: Obtaining data from individual group members; data are later aggregated or manipulated into a representation of the group as a whole.; Complex performance: Refers to real world activities that require the integration of disparate measurement instrumentation as well as the need for timecritical experimental control.; Direct process measure: Continuous elicitation of data from beginning to end of the (group) process; direct process measures involve videotaping, audiotaping, direct researcher observation, or a combination of these methods.; Group learning process: Actions and interactions performed by group members during the group learning task.; Group: Two or more individuals working together to achieve a common goal.; Holistic data collection: Obtaining data from the group as a whole; as this type of data collection results in a representation of the group rather than individual group member, it is not necessary to aggregate or manipulate data.; Indirect process measure: Discrete measure at a specific point in time during the (group) process; often involves multiple points of data collection; indirect process measures may measure processes, outcomes, products, or other factors related to group process.; Instrumentation: Hardware devices used to assist with the process of data acquisition and measurement.; Mixed methods research: Studies that rely on quantitative and qualitative as well as other methods for formulating research questions, collecting and analyzing data, and interpreting findings.; Online offline measures: Online measures are recorded during task performance, offline measures are recorded after task performance.; Process tracing techniques: Records performance process data such as verbal reports, eye movements, and actions that can be used to make inferences about the cognitive processes or knowledge underlying task performance.; Qualitative research: Sometimes called naturalistic; research on human systems whose hallmarks include researcher as instrument, natural settings, and little manipulation.; Quantitative research: Often conceived of as more traditional or positivistic; typified by experimental or correlational studies. Data and findings are usually represented through numbers and results of statistical tests.; Task complexity: Can be defined subjectively (individual characteristics, such as expertise or perception), objectively (task characteristics, such as multiple solution paths or goals), or as an interaction (individual and task characteristics).

Indexed keywords

EID: 85144933181     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203880869-61     Document Type: Chapter

References (216)

1. Adler, P. A. and Adler, P. (1994). Observational techniques. In Handbook of Qualitative Research, edited by N. K. Denzin and Y. S. Lincoln, pp. 377-392. Thousand Oaks, CA: Sage.*
2. Airasian, P. W. (1996). Assessment in the Classroom. New York: McGraw-Hill.
3. Alavi, M. (1994). Computer-mediated collaborative learning: an empirical evaluation. MIS Q., 18, 159-174.
4. American Evaluation Association. (2007). Qualitative software, www.eval.org/Resources/QDA.htm.
5. Ancona, D. G. and Caldwell, D. F. (1991). Demography and Design: Predictors of New Product Team Performance, No. 3236-91. Cambridge, MA: MIT Press.
6. Anderson, J. R. (1993). Rules of the Mind. Hillsdale, NJ: Lawrence Erlbaum Associates.
7. Anderson, J. R. and Lebiere, C. (1998). The Atomic Components of Thought. Mahwah, NJ: Lawrence Erlbaum Associates.
8. Anderson, J. R., Reder, L. M., and Simon, H. A. (1996). Situated learning and education. Educ. Res., 25(4), 5-11.
9. Arter, J. A. and Spandel, V. (1992). An NCME instructional module on: using portfolios of student work in instruction and assessment. Educ. Meas. Issues Pract., 11, 36-45.
10. Aviv, R. (2003). Network analysis of knowledge construction in asynchronous learning networks. J. Asynch. Learn. Netw., 7(3), 1-23.
11. Ayres, P. (2006). Using subjective measures to detect variations of intrinsic cognitive load within problems. Learn. Instruct., 16, 389-400.*
12. Bales, R. F. (1950). Interaction Process Analysis: A Method for the Study of Small Groups. Cambridge, MA: Addison-Wesley.
13. Battistich, V., Solomon, D., and Delucchi, K. (1993). Interaction processes and student outcomes in cooperative learning groups. Element. School J., 94(1), 19-32.
14. Bazeley, P. and Richards, L. (2000). The NVivo Qualitative Project Book. London: SAGE.
15. Bellman, B. L. and Jules-Rosette, B. (1977). A Paradigm for Looking: Cross-Cultural Research with Visual Media. Norwood, NJ: Ablex Publishing.
16. Birenbaum, M. and Dochy, F. (1996). Alternatives in assessment of achievements, learning processes and prior knowledge. Boston, MA: Kluwer.
17. Bjork, R. A. (1999). Assessing our own competence: heuristics and illusions. In Attention and Performance. Vol. XVII. Cognitive Regulation of Performance: Interaction of Theory and Application, edited by D. Gopher and A. Koriat, pp. 435-459. Cambridge, MA: MIT Press.
18. Bogaart, N. C. R. and Ketelaar, H. W. E. R., Eds. (1983). Methodology in Anthropological Filmmaking: Papers of the IUAES Intercongress, Amsterdam, 1981. Gottingen, Germany: Edition Herodot.
19. Bogdan, R. C. and Biklen, S. K. (1992). Qualitative Research for Education: An Introduction to Theory and Methods, 2nd ed. Boston, MA: Allyn and Bacon.*
20. Boren, M. T. and Ramey, J. (2000). Thinking aloud: reconciling theory and practice. IEEE Trans. Prof. Commun., 43, 261-278.
21. Borg, W. R. and Gall, M. D. (1989). Educational Research: An Introduction, 5th ed. New York: Longman.
22. Bowers, C. A. (2006). Analyzing communication sequences for team training needs assessment. Hum. Factors, 40, 672-678.*
23. Bowers, C. A., Jentsch, F., Salas, E., and Braun, C. C. (1998). Analyzing communication sequences for team training needs assessment. Hum. Factors, 40, 672-678.*
24. Bridgeman, B., Cline, F., and Hessinger, J. (2004). Effect of extra time on verbal and quantitative GRE scores. Appl. Meas. Educ., 17(1), 25-37.
25. Brünken, R., Plass, J. L., and Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educ. Psychol., 38, 53-61.
26. Byström, K. and Järvelin, K. (1995). Task complexity affects information seeking and use. Inform. Process. Manage., 31, 191-213.
27. Campbell, D. J. (1988). Task complexity: a review and analysis. Acad. Manage. Rev., 13, 40-52.*
28. Camps, J. (2003). Concurrent and retrospective verbal reports as tools to better understand the role of attention in second language tasks. Int. J. Appl. Linguist., 13, 201-221.
29. Carnevale, A., Gainer, L., and Meltzer, A. (1989). Workplace Basics: The Skills Employers Want. Alexandria, VA: American Society for Training and Development.
30. Cascallar, A. and Cascallar, E. (2003). Setting standards in the assessment of complex performances: the optimised extended-response standard setting method. In Optimising New Modes of Assessment: In Search of Qualities and Standards, edited by M. Segers, F. Dochy, and E. Cascallar, pp. 247-266. Dordrecht: Kluwer.
31. Chandler, P. and Sweller, J. (1991). Cognitive load theory and the format of instruction. Cogn. Instruct., 8, 293-332.*
32. Charness, N., Reingold, E. M., Pomplun, M., and Stampe, D. M. (2001). The perceptual aspect of skilled performance in chess: evidence from eye movements. Mem. Cogn., 29, 1146-1152.*
33. Chase, C. I. (1999). Contemporary Assessment for Educators. New York: Longman.
34. Chen, H. (2005). The Effect of Type of Threading and Level of Self-Efficacy on Achievement and Attitudes in Online Course Discussion, Ph.D. dissertation. Tempe: Arizona State University.
35. Christensen, L. B. (2006). Experimental Methodology, 10th ed. Boston, MA: Allyn and Bacon.
36. Collier, J. and Collier, M. (1986). Visual Anthropology: Photography as a Research Method. Albuquerque, NM: University of New Mexico Press.
37. Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. Int. J. Hum.-Comput. Stud., 41, 801-849.
38. Cooke, N. J., Salas E., Cannon-Bowers, J. A., and Stout R. J. (2000). Measuring team knowledge. Hum. Factors, 42, 151-173.*
39. Cornu, B. (2004). Information and communication technology transforming the teaching profession. In Instructional Design: Addressing the Challenges of Learning Through Technology and Curriculum, edited by N. Seel and S. Dijkstra, pp. 227-238. Mahwah, NJ: Lawrence Erlbaum Associates.*
40. Crooks, S. M., Klein, J. D., Jones, E. K., and Dwyer, H. (1995). Effects of Cooperative Learning and Learner Control Modes in Computer-Based Instruction. Paper presented at the Association for Communications and Technology Annual Meeting, February 8-12, Anaheim, CA.
41. Cuneo, C. (2000). WWW Virtual Library: Sociology Software, http://socserv.mcmaster.ca/w3virtsoclib/software.htm
42. Demetriadis, S., Barbas, A., Psillos, D., and Pombortsis, A. (2005). Introducing ICT in the learning context of traditional school. In Preparing Teachers to Teach with Technology, edited by C. Vrasidas and G. V. Glass, pp. 99-116. Greenwich, CO: Information Age Publishers.
43. Dewey, J. (1916/1966). Democracy and Education: An Introduction to the Philosophy of Education. New York: Free Press.
44. Dijkstra, S. (2004). The integration of curriculum design, instructional design, and media choice. In Instructional Design: Addressing the Challenges of Learning Through Technology and Curriculum, edited by N. Seel and S. Dijkstra, pp. 145-170. Mahwah, NJ: Lawrence Erlbaum Associates.*
45. Downing, S. M. and Haladyna, T. M. (1997). Test item development: validity evidence from quality assurance procedures. Appl. Meas. Educ., 10(1), 61-82.
46. Driscoll, M. P. (1995). Paradigms for research in instructional systems. In Instructional Technology: Past, Present and Future, 2nd ed., edited by G. J. Anglin, pp. 322-329. Englewood, CO: Libraries Unlimited.*
47. Duchowski, A. T. (2003). Eye Tracking Methodology: Theory and Practice. London: Springer.
48. Eccles, D. W. and Tenenbaum, G. (2004). Why an expert team is more than a team of experts: a social-cognitive conceptualization of team coordination and communication in sport. J. Sport Exer. Psychol., 26, 542-560.
49. Ericsson, K. A. (2002). Attaining excellence through deliberate practice: insights from the study of expert performance. In The Pursuit of Excellence Through Education, edited by M. Ferrari, pp. 21-55. Hillsdale, NJ: Lawrence Erlbaum Associates.
50. Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad. Med., 79(10), 70-81.*
51. Ericsson, K. A. and Lehmann, A. C. (1996). Expert and exceptional performance: evidence for maximal adaptation to task constraints. Annu. Rev. Psychol., 47, 273-305.
52. Ericsson, K. A. and Simon, H. A. (1980). Verbal reports as data. Psychol. Rev., 87, 215-251.*
53. Ericsson, K. A. and Simon, H. A. (1984). Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT Press.*
54. Ericsson, K. A. and Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data, rev. ed. Cambridge, MA: MIT Press.
55. Ericsson, K. A. and Smith, J., Eds. (1991). Toward a General Theory of Expertise: Prospects and Limits. Cambridge, U.K.: Cambridge University Press.
56. Ericsson, K. A. and Staszewski, J. J. (1989). Skilled memory and expertise: mechanisms of exceptional performance. In Complex Information Processing: The Impact of Herbert A. Simon, edited by D. Klahr and K. Kotovsky, pp. 235-267. Hillsdale, NJ: Lawrence Erlbaum Associates.
57. Erkens, G. (2002). MEPA: Multiple Episode Protocol Analysis, Version 4.8, http://edugate.fss.uu.nl/mepa/index.htm.
58. Espey, L. (2000). Technology planning and technology integration: a case study. In Proceedings of Society for Information Technology and Teacher Education International Conference 2000, edited by C. Crawford et al., pp. 95-100. Chesapeake, VA: Association for the Advancement of Computing in Education.
59. Florer, F. (2007). Software for Psychophysics, http://vision.nyu.edu/Tips/FaithsSoftwareReview.html.
60. Fu, W.-T. (2001). ACT-PRO action protocol analyzer: a tool for analyzing discrete action protocols. Behav. Res. Methods Instrum. Comput., 33, 149-158.
61. Fussell, S. R., Kraut, R. E., Lerch, F. J., Shcerlis, W. L., McNally, M. M., and Cadiz, J. J. (1998). Coordination, Overload and Team Performance: Effects of Team Communication Strategies. Paper presented at the Association for Computing Machinery Conference on Computer Supported Cooperative Work, November 14-18, Seattle, WA.
62. Gamma, E., Helm, R., Johnson, R., and Vlissides, J. (2005). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley: Reading, MA.
63. Garfinkel, H. (1967). Studies in Ethnomethodology: A Return to the Origins of Ethnomethodology. Englewood Cliffs, NJ: Prentice Hall.
64. Gerjets, P., Scheiter, K., and Catrambone, R. (2004). Designing instructional examples to reduce cognitive load: molar versus modular presentation of solution procedures. Instruct. Sci., 32, 33-58.*
65. Gerjets, P., Scheiter, K., and Catrambone, R. (2006). Can learning from molar and modular worked examples be enhanced by providing instructional explanations and prompting selfexplanations? Learn. Instruct., 16, 104-121.
66. Goetz, J. P. and LeCompte, M. D. (1984). Ethnography and Qualitative Design in Educational Research. Orlando, FL: Academic Press.*
67. Goodyear, P. (2000). Environments for lifelong learning: ergonomics, architecture and educational design. In Integrated and Holistic Perspectives on Learning, Instruction, and Technology: Understanding Complexity, edited by J. M. Spector and T. M. Anderson, pp. 1-18. Dordrecht: Kluwer.*
68. Goswami, U. (2004). Neuroscience and education. Br. J. Educ. Psychol., 74, 1-14.
69. Gulikers, J. T. M., Bastiaens, T. J., and Kirschner, P. A. (2004). A five-dimensional framework for authentic assessment. Educ. Technol. Res. Dev., 52(3), 67-86.*
70. Guzzo, R. A. and Shea, G. P. (1992). Group performance and intergroup relations in organizations. In Handbook of Industrial and Organizational Psychology Vol. 3, 2nd ed., edited by M. D. Dunnette and L. M. Hough, pp. 269-313. Palo Alto, CA: Consulting Psychologists Press.
71. Haider, H. and Frensch, P. A. (1999). Eye movement during skill acquisition: more evidence for the information reduction hypothesis. J. Exp. Psychol. Learn. Mem. Cogn., 25, 172-190.
72. Hambleton, R. K., Jaegar, R. M., Plake, B. S., and Mills, C. (2000). Setting performance standards on complex educational assessments. Appl. Psychol. Meas., 24, 355-366.
73. Hara, N., Bonk, C. J., and Angeli, C. (2000). Content analysis of online discussion in an applied educational psychology course. Instruct. Sci., 28, 115-152.
74. Hardwidge, B. (2006) Building Extreme PCs: The Complete Guide to Computer Modding. Cambridge, MA: O’Reilly Media.
75. Heider, K. G. (1976). Ethnographic Film. Austin, TX: The University of Texas Press.
76. Herl, H. E., O’Neil, H. F., Chung, G. K. W. K., and Schacter, J. (1999). Reliability and validity of a computer-based knowledge mapping system to measure content understanding. Comput. Hum. Behav., 15, 315-333.
77. Higgins, N. and Rice, E. (1991). Teachers’ perspectives on competency-based testing. Educ. Technol. Res. Dev., 39(3), 59-69.
78. Hobma, S. O., Ram, P. M., Muijtjens, A. M. M., Grol, R. P. T. M., and Van der Vleuten, C. P. M. (2004). Setting a standard for performance assessment of doctor-patient communication in general practice. Med. Educ., 38, 1244-1252.
79. Hockings, P., Ed. (1975). Principles of Visual Anthropology. The Hague: Mouton Publishers.
80. Horber, E. (2006). Qualitative Data Analysis Links, http://www.unige.ch/ses/sococ/qual/qual.html.
81. Ifenthaler, D. (2005). The measurement of change: learningdependent progression of mental models. Technol. Instruct. Cogn. Learn., 2, 317-336.*
82. Jeong, A. C. (2003). The sequential analysis of group interaction and critical thinking in online threaded discussions. Am. J. Distance Educ., 17(1), 25-43.*
83. Johnson, D. W., Johnson, R. T., and Stanne, M. B. (2000). Cooperative Learning Methods: A Meta-Analysis, http://www.co-operation.org/pages/cl-methods.html.*
84. Jones, E. K., Crooks, S., and Klein, J. (1995). Development of a Cooperative Learning Observational Instrument. Paper presented at the Association for Educational Communications and Technology Annual Meeting, February 8-12, Anaheim, CA.
85. Jorgensen, D. L. (1989). Participant Observation: A Methodology for Human Studies. London: SAGE.*
86. Katzir, T. and Paré-Blagoev, J. (2006). Applying cognitive neuroscience research to education: the case of literacy. Educ. Psychol., 41, 53-74.
87. Kirschner, P., Carr, C., van Merrienboer, J., and Sloep, P. (2002). How expert designers design. Perform. Improv. Q., 15(4), 86-104.
88. Klein, J. D. and Pridemore, D. R. (1994). Effects of orienting activities and practice on achievement, continuing motivation, and student behaviors in a cooperative learning environment. Educ. Technol. Res. Dev., 41(4), 41-54.*
89. Klimoski, R. and Mohammed, S. (1994). Team mental model: construct or metaphor. J. Manage., 20, 403-437.
90. Ko, S. and Rossen, S. (2001). Teaching Online: A Practical Guide. Boston, MA: Houghton Mifflin.
91. Koschmann, T. (1996). Paradigm shifts and instructional technology. In Computer Supportive Collaborative Learning: Theory and Practice of an Emerging Paradigm, edited by T. Koschmann, pp. 1-23. Mahwah, NJ: Lawrence Erlbaum Associates.*
92. Kuusela, H. and Paul, P. (2000). A comparison of concurrent and retrospective verbal protocol analysis. Am. J. Psychol., 113, 387-404.
93. Langan-Fox, J. (2000). Team mental models: techniques, methods, and analytic approaches. Hum. Factors, 42, 242-271.*
94. Langan-Fox, J. and Tan, P. (1997). Images of a culture in transition: personal constructs of organizational stability and change. J. Occup. Org. Psychol., 70, 273-293.
95. Langan-Fox, J., Code, S., and Langfield-Smith, K. (2000). Team mental models: techniques, methods, and analytic approaches. Hum. Factors, 42, 242-271.
96. Langan-Fox, J., Anglim, J., and Wilson, J. R. (2004). Mental models, team mental models, and performance: process, development, and future directions. Hum. Factors Ergon. Manuf., 14, 331-352.
97. Lawless, C. J. (1994). Investigating the cognitive structure of students studying quantum theory in an Open University history of science course: a pilot study. Br. J. Educ. Technol., 25, 198-216.
98. Lesh, R. and Dorr, H. (2003). A Models and Modeling Perspective on Mathematics Problem Solving, Learning, and Teaching. Mahwah, NJ: Lawrence Erlbaum Associates.*
99. Levine, J. M. and Moreland, R. L. (1990). Progress in small group research. Annu. Rev. Psychol., 41, 585-634.
100. Lincoln, Y. S. and Guba, E. G. (1985). Naturalistic Inquiry. Beverly Hills, CA: SAGE.
101. Lingard, L. (2002). Team communications in the operating room: talk patterns, sites of tension, and implications for novices. Acad. Med., 77, 232-237.
102. Losada, M. (1990). Collaborative Technology and Group Process Feedback: Their Impact on Interactive Sequences in Meetings. Paper presented at the Association for Computing Machinery Conference on Computer Supported Cooperative Work, October 7-10, Los Angeles, CA.
103. Lowyck, J. and Elen, J. (2004). Linking ICT, knowledge domains, and learning support for the design of learning environments. In Instructional Design: Addressing the Challenges of Learning Through Technology and Curriculum, edited by N. Seel, and S. Dijkstra, pp. 239-256. Mahwah, NJ: Lawrence Erlbaum Associates.*
104. Magliano, J. P., Trabasso, T., and Graesser, A. C. (1999). Strategic processing during comprehension. J. Educ. Psychol., 91, 615-629.
105. Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., and Cannon-Bowers, J. A. (2000). The influence of shared mental models on team process and performance. J. Appl. Psychol., 85, 273-283.
106. Mehrens, W.A., Popham, J. W., and Ryan, J. M. (1998). How to prepare students for performance assessments. Educ. Measure. Issues Pract., 17(1), 18-22.
107. Meloy, J. M. (1994). Writing the Qualitative Dissertation: Understanding by Doing. Hillsdale, NJ: Lawrence Erlbaum Associates.
108. Merrill, M. D. (2002). First principles of instruction. Educ. Technol. Res. Dev., 50(3), 43-55.*
109. Michaelsen, L. K., Knight, A. B., and Fink, L. D. (2004). Team-Based Learning: A Transformative Use of Small Groups in College Teaching. Sterling, VA: Stylus Publishing.
110. Miles, M. B. and Huberman, A. M. (1994). Qualitative Data Analysis: An Expanded Sourcebook, 2nd ed. Thousand Oaks, CA: SAGE.
111. Miles, M. B. and Weitzman, E. A. (1994). Appendix: choosing computer programs for qualitative data analysis. In Qualitative Data Analysis: An Expanded Sourcebook, 2nd ed., edited by M. B. Miles and A. M. Huberman, pp. 311-317. Thousand Oaks, CA: SAGE.
112. Moallem, M. (1994). An Experienced Teacher’s Model of Thinking and Teaching: An Ethnographic Study on Teacher Cognition. Paper presented at the Association for Educational Communications and Technology Annual Meeting, February 16-20, Nashville, TN.
113. Morgan, D. L. (1996). Focus Groups as Qualitative Research Methods, 2nd ed. Thousand Oaks, CA: SAGE.
114. Morris, L. L., Fitz-Gibbon, C. T., and Lindheim, E. (1987). How to Measure Performance and Use Tests. Newbury Park, CA: SAGE.
115. Myllyaho, M., Salo, O., Kääriäinen, J., Hyysalo, J., and Koskela, J. (2004). A Review of Small and Large Post-Mortem Analysis Methods. Paper presented at the 17th International Conference on Software and Systems Engineering and their Applications, November 30-December 2, Paris, France.
116. Newell, A. and Rosenbloom, P. (1981). Mechanisms of skill acquisition and the law of practice. In Cognitive Skills and Their Acquisition, edited by J. R. Anderson, pp. 1-56. Hillsdale, NJ: Lawrence Erlbaum Associates.
117. Nitko, A. (2001). Educational Assessment of Students, 3rd ed. Upper Saddle River, NJ: Prentice Hall.
118. Noldus, L. P. J. J., Trienes, R. J. H., Hendriksen, A. H. M., Jansen, H., and Jansen, R. G. (2000). The Observer Video-Pro: new software for the collection, management, and presentation of time-structured data from videotapes and digital media files. Behav. Res. Methods Instrum. Comput., 32, 197-206.
119. O’Connor, D. L. and Johnson, T. E. (2004). Measuring team cognition: concept mapping elicitation as a means of constructing team shared mental models in an applied setting. In Concept Maps: Theory, Methodology, Technology, Proceedings of the First International Conference on Concept Mapping Vol. 1, edited by A. J. Cañas, J. D. Novak, and F. M. Gonzalez, pp. 487-493. Pamplona, Spain: Public University of Navarra.*
120. Olansen, J. B. and Rosow, E. (2002). Virtual Bio-Instrumentation. Upper Saddle River, NJ: Prentice Hall.
121. Olkinuora, E., Mikkila-Erdmann, M., and Nurmi, S. (2004). Evaluating the pedagogical value of multimedia learning material: an experimental study in primary school. In Instructional Design: Addressing the Challenges of Learning Through Technology and Curriculum, edited by N. Seel and S. Dijkstra, pp. 331-352. Mahwah, NJ: Lawrence Erlbaum Associates.
122. O’Neal, M. R. and Chissom, B. S. (1993). A Comparison of Three Methods for Assessing Attitudes. Paper presented at the Annual Meeting of the Mid-South Educational Research Association, November 10-12, New Orleans, LA.
123. O’Neil, H. F., Wang, S., Chung, G., and Herl, H. E. (2000). Assessment of teamwork skills using computer-based teamwork simulations. In Aircrew Training and Assessment, edited by H. F. O’Neil and D. H. Andrews, pp. 244-276. Mahwah, NJ: Lawrence Erlbaum Associates.*
124. Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive load approach. J. Educ. Psychol., 84, 429-434.
125. Paas, F. and van Merriënboer, J. J. G. (1993). The efficiency of instructional conditions: an approach to combine mental-effort and performance measures. Hum. Factors, 35, 737-743.*
126. Paas, F. and van Merriënboer, J. J. G. (1994a). Instructional control of cognitive load in the training of complex cognitive tasks. Educ. Psychol. Rev., 6, 51-71.
127. Paas, F. and van Merriënboer, J. J. G. (1994b). Variability of worked examples and transfer of geometrical problem-solving skills: a cognitive load approach. J. Educ. Psychol., 86, 122-133.
128. Paas, F., Tuovinen, J. E., Tabbers, H., and Van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol., 38, 63-71.
129. Paterson, B., Bottorff, J., and Hewatt, R. (2003). Blending observational methods: possibilities, strategies, and challenges. Int. J. Qual. Methods, 2(1), article 3.
130. Patton, M. Q. (2001). Qualitative Research and Evaluation Methods, 3rd ed. Thousand Oaks, CA: SAGE.
131. Paulsen, M. F. (2003). An overview of CMC and the online classroom in distance education. In Computer-Mediated Communication and the Online Classroom, edited by Z. L. Berge and M. P. Collins, pp. 31-57. Cresskill, NJ: Hampton Press.
132. Pavitt, C. (1998). Small Group Discussion: A Theoretical Approach, 3rd ed. Newark: University of Delaware (http://www.udel.edu/communication/COMM356/pavitt/).
133. Pelto, P. J. and Pelto, G. H. (1978). Anthropological Research: The Structure of Inquiry, 2nd ed. Cambridge, U.K.: Cambridge University Press.
134. Perez-Prado, A. and Thirunarayanan, M. (2002). A qualitative comparison of online and classroom-based sections of a course: exploring student perspectives. Educ. Media Int., 39(2), 195-202.
135. Pirnay-Dummer, P. (2006). Expertise und modellbildung: Mitocar [Expertise and Model Building: Mitocar]. Ph.D. dissertation. Freiburg, Germany: Freiburg University.
136. Popham, J. W. (1991). Appropriateness of instructor’s test-preparation practices. Educ. Meas. Issues Pract., 10(4), 12-16.
137. Prichard, J. S. (2006). Team-skills training enhances collaborative learning. Learn. Instruct., 16, 256-265.
138. Qureshi, S. (1995). Supporting Electronic Group Processes: A Social Perspective. Paper presented at the Association for Computing Machinery (ACM) Special Interest Group on Computer Personnel Research Annual Conference, April 6-8, Nashville, TN.
139. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychol. Bull., 124, 372-422.
140. Reigeluth, C. M. (1989). Educational technology at the crossroads: new mindsets and new directions. Educ. Technol. Res. Dev., 37 (1), 67-80.*
141. Reilly, B. (1994). Composing with images: a study of high school video producers. In Proceedings of ED-MEDIA 94: Educational Multimedia and Hypermedia. Charlottesville, VA: Association for the Advancement of Computing in Education.
142. Reiser, R. A. and Mory, E. H. (1991). An examination of the systematic planning techniques of two experienced teachers. Educ. Technol. Res. Dev., 39(3), 71-82.
143. Rentsch, J. R. and Hall, R. J., Eds. (1994). Members of Great Teams Think Alike: A Model of Team Effectiveness and Schema Similarity among Team Members, Vol. 1, pp. 22-34. Stamford, CT: JAI Press.
144. Rentsch, J. R., Small, E. E., and Hanges, P. J. (in press). Cognitions in organizations and teams: What is the meaning of cognitive similarity? In The People Make the Place, edited by B. S. B. Schneider. Mahwah, NJ: Lawrence Erlbaum Associates.
145. Robinson, R. S. (1994). Investigating Channel One: a case study report. In Watching Channel One, edited by De Vaney, pp. 21-41. Albany, NY: SUNY Press.
146. Robinson, R. S. (1995). Qualitative research: a case for case studies. In Instructional Technology: Past, Present and Future, 2nd ed., edited by G. J. Anglin, pp. 330-339. Englewood, CO: Libraries Unlimited.
147. Ross, S. M. and Morrison, G. R. (2004). Experimental research methods. In Handbook of Research on Educational Communications and Technology, 2nd ed., edited by D. Jonassen, pp. 1021-1043. Mahwah, NJ: Lawrence Erlbaum Associates.
148. Rourke, L., Anderson, T., Garrison, D. R., and Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. Int. J. Artif. Intell. Educ., 12, 8-22.
149. Rowe, A. L. and Cooke, N. J. (1995). Measuring mental models: choosing the right tools for the job. Hum. Resource Dev. Q., 6, 243-255.
150. Russo, J. E., Johnson, E. J., and Stephens, D. L. (1989). The validity of verbal protocols. Mem. Cogn., 17, 759-769.
151. Salas, E. and Cannon-Bowers, J. A. (2000). The anatomy of team training. In Training and Retraining: A Handbook for Business, Industry, Government, and the Military, edited by S. T. J. D. Fletcher, pp. 312-335. New York: Macmillan.
152. Salas, E. and Cannon-Bowers, J. A. (2001). Special issue preface. J. Org. Behav., 22, 87-88.
153. Salas, E. and Fiore, S. M. (2004). Why team cognition? An overview. In Team Cognition: Understanding the Factors That Drive Process and Performance, edited by E. Salas and S. M. Fiore. Washington, D.C.: American Psychological Association.
154. Salomon, G. and Perkins, D. N. (1998). Individual and social aspects of learning. In Review of Research in Education, Vol. 23, edited by P. Pearson and A. Iran-Nejad, pp. 1-24. Washington, D.C.: American Educational Research Association.*
155. Salvucci, D. D. (1999). Mapping eye movements to cognitive processes [doctoral dissertation, Carnegie Mellon University]. Dissert. Abstr. Int., 60, 5619.
156. Sapsford, R. and Jupp, V. (1996). Data Collection and Analysis. London: SAGE.
157. Savenye, W. C. (1989). Field Test Year Evaluation of the TLTG Interactive Videodisc Science Curriculum: Effects on Student and Teacher Attitude and Classroom Implementation. Austin, TX: Texas Learning Technology Group of the Texas Association of School Boards.
158. Savenye, W. C. (2004a). Evaluating Web-based learning systems and software. In Curriculum, Plans, and Processes in Instructional Design: International Perspectives, edited by N. Seel and Z. Dijkstra, pp. 309-330. Mahwah, NJ: Lawrence Erlbaum Associates.
159. Savenye, W. C. (2004b). Alternatives for assessing learning in Web-based distance learning courses. Distance Learn., 1(1), 29-35.*
160. Savenye, W. C. (2006). Improving online courses: what is interaction and why use it? Distance Learn., 2(6), 22-28.
161. Savenye, W. C. (2007). Interaction: the power and promise of active learning. In Finding Your Online Voice: Stories Told by Experienced Online Educators, edited by M. Spector. Mahwah, NJ: Lawrence Erlbaum Associates.
162. Savenye, W. C. and Robinson, R. S. (2004). Qualitative research issues and methods: an introduction for instructional technologists. In Handbook of Research on Educational Communications and Technology, 2nd ed., edited by D. Jonassen, pp. 1045-1071. Mahwah, NJ: Lawrence Erlbaum Associates.
163. Savenye, W. C. and Robinson, R. S. (2005). Using qualitative research methods in higher education. J. Comput. Higher Educ., 16(2), 65-95.
164. Savenye, W. C. and Strand, E. (1989). Teaching science using interactive videodisc: results of the pilot year evaluation of the Texas Learning Technology Group Project. In Eleventh Annual Proceedings of Selected Research Paper Presentations at the 1989 Annual Convention of the Association for Educational Communications and Technology in Dallas, Texas, edited by M. R. Simonson and D. Frey. Ames, IA: Iowa State University.
165. Savenye, W. C., Leader, L. F., Schnackenberg, H. L., Jones, E. E. K., Dwyer, H., and Jiang, B. (1996). Learner navigation patterns and incentive on achievement and attitudes in hypermedia-based CAI. Proc. Assoc. Educ. Commun. Technol., 18, 655-665.
166. Sax, G. (1980). Principles of Educational and Psychological Measurement and Evaluation, 2nd ed. Belmont, CA: Wadsworth.
167. Schneider, W. and Shiffrin, R. M. (1977). Controlled and automatic human information processing. I. Detection, search, and attention. Psychol. Rev., 84, 1-66.
168. Schweiger, D. M. (1986). Group approaches for improving strategic decision making: a comparative analysis of dialectical inquiry, devil’s advocacy, and consensus. Acad. Manage. J., 29(1), 51-71.
169. Seel, N. M. (1999). Educational diagnosis of mental models: assessment problems and technology-based solutions. J. Struct. Learn. Intell. Syst., 14, 153-185.
170. Seel, N. M. (2004). Model-centered learning environments: theory, instructional design, and effects. In Instructional Design: Addressing the Challenges of Learning Through Technology and Curriculum, edited by N. Seel and S. Dijkstra, pp. 49-73. Mahwah, NJ: Lawrence Erlbaum Associates.
171. Seel, N. M., Al-Diban, S., and Blumschein, P. (2000). Mental models and instructional planning. In Integrated and Holistic Perspectives on Learning, Instruction, and Technology: Understanding Complexity, edited by J. M. Spector and T. M. Anderson, pp. 129-158. Dordrecht: Kluwer.*
172. Segers, M., Dochy, F., and Cascallar, E., Eds. (2003). Optimising New Modes of assessment: In Search of Qualities and Standards. Dordrecht: Kluwer.
173. Shepard, L. (2000). The role of assessment in a learning culture. Educ. Res., 29(7), 4-14.
174. Shiffrin, R. M. and Schneider, W. (1977). Controlled and automatic human information processing. II. Perceptual learning, automatic attending, and a general theory. Psychol. Rev., 84, 127-190.*
175. Shin, E. J., Schallert, D., and Savenye, W. C. (1994). Effects of learner control, advisement, and prior knowledge on young students’ learning in a hypertext environment. Educ. Technol. Res. Dev., 42(1), 33-46.
176. Smith, P. L. and Wedman, J. F. (1988). Read-think-aloud protocols: a new data source for formative evaluation. Perform. Improv. Q., 1(2), 13-22.
177. Spector, J. M. and Koszalka, T. A. (2004). The DEEP Methodology for Assessing Learning in Complex Domains. Arlington, VA: National Science Foundation.*
178. Spradley, J. P. (1979). The Ethnographic Interview. New York: Holt, Rinehart and Winston.*
179. Spradley, J. P. (1980). Participant Observation. New York: Holt, Rinehart and Winston.*
180. Stahl, G. (2006). Group Cognition: Computer Support for Building Collaborative Knowledge. Cambridge, MA: MIT Press.*
181. Staller, L. (2005). Understanding analog to digital converter specifications. [electronic version]. Embedded Syst. Design, February, 24, http://www.embedded.com/.
182. Stelmach, L. B., Campsall, J. M., and Herdman, C. M. (1997). Attentional and ocular movements. J. Exp. Psychol. Hum. Percept. Perform., 23, 823-844.
183. Stevenson, W. G. and Soejima, K. (2005). Recording techniques for electrophysiology. J. Cardiovasc. Electrophysiol., 16, 1017-1022.
184. Strauss, A. L. and Corbin, J. M. (1994) Grounded theory methodology: an overview. In Handbook of Qualitative Research, edited by N. K. Denzin and Y. Lincoln, pp. 273-285. Thousand Oaks, CA: SAGE.*
185. Sweller, J. (1988). Cognitive load during problem solving: effects on learning. Cogn. Sci., 12, 257-285.*
186. Sweller, J., van Merriënboer, J. J. G., and Paas, F. (1998). Cognitive architecture and instructional design. Educ. Psychol. Rev., 10, 251-295.
187. Sy, T. (2005). The contagious leader: Impact of the leader’s mood on the mood of group members, group affective tone, and group processes. J. Appl. Psychol., 90(2), 295-305.
188. Taylor, K. L. and Dionne, J. P. (2000). Accessing problemsolving strategy knowledge: the complementary use of concurrent verbal protocols and retrospective debriefing. J. Educ. Psychol., 92, 413-425.
189. Thompson, S. (2001). The authentic standards movement and its evil twin. Phi Delta Kappan, 82(5), 358-362.
190. Thorndike, R. M. (1997). Measurement and Evaluation in Psychology and Education, 6th ed. Upper Saddle River, NJ: Prentice Hall.*
191. Tiffin, J. and Rajasingham, L. (1995). In Search of the Virtual Class: Education in an Information Society. London: Routledge.
192. Titchener, E. B. (1900). The equipment of a psychological laboratory. Am. J. Psychol., 11, 251-265.*
193. Tuovinen, J. E. and Paas, F. (2004). Exploring multidimensional approaches to the efficiency of instructional conditions. Instruct. Sci., 32, 133-152.
194. Underwood, G., Chapman, P., Brocklehurst, N., Underwood, J., and Crundall, D. (2003). Visual attention while driving: sequences of eye fixations made by experienced and novice drivers. Ergonomics, 46, 629-646.
195. Underwood, G., Jebbett, L., and Roberts, K. (2004). Inspecting pictures for information to verify a sentence: eye movements in general encoding and in focused search. Q. J. Exp. Psychol., 57, 165-182.
196. Urch Druskat, V. and Kayes, D. C. (2000). Learning versus performance in short-term project teams. Small Group Res., 31, 328-353.
197. Van der Vleuten, C. P. M. and Schuwirth, L. W. T. (2005). Assessing professional competence: from methods to programmes. Med. Educ., 39, 309-317.
198. Van Gerven, P. W. M., Paas, F., van Merriënboer, J. J. G., and Schmidt, H. (2004). Memory load and the cognitive pupillary response in aging. Psychophysiology, 41, 167-174.
199. van Gog, T. (2006). Uncovering the Problem-Solving Process to Design Effective Worked Examples. Ph.D. dissertation. Heerlen: Open University of the Netherlands.
200. van Gog, T., Paas, F., and van Merriënboer, J. J. G. (2005a). Uncovering expertise-related differences in troubleshooting performance: combining eye movement and concurrent verbal protocol data. Appl. Cogn. Psychol., 19, 205-221.*
201. van Gog, T., Paas, F., van Merriënboer, J. J. G., and Witte, P. (2005b). Uncovering the problem-solving process: cued retrospective reporting versus concurrent and retrospective reporting. J. Exp. Psychol. Appl., 11, 237-244.
202. Van Maanen, J. (1988). Tales of the Field: On Writing Ethnography. Chicago, IL: The University of Chicago Press.
203. van Merriënboer, J. J. G. (1997). Training Complex Cognitive Skills: A Four-Component Instructional Design Model for Technical Training. Englewood Cliffs, NJ: Educational Technology Publications.*
204. van Merriënboer, J. J. G., Jelsma, O., and Paas, F. (1992). Training for reflective expertise: a four-component instructional design model for complex cognitive skills. Educ. Technol. Res. Dev., 40(2), 1042-1629.
205. Van Someren, M. W., Barnard, Y. F., and Sandberg, J. A. C. (1994). The Think Aloud Method: A Practical Guide to Modeling Cognitive Processes. London: Academic Press.
206. VanLehn, K. (1996). Cognitive skill acquisition. Annu. Rev. Psychol., 47, 513-539.*
207. Wainer, H. (1989). The future of item analysis. J. Educ. Meas., 26(2), 191-208.
208. Webb, E. J., Campbell, D. T., Schwartz, R. D., and Sechrest, L. (1966). Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago, IL: Rand McNally.
209. Webb, N. M. (1982). Student interaction and learning in small groups. Rev. Educ. Res., 52(3), 421-445.
210. Weitzman, E. A. and Miles, M. B. (1995). A Software Sourcebook: Computer Programs for Qualitative Data Analysis. Thousand Oaks, CA: SAGE.
211. Willis, S. C., Bundy, C., Burdett, K., Whitehouse, C. R., and O’Neill, P. A. (2002). Small-group work and assessment in a problem-based learning curriculum: a qualitative and quantitative evaluation of student perceptions of the process of working in small groups and its assessment. Med. Teacher, 24, 495-501.
212. Wolcott, H. F. (1990). Writing Up Qualitative Research. Newbury Park, CA: SAGE.*
213. Woods, D. R., Felder, R. M., Rugarcia, A., and Stice, J. E. (2000). The future of engineering education. Part 3. Development of critical skills. Chem. Eng. Educ., 34, 108-117.
214. Woolf, H. (2004). Assessment criteria: reflections on current practices. Assess. Eval. Higher Educ., 29, 479-493.*
215. Worchel, S., Wood, W., and Simpson, J. A., Eds. (1992). Group Process and Productivity. Newbury Park, CA: SAGE.
216. Yeo, G. B. and Neal, A. (2004). A multilevel analysis of effort, practice and performance: effects of ability, conscientiousness, and goal orientation. J. Appl. Psychol., 89, 231-247.*