Using cognitive interviews to validate an interpretive argument for the ETS iSkills™ assessment

Communications in Information Literacy

Volumn 3, Issue 2, 2009, Pages 99-127

  Snow, Eric ^a   Katz, Irvin R ^b  

^a SRI INTERNATIONAL   (United States)

^b EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77949506471     PISSN: 19335954     EISSN: None     Source Type: Journal    
DOI: 10.15760/comminfolit.2010.3.2.75     Document Type: Article

References (53)

1. American Association of School Librarians (AASL), Association for Educational Communications and Technology (AECT). (1998). Information literacy standards for student learning. In, Information power: Building partnerships for learning. Retrieved December 2003 from http://www.ala.org/
2. American Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME). (1999). Standards for educational and psychological testing. Washington, D.C.: AERA.
3. American Library Association (ALA) Presidential Committee on Information Literacy. (1989). Final report. Chicago, IL: ALA.
4. Association for College & Research Libraries. (2002). Information literacy competency standards for higher education. Retrieved December 2009, from http://www.ala.org/ala/mgrps/divs/acrl/standards/standards.pdf.
5. Borsboom, D., Mellenbergh, G., & Heerden, J. V. (2004). The concept of validity. Psychological review, 111, pp. 1061-1071.
6. st Century learning and information literacy. Change Magazine.
7. st century (2nd ed.). Washington, DC: National Education Association.
8. Brennan, R. L. (Ed.). (2006). Educational Measurement (4th ed.). Westport, CT: American Council on Education, Praeger Publishers.
9. Caravello, P. S., Herschman, J., & Mitchell, E. (2001). Assessing the information literacy of undergraduates: Reports from the UCLA library's information competencies survey project, ACRL Tenth National Conference. Denver, CO.
10. Cronbach, L. J. (1988). Five perspectives on the validity argument. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 3-15). Hillsdale, NJ: Lawrence Erlbaum.
11. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
12. st Century: What higher education must do to address the gap in information and communication technology proficiencies. Retrieved December 2009 from http://www.ets.org/Media/Tests/Information_and_Communication_Technology_Literacy/ICTwhitepaperfinal.pdf.
13. Ericsson, K. A, & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
14. Fidel, R., Davies, R. K., Douglass, M. H., Jenny, K. et al. (1999). A visit to the information mall: Web searching behavior of high school students. Journal of the American Society for Information Science and Technology, 50(1), 24-37.
15. Friedlander, A. (2002). Dimensions and use of scholarly information environment. Introduction to a data set assembled by the Digital Library Federation and Outsell, Inc. Washington, DC: Digital Library Federation and Council on Library and Information Resources. Retrieved December 20, 2007 from http://www.clir.org/pubs/reports/pub110/contents.html.
16. Hepworth, M. (1999, August). A study of tertiary students' information literacy and skills: The inclusion of information literacy and skills in the undergraduate curriculum. Paper presented at the 65th IFLA Council and General Conference, Bangkok, Thailand.
17. Gorin, J. (2007). Reconsidering issues in validity theory. Educational Researcher, 8(36), 456-462. Washington, DC: American Educational Research Association.
18. Gorin, J. (2006). Test design with cognition in mind. Educational measurement: Issues and practice, 4(25), 21-35.
19. International Technology Education Association (ITEA). (2003). Advancing excellence in technological literacy: Student assessment, professional development, and program standards. Available from ITEA website, http://www.iteaconnect.org/Publications/publications.htm.
20. International Technology Education Association (ITEA). (2000). Standards for technological literacy: Content for the study of technology. Available from ITEA website, http://www.iteaconnect.org/Publications/publications.htm.
21. International ICT Literacy Panel (2002). Digital transformation: A framework for ICT literacy. Princeton, NJ: Educational Testing Service. Retrieved November 12, 2007, from www.ets.org/ictliteracy/digital1.html.
22. International Society for Technology in Education (ISTE). (1998). National Educational Technology Standards for Students (NETS·S). Retrieved June 2003 from http://cnets.iste.org/students/s_stands.html.
23. Kane, M. (2006). Validation. In R.L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). American Council on Education, Praeger Publishers.
24. Kane, M. (2004). Certification testing as an illustration of argument-based validation. Measurement, 2(3), 135-170.
25. Kane, M. (1992). An argument-based approach to validity. Psychological bulletin, (3), 527-535.
26. Katz, I. R. (2007). Testing information literacy in digital environments: ETS's iSkills™ assessment. Information Technology and Libraries, 26(3), 3-12.
27. Katz, I. R. (2005). Beyond technical competence: Literacy in information and communication technology. Educational Technology Magazine, 45(6), 44-47.
28. Katz, I. R., Elliot, N., Attali, Y., Scharf, D., Powers, D., Huey, H., et al. (2009). Multiple methods of assessing information literacy: A 125 case study. ETS Research Spotlight, Issue 2, 21-27.
29. Katz, I. R., Williamson, D., Nadelman, H., Kirsch, I., Almond, R., Cooper, P., et al. (2004). Assessing information and communications technology literacy for higher education. Paper presented at the 30th Annual Conference of the International Association for Educational Assessment (IAEA), Philadelphia, PA, June 13-18, 2004.
30. Katz, I. R. & Macklin, A. S. (2007). Information and communication technology literacy: Integration and assessment in higher education. Journal of Systemics, Cybernetics and Informatics, 5(4), 50-55.
31. LeCompte, M. D., & Schensul, J. (1999). Designing and conducting ethnographic research (Book 1, Ethnographer's Toolkit). Walnut Creek, CA: AltaMira Press.
32. Lord, F. M., & Novick, M. R, (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
33. Manuel, K. (2002). Teaching information literacy to Generation Y. Journal of Library Administration, 36(1/2), 195-217.
34. McDonald, R. (1999). Test theory: A unified treatment. New Jersey: Lawrence Erlbaum Associates.
35. Metzger, M. J. (2007). Making sense of credibility on the web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, 58(13), 2078-2091.
36. National Research Council Committee on the Foundations of Assessment; Pellegrino, J., Chudowsky, N., & Glaser, R. (Eds.) (2001). Knowing what students know: The science and design of educational assessment. Washington, D.C.: National Academy Press. Available at: http://www.nap.edu/books/0309072727/html/
37. National Research Council Committee on Information Technology Literacy. (1999). Being fluent with information technology. Washington, D.C.: National Academy Press. Retrieved January 2004 from http://www.nap.edu/html/beingfluent/
38. Oblinger, D. G., & Oblinger, J. (2005). Educating the net generation. Boulder, CO: Educause.
39. Online Computer Library Center [OCLC] (2006, June). College students' perceptions of libraries and information resources. Retrieved December 20, 2007, from http://www.oclc.org/reports/pdfs/studentperceptions.pdf.
40. Online Computer Library Center [OCLC] (2002, June). OCLC white paper on the information habits of college students. Retrieved February 2, 2008, from http://www5.oclc.org/downloads/community/informationhabits.pdf.
41. st century skills: The current landscape. Tucson, AZ: Author. Retrieved September 2005 from http://www.21stcenturyskills.org/images/stories/otherdocs/Assessment_Landscape.pdf
42. Partnership for 21st Century Skills. (2003). Learning for the 21st Century: A report and mile guide for 21st century skills. Tucson, AZ: Author. Retrieved December 2009 from http://www.21stcenturyskills.org/downloads/P21_Report.pdf.
43. Rieh, S. Y. (2002). Judgment of information quality and cognitive authority in the Web. Journal of the American Society for Information Science and Technology, 53, 145-161.
44. Snow, E. (2008). Can college students evaluate information sources? Validating a web-based assessment of evaluation skills. (Doctoral dissertation, University of Colorado, Boulder, 9 August 2008). Ann Arbor: UMI, 2008. 3315820.
45. Someren, M. W. (1994). The think aloud method: A practical guide to modeling cognitive processes. San Diego: Academic Press.
46. Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72-101.
47. Spradley, J. (1980). Participant observation. Orlando: Harcourt Brace College Publishers.
48. Tannenbaum, R. J. & Katz, I. R. (2008). Setting standards on the core and advanced iSkills assessments (ETS RM-08-04). Princeton, NJ: Educational Testing Service.
49. VanLehn, K. (1989). Problem-solving and cognitive skill acquisition. In M. Posner (Ed.), The foundations of cognitive science (pp. 527-580). Cambridge, MA: MIT Press.
50. Willis, G. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks: Sage.
51. Willis, G. (1994). Cognitive interviewing: A "how to" guide. North Carolina: Research Triangle Institute.
52. Wolcott, H. F. (1995). The art of fieldwork. Walnut Creek, CA: AltaMira Press.
53. Zimmerman, D. W. & Williams, R. H. (1997). Properties of the Spearman correction for attenuation formula for normal and realistic non-normal distributions. Applied Psychological Measurement, 21, 253-270.