Keeping your audience in mind: applying audience analysis to the design of interactive score reports

Assessment in Education: Principles, Policy and Practice

Volumn 21, Issue 4, 2014, Pages 442-463

  Zapata Rivera, Juan Diego ^a   Katz, Irvin R ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

assessment; audience analysis; score interpretation; score reports; validity

Indexed keywords

EID: 84907676184     PISSN: 0969594X     EISSN: 1465329X     Source Type: Journal    
DOI: 10.1080/0969594X.2014.936357     Document Type: Article

References (78)

1. Aleven, V., & Koedinger, K. (2000). Limitations of student control: Do students know when they need help? In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Proceedings of ITS 2000 (pp. 292–303). New York, NY: Springer.
2. Alexander, C., Ishikawa, S., & Silverstein, M. (1977). A pattern language: Towns, buildings, construction. Oxford: Oxford University Press.
3. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: Author.
4. Aschbacher, P. R., & Herman, J. L. (1991). Guidelines for effective score reporting (CSE Technical Report). Los Angeles, CA: UCLA Center for Research on Evaluation, Standards, and Student Testing.
5. Behrens, J. T., DiCerbo, K., Murphy, D., & Robinson, D. (2013, April 28–30). Conceptual frameworks for reporting results of assessment activities. Paper presented at the annual meeting of the National Council for Measurement in Education, San Francisco, CA.
6. Bennett, R. E., & Gitomer, D. H. (2009). Transforming K–12 assessment: Integrating accountability testing, formative assessment and professional support. In C. Wyatt-Smith & J. Cumming (Eds.), Educational assessment in the 21st century (pp. 43–61). New York, NY: Springer.10.1007/978-1-4020-9964-9
7. Brown, G. T. L., & Hirschfeld, G. H. F. (2008). Students’ conceptions of assessment: Links to outcomes. Assessment in Education: Principles, Policy & Practice, 15, 3–17.
8. Brunner, C., Fasca, C., Heinze, J., Honey, M., Light, D., Mandinach, E., & Wexler, D. (2005). Linking data and learning: The Grow Network study. Journal of Education for Students Placed at Risk (JESPAR), 10, 241–267.10.1207/s15327671espr1003_2
9. Bull, S., Brna, P., & Dimitrova, V. (Eds.). (2003). Proceedings of Learner Modeling for Reflection Workshop, Volume V of the AIED2003 Supplementary Proceedings. Retrieved from http://sydney.edu.au/engineering/it/~aied/Supp_procs.html#vol5
10. Bull, S., & Pain, H. (1995). Did I say what I think I said, and do you agree with me? Inspecting and questioning the student model. In J. Greer (Ed.), Proceedings of the World Conference on Artificial Intelligence in Education (pp. 501–508). Charlottesville, VA: AACE.
11. Chen, E., Heritage, M., & Lee, J. (2005). Identifying and monitoring students’ learning needs with technology. Journal of Education for Students Placed at Risk (JESPAR), 10, 309–332.10.1207/s15327671espr1003_6
12. Coburn, C. E., Honig, M. I., & Stein, M. K. (2009). What’s the evidence on districts’ use of evidence? In J. Bransford, D. J. Stipek, N. J. Vye, L. Gomez, & D. Lam (Eds.), The role of research in educational improvement (pp. 67–88). Cambridge, MA: Harvard Education Press.
13. Coburn, C. E., & Talbert, J. E. (2005, April). Conceptions of evidence use in school districts: Mapping the terrain. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.
14. Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York, NY: Harper Perennial.
15. Deng, N., & Yoo, H. (2009). Resources for reporting test scores: A bibliography for the assessment community. Retrieved from https://ncme.org/resources/blblio1/NCME.Bibliography-5-6-09_score_reporting.pdf
16. Dimitrova, V., Self, J., & Brna, P. (2001). Applying interactive open learner models to learning technical terminology. In M. Bauer, P. J. Gmytrasiewics, & J. Vassileva (Eds.), Proceedings of UM 2001 (pp. 148–157). New York, NY: Springer.
17. Fast, E. F. (2002). A guide to effective accountability reporting: Designing public reports that effectively communicate accountability, assessment and other quantitative education indicators in an easily understood format. Washington, DC: Council of Chief State School Officers.
18. Flower, L. (1985). Problem-solving strategies for writing (2nd ed.). New York, NY: Harcourt Brace Jovanovich.
19. Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1994). Design patterns: Elements of reusable object-oriented software. New York, NY: Addison Wesley.
20. Goodman, D. P., & Hambleton, R. K. (2004). Student test score reports and interpretive guides: Review of current practices and suggestions for future research. Applied Measurement in Education, 17, 145–220.10.1207/s15324818ame1702_3
21. Guerard, E. B. (2001, May 17). School leaders learn to make data-driven decisions. School News. Retrieved from http://www.eschoolnews.com/news/showstory.cfm?ArticleID=2621
22. Hackos, J., & Redish, J. (1998). User and task analysis for interface design. New York, NY: Wiley.
23. Hambleton, R. K., & Slater, S. (1997). Are NAEP executive summary reports understandable to policy makers and educators? (CSE Technical Report 430). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing.
24. Hambleton, R. K., & Zenisky, A. (2010, November). Improvements to student score reporting: Steps and more use of suitable methodologies. Paper presented at the ETS Conference on Score Reporting, Princeton, NJ.
25. Hambleton, R. K., & Zenisky, A. L. (2013). Reporting test scores in more meaningful ways: A research-based approach to score report design. In K. F. Geisinger, B. A. Bracken, J. F. Carlson, J. I. C. Hansen, N. R. Kuncel, S. P. Reise, & M. C. Rodriguez (Eds.), APA handbook of testing and assessment in psychology, Vol. 3: Testing and assessment in school psychology and education (pp. 479–494). Washington, DC: American Psychological Association.
26. Hartley, D., & Mitrovic, A. (2002). Supporting learning by opening the student model. In S. Cerri, G. Gouarderes, & F. Paraguacu (Eds.), Proceedings of the 6th International conference on Intelligent Tutoring Systems, ITS 2002 (pp. 453–462). Biarritz: LCNS 2363.
27. Hattie, J. (2009). Visibly learning from reports: The validity of score reports. Retrieved from http://www.oerj.org/View?action=viewPDF&paper=6
28. Hix, D., & Hartson, H. R. (1993). Developing user interfaces: Ensuring usability through product and process. New York, NY: Wiley.
29. Honig, M. I. (2003). Building policy from practice: District central office administrators’ roles and capacity for implementing collaborative education policy. Educational Administration Quarterly, 39, 292–338.10.1177/0013161X03253414
30. Honig, M. I., & Coburn, C. E. (2008). Evidence-based decision-making in school district central offices: Toward a policy and research agenda. Educational Policy, 22, 578–608.
31. Hullman, J., Rhodes, R., Rodriguez, F., & Shah, P. (2011). Research on graph comprehension and data interpretation: Implications for score reporting. In D. Zapata-Rivera & R. Zwick (Eds.), Test score reporting: Perspectives from the ETS score reporting conference(Research report 11–45, pp. 1–16). Princeton, NJ: Educational Testing Service.
32. Jaeger, R. M. (1998). Reporting the results of the National Assessment of Educational Progress. Washington, DC: American Institutes for Research.
33. Jaeger, R. M. (2003). NAEP validity studies: Reporting the results of the National Assessment of Educational Progress. Washington, DC: National Center for Education Statistics.
34. Kane, M. (2006). Validation. In R. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: Greenwood.
35. Kane, M. (2013). Validating the interpretation and use of test scores. Journal of Educational Measurement, 50, 1–73.10.1111/jedm.2013.50.issue-1
36. Kay, J. (1995). The UM toolkit for cooperative user modelling. User Modeling and User-Adapted Interaction, 4, 149–196.10.1007/BF01100243
37. Landauer, T. K. (1995). The trouble with computers: Usefulness, usability, and productivity. Cambridge, MA: The MIT Press.
38. Lukin, L. E., Bandalos, D. L., Eckhout, T. J., & Mickelson, K. (2004). Facilitating the development of assessment literacy. Educational Measurement: Issues and Practice, 23, 26–32.
39. Mac Iver, M.A., & Farley, E. (2003). Bringing the district back in: The role of the central office in improving instruction and student achievement (CRESPAR Report No. 65). Baltimore, MD: Johns Hopkins University.
40. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.10.1037/0003-066X.50.9.741
41. Miller, N. (2003, February). Best practices. Super Tech News, 2(1). Retrieved from http://www.blegroup.com/supertechnews/feb03.htm#best
42. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessment (with discussion). Measurement: Interdisciplinary Research and Perspective, 1, 3–62.
43. NAEP Data Explorer. (2011). NAEP Data Explorer (NDE). Retrieved from http://nces.ed.gov/nationsreportcard/naepdata/
44. National Education Goals Panel. (1998). Talking about tests: An idea book for state leaders. Washington, DC: US Government Printing Office. Retrieved from http://govinfo.library.unt.edu/negp/reports/98talking.PDF
45. Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 175–220.
46. Nielsen, J. (1993). Iterative user-interface design. Computer, 26(11), 32–41.10.1109/2.241424
47. Nielsen, J. (1994). Usability engineering. San Francisco, CA: Morgan Kaufmann.
48. PADI Research Group. (2004). Design patterns for assessing science inquiry. Menlo Park, CA: SRI International.
49. Ryan, J. M. (2003). An analysis of item mapping and test reporting strategies. Retrieved from http://center.serve.org/_downloads/publications/Analysis_of_State_Test_Score_Reporting_Strategies.pdf
50. Ryan, J. M. (2006). Practices, issues, and trends in student test score reporting. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 677–710). Mahwah, NJ: Lawrence Erlbaum.
51. Simon, H. A. (1981). The sciences of the artificial (2nd ed.). Cambridge, MA: The MIT Press.
52. Tidwell, J. (2005). Designing interfaces: Patterns for effective interaction design. Sebastopol, CA: O’Reilly Media.
53. Tufte, E. R. (1983). The visual display of quantitative information. Cheshire, CT: Graphics Press.
54. Tufte, E. R. (1996). Visual explanations. Cheshire, CT: Graphics Press.
55. Underwood, J. S., Reshetar, R., & Leahy, S. (2006). Score report design heuristics (Unpublished report). Educational Testing Service, Princeton, NJ.
56. Underwood, J. S., Zapata-Rivera, D., & VanWinkle, W. (2007). Growing pains: Teachers using and learning to use IDMS (Research Memorandum 08–07). Princeton, NJ: Educational Testing Service.
57. Underwood, J. S., Zapata-Rivera, D., & VanWinkle, W. (2010). An evidence-centered approach to using assessment data for policymakers (ETS Research Report No. RR-10-03). Princeton, NJ: ETS.
58. Van Labeke, N., Brna, P., & Morales, R. (2007). Opening up the interpretation process in an open learner model. International Journal of Artificial Intelligence in Education, 17, 305–338.
59. van Welie, M. (2011). A pattern library for interaction design. Retrieved from http://www.welie.com/patterns/index.php
60. VanWinkle, W., Vezzu, M., & Zapata-Rivera, D. (2011). Question-based reports for policymakers (Research Memorandum 11–16). Princeton, NJ: Educational Testing Service.
61. Vezzu, M., VanWinkle, W., & Zapata-Rivera, D. (2011, April). Designing and evaluating an interactive score report for students. Paper presented at the meeting of the American Educational Research Association, New Orleans, LA.
62. Wainer, H. (1997). Visual revelations. New York, NY: Copernicus Books.10.1007/978-1-4612-2282-8
63. Wainer, H. (2005). Graphic discovery. Princeton, NJ: Princeton University Press.
64. Wainer, H., Hambleton, R. K., & Meara, K. (1999). Alternative displays for communicating NAEP results: A redesign and validity study. Journal of Educational Measurement, 36, 301–335. doi:10.1111/j.1745-3984.1999.tb00559.x10.1111/jedm.1999.36.issue-4
65. Wayman, J. C. (2005). Involving teachers in data-driven decision making: Using computer data systems to support teacher inquiry and reflection. Journal of Education for Students Placed at Risk (JESPAR), 10, 295–308.10.1207/s15327671espr1003_5
66. Wayman, J. C., Midgley, S., & Stringfield, S. (2005, April). Collaborative teams to support data-based decision making and instructional improvement. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.
67. Williamson, D. M., Bauer, M. I., Mislevy, R. J., & Behrens, J. T. (2003, April). An ECD approach to designing for reusability in innovative assessment. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
68. Zapata-Rivera, D. (2011). Designing and evaluating score reports for particular audiences. In D. Zapata-Rivera & R. Zwick (Eds.), Test score reporting: Perspectives from the ETS score reporting conference(Research Report 11–45, pp. 32–61). Princeton, NJ: Educational Testing Service.
69. Zapata-Rivera, D., Hansen, E. G., Shute, V. J., Underwood, J. S., & Bauer, M. I. (2007). Evidence-based approach to interacting with open student models. International Journal of Artificial Intelligence in Education, 17, 273–303.
70. Zapata-Rivera, D., & VanWinkle, W. (2010). A research-based approach to designing and evaluating score reports for teachers (Research Memorandum 10–01). Princeton, NJ: Educational Testing Service.
71. Zapata-Rivera, D., VanWinkle, W., & Zwick, R. (2010, April–May). Exploring effective communication and appropriate use of assessment results through teacher score reports. Paper presented at the meeting of the National Council on Measurement in Education, Denver, CO.
72. Zapata-Rivera, D., VanWinkle, W., & Zwick, R. (2011, April). Exploring teachers’ understanding and use of assessment results. Paper presented at the meeting of the American Educational Research Association, New Orleans, LA.
73. Zapata-Rivera, D., VanWinkle, W., & Zwick, R. (2012). Applying score design principles in the design of score reports for CBAL™ teachers (ETS Research Memorandum RM-12-20). Princeton, NJ: ETS.
74. Zapata-Rivera, D., Vezzu, M., & Biggers, K. (2013, April 27–May 1). Supporting teacher communication with parents and students using score reports. Paper presented at the annual meeting of the American Educational Research Association (AERA), San Francisco, CA.
75. Zapata-Rivera, J. D., & Greer, J. (2004). Interacting with Bayesian student models. International Journal of Artificial Intelligence in Education, 14, 127–163.
76. Zenisky, A. L., & Hambleton, R. K. (2012). Developing test score reports that work: The process and best practices for effective communication. Educational Measurement: Issues and Practice, 31, 21–26. doi:10.1111/j.1745-3992.2012.00231.x10.1111/emip.2012.31.issue-2
77. Zwick, R., Sklar, J., Wakefield, G., Hamilton, C., Norman, A., & Folsom, D. (2008). Instructional tools in educational measurement and statistics (ITEMS) for school personnel: Evaluation of three web-based training modules. Educational Measurement: Issues and Practice, 27, 14–27.10.1111/j.1745-3992.2008.00119.x
78. Zwick, R., Zapata-Rivera, D., & Hegarty, M. (2012, April). Comparing graphical and verbal representations of measurement error in test score reports. Paper presented at the meeting of the National Council on Measurement in Education, Vancouver, BC, Canada.