Improving Survey Methods With Cognitive Interviews in Small- and Medium-Scale Evaluations

American Journal of Evaluation

Volumn 33, Issue 3, 2012, Pages 414-430

  Ryan, Katherine ^a   Gannon Slater, Nora ^a   Culbertson, Michael J ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

cognitive aspects of survey methodology; cognitive interviews; evaluation methodology; survey design; validity

Indexed keywords

EID: 84864384002     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214012441499     Document Type: Article

References (48)

1. Abt Associates. (2011). Retrieved fromhttp://www.abtassociates.com/page.cfm?PageID=1468.
2. Ackerman A.,Blair J. Paper presented at the Annual Meeting of the American Association of Public Opinion Research; 2006, MayHollywood, FL; 2006, May.
3. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 1999:.
4. American Institute for Research (2011). Retrieved fromhttp://www.air.org/files/4_pager_AIR_Health_Polict_2011_V10F.pdf.
5. Anderson L.W.,Postlethaite N.Program evaluation: large and small studies. Paris, France: International Institute for Educational Planning/UNESCO; 2007:.
6. Beatty P.C.,Willis G.B.Research synthesis: The practice of cognitive interviewing.Public Opinion Quarterly. 2007;71:287-311.
7. Blair J.,Brick P.D.May). Paper presented at the Annual Meeting of the American Association of Public Opinion Research Hollywood; 2009FL; 2009.
8. Blair J.,Conrad F.G.Sample size for cognitive interview pretesting.Public Opinion Quarterly. 2011;75:636-658.
9. Blair J.,Presser S.Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association. Alexandria, VA: American Statistical Association; 1993:370-375.
10. Bradburn N.M.Understanding the question-answer process.Statistics Canada. 2004;30:5-15.
11. Collins D.Proceedings of the 6th Questionnaire Evaluation Standard for Testing Conference. Ottawa, Statistics Canada:; 2007:.
12. Conrad F.G.,Blair J.Sources of errors in cognitive interviews.Public Opinion Quarterly. 2009;73:32-55.
13. Czaja R.,Blair J.Designing surveys: A guide to decisions and procedures. Thousand Oaks, CA: Pine Forge Press; 2005:.
14. DeMaio T.J.,Landreth A.Methods for testing and evaluating survey questionnaire. Presser S.Rothgeb J.M.Couper M.P.Lessler J.T.Martin E.Martin J.Singer E., ed. Hoboken, NJ: John Wiley; 2004:89-108.
15. Desimone L.M.,le Floch K.C.Are we asking the right questions? Using cognitive interviews to improve surveys in education research.Education Evaluation and Policy Analysis. 2004;26:1-22.
16. Donaldson S.I.What counts as credible evidence in applied research and evaluation practice?. Donaldson S.I.Christie C.A.Mark M.M., ed. Thousand Oaks, CA: Sage; 2008:2-18.
17. Elmore R.School reform from the inside out: Policy, practice, and performance. Cambridge, MA: Harvard Education Press; 2004:.
18. Ericcson K.A.,Simon H.A.A protocol analysis: Verbal reports as data. 1993). A protocol analysis: Verbal reports as data (2nd ed.). Cambridge: MIT Press.Cambridge: MIT Press; 1993:.
19. Gottfredson D.C.,Gottfredson G.D.Quality of school-based prevention programs: Results from a national survey.Journal of Research in Crime and Delinquency. 2002;39:3-35.
20. Groves R.M.,Fowler F.J. J.,Couper M.P.,Lepkowski J.M.,Singer E.,Tourangeau R.Survey methodology. 2009). Survey methodology (2nd ed.). New York, NY: Wiley.New York, NY: Wiley; 2009:.
21. Hamilton L.,Stecher B.,Marsh J.,McCombs J.,Robyn A.,Russell J.,Barney HStandards-based accountability under no child left behind. Santa Monica, CA: Rand Education; 2007:.
22. Hawley W.D.,Rollie D.L.The keys to effective schools: Educational reform as a continuous improvement. Thousand Oaks, CA: Corwin Press; 2007:.
23. Howell E.M.,Yemane A.As assessment of evaluation designs.American Journal of Evaluation. 2006;27:219-236.
24. Irwin D.E.,Varni J.M.,Yeatts K.,DeWalt D.A.Cognitive interviewing methodology in the development of a pediatric item bank: A Patient Reported Outcomes Measurement Information System (PROMIS) study.Health and Quality of Life Outcomes. 2009;7:3.
25. Krosnick J.A.,Presser S.Handbook of survey research. 2010). Questionnaire design. In WrightJ. D.MarsdenP. V. (Eds.), Handbook of survey research (2nd ed., pp. 263-313). West Yorkshire, England: Emerald Group.Wright J.D.Marsden P.V., ed. West Yorkshire, England: Emerald Group; 2010:263-313.
26. Mayer D.Measuring instructional practice: Can policymakers trust survey data?.Educational Evaluation and Policy Analysis. 1999;21:29-45.
27. McFarland S.G.Effects of question order on survey responses.Public Opinion Quarterly. 1981;45:208-215.
28. Messick S.Validity of psychological assessment: Validation of inferences from persons responses and performances as scientific inquiry into score meaning.American Psychologist. 1995;50:741-749.
29. No Child Left Behind Act of 2001, Pub L. No. 107-110, 115 Stat. 1435. (2002).
30. Palardy G.J.,Rumberger R.W.Teacher effectiveness in the first grade: The importance of background qualifications, attitudes, and instructional practices for student learning.Educational Evaluation and Policy Analysis. 2008;30:111-140.
31. Ryan K.E.,Gandha T.,Culbertson M.,Lim E.,Wakita SIL Assessment consequences evaluation: Year one report. Urbana: University of Illinois at Champaign-Urbana, Illinois Assessment Consequences Evaluation; 2009:.
32. Sabaratnam P.,Klein J.D.Measuring youth development outcomes for community program evaluation and quality improvement: findings from Dissemination of the Rochester Evaluation of Asset Development for Youth (READY) tool.Journal of Public Health Management Practice. 2006;12:S88-S94.
33. Schwarz N.Cognitive aspects of survey methodology.Applied Cognitive Psychology. 2007;21:277-287.
34. Sudman S.,Bradburn N.,Schwarz N.Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass; 1996:.
35. Supovitz J.A.,Taylor B.S.Systemic education evaluation: Evaluating the impact of systemwide reform in education.American Journal of Evaluation. 2005;26:204-230.
36. Todorov A.The accessibility and applicability of knowledge: Predicting context effects in national surveys.Public Opinion Quarterly. 2000;64:429-451.
37. Tourangeau R.Cognitive aspects of survey methodology: Building a bridge between disciplines. Jabine T.Straf M.Tanur J.Tourangeau R., ed. Washington, DC: National Academy Press; 1984:73-100.
38. Tourangeau R.Cognitive aspects of survey measurement and mismeasurement.International Journal of Public Opinion Research. 2003;15:3-7.
39. Tourangeau R.,Bradburn N.M.Handbook of survey research. 2010). The psychology of survey response. In WrightJ. D.MarsdenP. V. (Eds.), Handbook of survey research (2nd ed., pp. 315-346). West Yorkshire, England: Emerald Group.Wright J.D.Marsden P.V., ed. West Yorkshire, England: Emerald Group; 2010:315-346.
40. Tourangeau R.,Rasinsk K.A.Cognitive processes underlying context effects in attitude measurement.Psychological Bulletin. 1988;103:299-314.
41. Tourangeau R.,Rips L.J.,Rasinski K.The psychology of survey response. Cambridge, England: Cambridge University Press; 2000:.
42. Trenholm C.,Devaney B.,Fortson K.,Quay L.,Wheeler J.,Clark M.Impacts of four Title V, section 510 abstinence education programs. Princeton, NJ: Mathematica Policy Research; 2007:.
43. Trochim. (2006). Construct validity. Retrieved fromhttp://www.socialresearchmethods.net/kb/constval.php.
44. U.S. Department of Education. (2009). Elementary & secondary education: No child left behind: A desktop reference. Retrieved fromhttp://www2.ed.gov/admins/lead/account/nclbreference/index.html.
45. Westat. (2011). Retrieved fromhttp://www.westat.com/westat/research_areas/health_and_medical_studies/health_and_medical_pe.cfm.
46. Willis G.B.Methods for testing and evaluating questionnaires. Presser S.Rothgeb J.M.Couper M.P.Lessler J.T.Martin E.Martin J.Singer E., ed. Hoboken, NJ: John Wiley; 2004:23-44.
47. Willis G.B.Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage; 2005:.
48. Willis G.,Zahnd E.Questionnaire design from a cross-cultural perspective: An empirical investigation of Koreans and Non-Koreans.Journal of Health Care for the Poor and Underserved. 2007;18:197-217.