Direct behavior rating as a school-based behavior universal screener: Replication across sites

Journal of School Psychology

Volumn 52, Issue 1, 2014, Pages 63-82

  Kilgus, Stephen P ^a   Riley Tillman, T Chris ^b   Chafouleas, Sandra M ^c   Christ, Theodore J ^d   Welsh, Megan E ^c  

^a EAST CAROLINA UNIVERSITY   (United States)

^b UNIVERSITY OF MISSOURI   (United States)

^c UNIVERSITY OF CONNECTICUT   (United States)

^d UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Behavior assessment; Diagnostic accuracy; Direct behavior rating

Indexed keywords

CHILD; CHILD BEHAVIOR; CHILD BEHAVIOR DISORDERS; FEMALE; HUMAN; MALE; MASS SCREENING; PSYCHOLOGY; SCHOOL; SENSITIVITY AND SPECIFICITY; STUDENT; VALIDATION STUDY;

CHILD; CHILD BEHAVIOR; CHILD BEHAVIOR DISORDERS; FEMALE; HUMANS; MALE; MASS SCREENING; SCHOOLS; SENSITIVITY AND SPECIFICITY; STUDENTS;

EID: 84894905035     PISSN: 00224405     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jsp.2013.11.002     Document Type: Article

References (110)

1. Achenbach T.M., Rescorla L.A. Manual for the ASEBA school-age forms and profiles 2001, University of Vermont, Research Center for Children, Youth, and Families, Burlington, VT.
2. Alexander K.L., Entwisle D.R., Dauber S.L. First-grade classroom behavior: Long-term consequences for school performance. Child Development 1993, 64:801-814.
3. Breslau J., Miller E., Breslau N., Bohnert K., Lucia V., Schweitzer J. The impact of early behavior disturbances on academic achievement in high school. Pediatrics 2009, 123:1472-1476.
4. Briesch A.M., Chafouleas S.M., Riley-Tillman T.C. Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review 2010, 39:408-421.
5. Briesch A.M., Kilgus S.P., Chafouleas S.M., Riley-Tillman T.C., Christ T.J. The influence of alternative scale formats on the generalizability of data obtained from Direct Behavior Rating Single-Item Scales (DBR-SIS). Assessment for Effective Intervention 2013, 38:127-133.
6. Bryan T., Bryan J. Social interactions of learning disabled children. Learning Disability Quarterly 1978, 1:33-38.
7. Burt K.B., Roisman G.I. Competence and psychopathology: Cascade effects in the NICHD Study of Early Child Care and Youth Development. Development and Psychopathology 2010, 22:557-567.
8. Caldarella P., Merrell K.W. Common dimensions of social skills of children and adolescents: A taxonomy of positive behaviors. School Psychology Review 1997, 26:265-279.
9. Campbell N., Dobson J., Bost J. Educator perceptions of behavior problems of mainstreamed students. Exceptional Children 1985, 51:298-303.
10. Carran D.T., Scott K.G. Risk assessment in preschool children: Research implications for the early detection of educational handicaps. Topics in Early Childhood Special Education 1992, 12(2):196-211.
11. Catts H.W., Petscher Y., Schatschneider C., Bridges M.S., Mendoza K. Floor effects associated with universal screening and their impact on the early identification of reading disabilities. Journal of Learning Disabilities 2009, 42:163-176.
12. Chafouleas S.M. Direct Behavior Rating: A review of the issues and research in its development. Education & Treatment of Children 2011, 34:575-591.
13. Chafouleas S.M., Briesch A.M., Riley-Tillman T.C., Christ T.J., Black A.C., Kilgus S.P. An investigation of the generalizability and dependability of Direct Behavior Ratings (DBR-SIS) to measure engagement and disruptive behavior of middle school students. Journal of School Psychology 2010, 48:219-246.
14. Chafouleas S.M., Christ T.J., Riley-Tillman T.C. Generalizability of scaling gradients on Direct Behavior Ratings. Educational and Psychological Measurement 2009, 69:157-173.
15. Chafouleas S.M., Christ T., Riley-Tillman T.C., Briesch A.M., Chanese J. Generalizability and dependability of Direct Behavior Ratings to measure social behavior of preschoolers. School Psychology Review 2007, 36:63-79.
16. Chafouleas S.M., Kilgus S.P., Hernandez P. Using Direct Behavior Rating (DBR) to screen for school social risk: A preliminary comparison of methods in a kindergarten sample. Assessment for Effective Intervention 2009, 34:214-223.
17. Chafouleas S.M., Kilgus S.P., Jaffery R., Riley-Tillman T.C., Welsh M.E., Christ T.J. Direct Behavior Rating as a school-based behavior screener for elementary and middle grades. Journal of School Psychology 2013, 51:367-385.
18. Chafouleas S.M., Kilgus S.P., Riley-Tillman T.C., Jaffery R., Harrison S. Preliminary evaluation of various training components on accuracy of Direct Behavior Ratings. Journal of School Psychology 2012, 50:317-334.
19. Chafouleas S.M., McDougal J.L., Riley-Tillman T.C., Panahon C.J., Hilt A.M. What do daily behavior report cards (DBRCs) measure? An initial comparison of DBRCs with direct observation for on-task behavior. Psychology in the Schools 2005, 42:669-676.
20. Chafouleas S.M., Riley-Tillman T.C., Christ T.J. Direct Behavior Rating (DBR): An emerging method for assessing social behavior within a tiered intervention System. Assessment for Effective Intervention 2009, 34:195-200.
21. Chafouleas S.M., Riley-Tillman T.C., Jaffery R., Miller F.G., Harrison S.E. Preliminary investigation of the impact of a web-based module on Direct Behavior Rating accuracy 2013, (under review).
22. Chafouleas S.M., Riley-Tillman T.C., Sassu K.A., LaFrance M.J., Patwa S.S. Daily behavior report cards: An investigation of the consistency of on-task data across raters and methods. Journal of Positive Behavior Interventions 2007, 9:30-37.
23. Chafouleas S.M., Sanetti L.M.H., Kilgus S.P., Maggin D.M. Evaluating sensitivity to behavioral change using Direct Behavior Rating Single-Item Scales. Exceptional Children 2012, 78:491-505.
24. Chafouleas S.M., Volpe R.J., Gresham F.M., Cook C. School-based behavioral assessment within problem-solving models: Current status and future directions. School Psychology Review 2010, 34:343-349.
25. Chin J.K., Dowdy E., Quirk M.P. Universal screening in the middle school: Examining the behavioral and emotional screening system. Journal of Psychoeducational Assessment 2012, 10.1177/0734282912448137.
26. Christ T.J., Riley-Tillman T.C., Chafouleas S.M. Foundation for the development and use of Direct Behavior Rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention 2009, 34:201-213.
27. Christ T.J., Riley-Tillman T.C., Chafouleas S.M., Boice C.H. Generalizability and dependability of Direct Behavior Ratings (DBR) across raters and observations. Educational and Psychological Measurement 2010, 70:825-843.
28. Coie J., Dodge K. Continuities and changes in children's social status: A five-year longitudinal study. Merrill-Palmer Quarterly 1983, 29:261-281.
29. Coie J., Dodge K. Multiple sources of data on social behavior and social status in the school: A cross age comparison. Child Development 1988, 59:815-829.
30. Compton D.L., Fuchs D., Fuchs L.S., Bryant J.D. Selecting at-risk readers in first grade for early intervention: A two-year longitudinal study of decision rules and procedures. Journal of Educational Psychology 2006, 98:394-409.
31. Conners C.K. Conners' rating scales-revised 1997, Multi-Health Systems, North Tonawanda, NY.
32. Connolly J. A review of sociometric procedures in the assessment of social competencies in children. Applied Research in Mental Retardation 1983, 4:315-327.
33. Coopersmith J. Characteristics of public, private, and bureau of Indian education elementary and secondary school teachers in the United States: Results from the 2007-08 schools and staffing survey (NCES 2009-324) 2009, National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education, Washington DC.
34. Crocker L., Algina J. Introduction to classical and modern test theory 2006, Wadsworth, Pacific Grove, CA. 2nd ed.
35. Deno S.L., Fuchs L.S., Maston D., Shin J. Using curriculum-based measurement to establish growth standards for students with learning disabilities. School Psychology Review 2001, 30:507-524.
36. Dever B.V., Mays K.L., Kamphaus R.W., Dowdy E. The factor structure of the BASC-2 behavioral and emotional screening system teacher form, child/adolescent. Journal of psychoeducational assessment 2012, 30:488-495.
37. DiStefano C., Greer F.W., Kamphaus R.W. Multifactor modeling of emotional and behavioral risk of preschool-age children. Psychological assessment 2013, 10.1037/a0031393.
38. DiStefano C., Morgan G.B. Evaluation of the BESS TRS-CA using the Rasch rating scale model. School Psychology Quarterly 2010, 25:202-212.
39. DiStefano C., Morgan G. Examining classification criteria: A comparison of three cut score methods. Psychological Assessment 2011, 23:354-363.
40. Dodge K. Behavioral antecedents of peer social status. Children Development 1983, 54:1386-1399.
41. Drummond T. The student risk screening scale (SRSS) 1994, Josephine County Mental Health Program, Grants Pass, OR.
42. Duncan G.J., Dowsett C.J., Claessens A., Magnuson K., Huston A.C., Klebanov P., et al. School readiness and later achievement. Development Psychology 2007, 43:1428-1446.
43. Elliott S.N., Gresham F.M. Teacher and self-ratings of popular and rejected adolescent boys' behavior. Journal of Psychoeducational Assessment 1989, 7:323-333.
44. Elliott S.N., Gresham F.M. SSiS: Social Skills Improvement System 2008, Pearson, Minneapolis, MN.
45. Enders C.K. Applied missing data analysis 2010, Guilford, New York, NY.
46. Forstmeier S., Maercker A. Comparison of two diagnostic systems for complicated grief. Journal of Affect Disorders 2007, 99:203-211. 10.1016/j.jad.2006.09.013.
47. Fuchs L.S., Deno S.L. Paradigmatic distinctions between instructionally relevant measurement models. Exceptional Children 1991, 57:488-500.
48. Glover T.A., Albers C.A. Considerations for evaluating universal screening assessments. Journal of School Psychology 2007, 45:117-135.
49. Goodman R. The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry 1997, 38:581-586.
50. Gresham F.M. Social skills training with handicapped children: A review. Review of Educational Research 1981, 51:139-176.
51. Gresham F.M. Data-based decision making for students' social behavior. Journal of Evidence-Based Practices for Schools 2010, 11:149-168.
52. Gresham F.M., Elliott S.N. Social Skills Rating System 1990, Pearson Assessments, Bloomington, MN.
53. Gresham F.M., Elliott S.N. Social skills improvement system: Rating scales 2008, Pearson, Bloomington, MN.
54. Hintze J.M., Owen S.V., Shapiro E.S., Daly E.J. Generalizability of oral reading fluency measures: Application of G theory to curriculum-based measurement. School Psychology Quarterly 2000, 15:52-68.
55. Hintze J.M., Silberglitt B. A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high-stakes testing. School Psychology Review 2005, 34:372-386.
56. Hosp J.L., Hosp M.A., Dole J.K. Potential bias in predictive validity of universal screening measures across disaggregation groups. School Psychology Review 2011, 40(1):108-131.
57. Janssen K.J., Donders A.R., Harrell F.E., Vergouwe Y., Chen Q., Grobbee D.E., et al. Missing covariate data in medical research: To impute is better than to ignore. Journal of Clinical Epidemiology 2010, 63:721-727.
58. Jenkins J.R., Hudson R.F., Johnson E.S. Screening for at-risk readers in a response to intervention framework. School Psychology Review 2007, 36:582-600.
59. Juechter J.I., Dever B.V., Kamphaus R.W. Mental health screening and academic outcomes in elementary school. School Psychology Forum 2012, 6:137-147.
60. Kamphaus R.W., DiStefano C., Dowdy E., Eklund K., Dunn A.R. Determining the presence of a problem: Comparing two approaches for predicting youth behavioral risk. School Psychology Review 2010, 39:395-407.
61. Kamphaus R.W., Reynolds C.R. BASC-2 behavioral and emotional screening system 2007, Pearson, Minneapolis, MN.
62. Kamphaus R.W., Thorpe J.S., Winsor A.P., Kroncke A.P., Dowdy E.T., VanDeventer M.C. Development and predictive validity of a teacher screener for child behavioral and emotional problems at school. Educational and Psychological Measurement 2007, 67:342-356.
63. Kerr M.M., Zigmond N. What do high school teachers want? A study of expectations and standards. Education and Treatment of Children 1986, 9:239-249.
64. Kilgus S.P., Chafouleas S.M., Riley-Tillman T.C. Development and initial validation of the Social and Academic Behavior Risk Screener for elementary grades. School Psychology Quarterly 2013, 28:210-226. 10.1037/spq0000024.
65. Kilgus S.P., Chafouleas S.M., Riley-Tillman T.C., Welsh M.E. Direct Behavior Rating scales as screeners: A preliminary investigation of diagnostic accuracy in elementary school. School Psychology Quarterly 2012, 27:41-50.
66. King K., Reschly A.L., Appleton J.L. An examination of the validity of the Behavioral and Emotional Screening System in a rural elementary school: Validity of the BESS. Journal of Psychoeducational Assessment 2012, 10.1177/0734282912440673.
67. Lane K.L., Bruhn A.L., Eisner S.L., Kalberg J.R. Score reliability and validity of the Student Risk Screening Scale: A psychometrically sound, feasible tool for use in urban middle schools. Journal of Emotional and Behavioral Disorders 2010, 18:211-224.
68. Lane K.L., Givner M.R., Pierson C.C. Teacher expectations of student behavior: Which skills do elementary and secondary teachers deem necessary for success in the classroom?. Education and Treatment of Children 2003, 26:413-430.
69. Lane K.L., Kalberg J.R., Lambert E.W., Crnobori M., Bruhn A.L. A comparison of systematic screening tools for emotional and behavioral disorders: A replication. Journal of Emotional and Behavioral Disorders 2010, 18:100-112.
70. Lane K.L., Kalberg J.R., Parks R.J., Carter E.W. Student Risk Screening Scale: Initial evidence for score reliability and validity at the high school level. Journal of Emotional and Behavioral Disorders 2008, 16:178-190.
71. Lane K.L., Little M.A., Casey A.M., Lambert W., Wehby J., Weisenbach J.L., et al. A comparison of systematic screening tools for emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders 2009, 17:93-105.
72. Lane K.L., Oakes W.P., Harris P.J., Menzies H.M., Cox M., Lambert W. Initial evidence for the reliability and validity of the Student Risk Screening Scale for internalizing and externalizing behaviors at the elementary level. Behavioral Disorders 2012, 37:99-122.
73. Lane K.L., Parks R.J., Kalberg J.R., Carter E.W. Systematic screening at the middle school level: Score reliability and validity of the Student Risk Screening Scale. Journal of Emotional and Behavioral Disorders 2007, 15:209-222.
74. Lasko T.A., Bhagwat J.G., Zou K.H., Ohno-Machado L. The use of receiver operating characteristic curves in biomedical informatics. Journal of Biomedical Informatics 2005, 38:404-415.
75. LeBel T.J., Chafouleas S.M., Sugai G. Project VIABLE technical report for Direct Behavior Rating Scales: Rationale for selection of behaviors to be investigated 2006, University of Connecticut, Storrs, CT.
76. Levitt J.M., Saka N., Romanelli L.H., Hoagwood K. Early identification of mental health problems in schools: The status of instrumentation. Journal of School Psychology 2007, 45:163-191. 10.1016/j.jsp.2006.11.005.
77. Lipsey M.W., Puzio K., Yun C., Hebert M.A., Steinka-Fry K., Cole M.W., et al. Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000) 2012, U.S. Government Printing Office, Washington, DC, (http://ies.ed.gov/ncser/pubs/20133000/).
78. Masten A.S., Roisman G.I., Long J.D., Burt K.B., Obradović J., Riley J.R., et al. Developmental cascades: Linking academic achievement and externalizing and internalizing symptoms over 20years. Developmental Psychology 2005, 41:733-746.
79. McFall R. A review and reformulation of the concept of social skills. Behavioral Assessment 1982, 4:1-33.
80. Menzies H.M., Lane K.L. Validity of the Student Risk Screening Scale: Evidence of predictive validity in a diverse, suburban elementary setting. Journal of Emotional and Behavioral Disorders 2012, 20:82-91.
81. Metz C.E. Basic principles of ROC analysis. Seminars in Nuclear Medicine 1978, 8:283-298.
82. National Center for Education Statistics, U.S.Department of Education Schools and staffing survey (SASS): Public school teacher, BIE school teacher, and private school teacher data files (Retrieved August 14, 2013 from). http://nces.ed.gov/surveys/sass/tables/sass0708_029_t12n.asp.
83. National Center for Education Statistics, U.S.Department of Education Schools and staffing survey (SASS): Public teacher questionnaire (Retrieved August 14, 2013, from). http://nces.ed.gov/programs/digest/d12/tables/dt12_079.asp.
84. National Center for Education Statistics, U.S.Department of Education Public elementary/secondary school universe survey Retrieved August 14, 2013, from. http://nces.ed.gov/programs/stateprofiles/.
85. National Research Council and Institute of Medicine Pre- venting mental, emotional, and behavioral disorders among young people: Progress and possibilities 2009, National Academies Press, Washington, DC. M.E. O'Connell, T. Boat, K.E. Warner (Eds.).
86. Oakes W.P., Wilder K.S., Lane K.L., Powers L., Yokoyama L.T.K., O'Hare M.E., et al. Psychometric properties of the Student Risk Screening Scale: An effective tool for use in diverse urban elementary schools. Assessment for Effective Intervention 2010, 35:231-239.
87. Obradović J., Burt K.B., Masten A.S. Testing a dual cascade model linking competence and symptoms over 20years from childhood to adulthood. Journal of Clinical Child & Adolescent Psychology 2010, 39:90-102.
88. Parker J., Asher S. Peer relations and later personal adjustment: Are low accepted children at risk?. Psychological Bulletin 1987, 102:352-390.
89. Patterson G.R., Reid J.B., Dishion T.J. A social interactional approach. Antisocial boys 1992, Vol. 4. Eugene, OR: Castaglia.
90. Petscher Y., Kim Y., Foorman B.R. The importance of predictive power in early screening assessments: Implications for placement in the response to intervention framework. Assessment for Effective Intervention 2011, 36:158-166.
91. Peugh J.L., Enders C.K. Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research 2004, 74:525-556.
92. Renshaw T.L., Eklund K., Dowdy E., Jimerson S.R., Hart S.R., Earhart J., et al. Examining the relationship between scores on the behavioral and emotional screening system and student academic, behavioral, and engagement outcomes: An investigation of concurrent validity in elementary school. The California School Psychologist 2009, 14:81-88.
93. Reynolds C.R., Kamphaus R.W. Behavior assessment system for children-Second edition (BASC-2) 2004, AGS, Circle Pines, MN.
94. Riley-Tillman T.C., Chafouleas S.M., Briesch A.M., Eckert T.L. Daily behavior report cards and systematic direct observation: An investigation of the acceptability, reported training and use, and decision reliability among school psychologists. Journal of Behavioral Education 2008, 17:313-327.
95. Riley-Tillman T.C., Chafouleas S.M., Christ T.J., Briesch A.M., LeBel T.J. The impact of wording and behavioral specificity on the accuracy of Direct Behavior Ratings (DBRs). School Psychology Quarterly 2009, 24:1-12.
96. Riley-Tillman T.C., Chafouleas S.M., Sassu K.A., Chanese J.M., Glazer A.D. Examining agreement between Direct Behavior Ratings (DBRs) and systematic direct observation data for on-task and disruptive behavior. Journal of Positive Behavior Interventions 2008, 10:136-143.
97. Riley-Tillman T.C., Christ T.J., Chafouleas S.M., Boice C.H., Briesch A.M. DBR observation duration and scaling anchoring: Effects on accuracy of ratings. Journal of Positive Behavior Interventions 2010, 13:119-128.
98. Romer D., McIntosh M. The roles and perspectives of school mental health professionals in promoting adolescent mental health. Treating and preventing adolescent mental health disorders: What we know and what we don't know 2005, 598-615. Oxford University Press, New York, NY. D. Evans, E. Foa, R. Gur, H. Hendin, C. O'Brien, M. Seligman, B. Walsh (Eds.).
99. Rusby J.C., Taylor T.K., Foster E.M. A descriptive study of school discipline referrals in first grade. Psychology in the Schools 2007, 44:333-350.
100. Severson H.H., Walker H.M., Hope-Doolittle J., Kratochwill T.R., Gresham F.M. Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology 2007, 45:193-223.
101. Shapiro E.S. Academic skills problems: Direct assessment and intervention 2004, Guilford, New York, NY. 3rd ed.
102. Shinn M.R. Curriculum-based measurement: Assessing special children 1989, Guilford, New York, NY.
103. Spaulding S.A., Irvin L.K., Horner R.H., May S.L., Emeldi M., Tobin T.J., et al. Schoolwide social-behavioral climate, student problem behavior, and related administrative decisions: Empirical patterns from 1510 schools nationwide. Journal of Positive Behavior Interventions 2010, 12:69-85.
104. Streiner D.L. Clinimetrics vs. psychometrics: An unnecessary distinction. Journal of Clinical Epidemiology 2003, 56:1142-1145.
105. Streiner D.L., Cairney J. What's under the ROC? An introduction to receiver operating characteristic curves. The Canadian Journal of Psychiatry/La Revue canadienne de psychiatrie 2007, 52:121-128.
106. United States Department of Health and Human Services Mental health: A report of the surgeon general 1999, U.S. Department of Health and Human Services, Washington, D.C.
107. Walker H.M., Irvin L.K., Noell J., Singer G.H.S. A construct score approach to the assessment of social competence: Rationale, technological considerations, and anticipated outcomes. Behavior Modification 1992, 16:448-474.
108. Walker H.M., Ramsey E., Gresham F.M. Heading off disruptive behavior: How early intervention can reduce defiant behavior and win back teaching time. American Education 2003, Winter, 6-28.
109. Walker H.M., Rankin R. Assessing the behavioral expectations and demands of less restrictive settings. School Psychology Review 1983, 12:274-284.
110. Walker H.M., Severson H. Systematic screening of behavior disorders (SSBD): A multiple gating procedure 1992, Sopris West, Longmont, CO.