Preliminary evaluation of various training components on accuracy of Direct Behavior Ratings

Journal of School Psychology

Volumn 50, Issue 3, 2012, Pages 317-334

  Chafouleas, Sandra M ^a   Kilgus, Stephen P ^a   Riley Tillman, T Chris ^c   Jaffery, Rose ^a   Harrison, Sayward ^b  

^a UNIVERSITY OF CONNECTICUT   (United States)

^b EAST CAROLINA UNIVERSITY   (United States)

^c UNIVERSITY OF MISSOURI   (United States)

Author keywords

Behavior assessment; Direct Behavior Rating; Formative assessment; Rating scales

Indexed keywords

ARTICLE; BEHAVIOR DISORDER; CHILD; CHILD BEHAVIOR; CLASSIFICATION; CLINICAL TRIAL; CONTROLLED CLINICAL TRIAL; CONTROLLED STUDY; EDUCATION; HUMAN; OBSERVER VARIATION; PSYCHOLOGICAL RATING SCALE; RANDOMIZED CONTROLLED TRIAL; REFERENCE VALUE; STUDENT;

CHILD; CHILD BEHAVIOR; CHILD BEHAVIOR DISORDERS; EDUCATION; HUMANS; OBSERVER VARIATION; PSYCHIATRIC STATUS RATING SCALES; REFERENCE VALUES; STUDENTS;

EID: 84861738886     PISSN: 00224405     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jsp.2011.11.007     Document Type: Article

References (49)

1. Angkaw A.C., Tran G.Q., Haaga D.A.F. Effects of training intensity on observers' ratings of anxiety, social skills, and alcohol-specific coping skills. Behaviour Research and Therapy 2006, 44:533-544.
2. Bernardin H.J., Buckley M.R. Strategies in rater training. Academy of Management Review 1981, 6:205-212.
3. Borman W.C. Consistency of rating accuracy and rater errors in the judgment of human performance. Organizational Behavior and Human Performance 1977, 20:238-252. (This journal is now called Organizational Behavior and Human Decision Processes.).
4. Chafouleas S.M. Direct Behavior Rating: A review of the issues and research in its development. Education and Treatment of Children 2011, 34:575-591.
5. Chafouleas S.M., Briesch A.M., Riley-Tillman T.C., Christ T.J., Black A.C., Kilgus S.P. An investigation of the generalizability and dependability of Direct Behavior Rating Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students. Journal of School Psychology 2010, 48:219-246.
6. Chafouleas S.M., Christ T.J., Riley-Tillman T.C., Briesch A.M., Chanese J.M. Generalizability and dependability of Daily Behavior Report Cards to assess social behavior of preschoolers. School Psychology Review 2007, 36:63-79.
7. Chafouleas S.M., McDougal J.L., Riley-Tillman T.C., Panahon C.J., Hilt A.M. What do daily behavior report cards (DBRCs) measure? An initial comparison of DBRCs with direct observation for off-task behavior. Psychology in the Schools 2005, 42:669-676.
8. Chafouleas S.M., Riley-Tillman T.C., Christ T.J. Direct Behavior Rating (DBR): An emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention 2009, 34:195-200.
9. Chafouleas S.M., Riley-Tillman T.C., McDougal J. Good, bad, or in-between: How does the daily behavior report card rate?. Psychology in the Schools 2002, 39:157-169.
10. Chafouleas S.M., Riley-Tillman T.C., Sugai G. School-based behavioral assessment: Informing instruction and intervention 2007, Guilford, New York, NY.
11. Chafouleas S.M., Volpe R.J., Gresham F.M., Cook C. School-based behavioral assessment within problem-solving models: Current status and future directions. School Psychology Review 2010, 34:343-349.
12. Chafouleas, S. M., Riley-Tillman, T. C., Jaffery, R., Sen, R., & Christ, T. J. (Submitted for publication). Direct Behavior Rating Single Item Scales (DBR-SIS): An evaluation of the impact of target, wording, and base rate on rater accuracy for three general outcomes of student behavior.
13. Chi M.T.H. Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences 1997, 6:271-315.
14. Christ T.J., Riley-Tillman T.C., Chafouleas S.M. Foundation for the development and use of Direct Behavior Rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention 2009, 34:201-213.
15. Christ T.J., Riley-Tillman T.C., Chafouleas S.M., Boice C.H. Generalizability and dependability of Direct Behavior Ratings (DBR) across raters and observations. Educational and Psychological Measurement 2010, 70:825-843.
16. Christ T.J., Riley-Tillman T.C., Chafouleas S.M., Jaffery R. Direct Behavior Rating: An evaluation of alternate behaviors with positive and negative definitions. School Psychology Review 2011, 40:181-199.
17. Cronbach L.J. Processes affecting scores on "understanding of others" and "assumed similarity". Psychological Bulletin 1955, 52:177-193.
18. Cronbach L.J., Gleser G.C., Nanda H., Rajaratnam N. The dependability ofbehavior measurements: Theory of generalizability for scores and profiles 1972, Wiley, New York, NY.
19. Edelbrock C. Problems and issues in using rating scales to assess child personality and psychopathology. School Psychology Review 1983, 12:293-299.
20. Epstein M., Atkins M., Cullinan D., Kutash K., Weaver R. Reducing behavior problems in the elementary school classroom: A practice guide (NCEE #2008-012) 2008, Retrieved from, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, Washington, DC. http://ies.ed.give/ncee/wwc/publications/practiceguides.
21. Floyd R.G., Bose J.E. Behavior rating scales for assessment of emotional disturbance: A critical review of measurement characteristics. Journal of Psychoeducational Assessment 2003, 21:43-78.
22. Foster G.G., Salvia J. Teacher response to the label of learning disabled as a function of demand characteristics. Exceptional Children 1977, 43:533-534.
23. Foster G.G., Ysseldyke J.E., Reese J.H. I wouldn't have seen it if I hadn't believed it. Exceptional Children 1975, 41:469-473.
24. Guion R.M. Personnel testing 1965, Mc-Graw-Hill, New York, NY.
25. Harrison, S. E., Riley-Tillman, T. C., & Chafouleas, S. M. (Submitted for publication). Practice with feedback and base rates of target behavior: Implications for rater accuracy using Direct Behavior Ratings.
26. Hay L.R., Nelson R.O., Hay W.M. The use of teachers as behavioral observers. Journal of Applied Behavior Analysis 1977, 10:345-348.
27. LeBel T.J., Kilgus S.P., Briesch A.M., Chafouleas S.M. The impact of training on the accuracy of teacher-completed Direct Behavior Ratings (DBRs). Journal of Positive Behavior Interventions 2010, 12:55-63.
28. Marshburn, J. A. (2006). The relationship between teacher perception and daily behavior report cards. Unpublished master's thesis, East Carolina University, Greenville, North Carolina.
29. McIntyre R.M., Smith D.E., Hassett C.E. Accuracy of performance ratings as affected by rater training and perceived purpose of rating. Journal of Applied Psychology 1984, 69:147-156.
30. Riley-Tillman T.C., Chafouleas S.M., Christ T.J., Briesch A.M., LeBel T.J. The impact of wording and behavioral specificity on the accuracy of Direct Behavior Ratings (DBRs). School Psychology Quarterly 2009, 24:1-12.
31. Riley-Tillman T.C., Christ T.J., Chafouleas S.M., Boice-Mallach C.H., Briesch A.M. The impact of observation duration on the accuracy of data obtained from Direct Behavior Rating (DBR). Journal of Positive Behavior Interventions 2010, 13:119-128.
32. Romer D., McIntosh M. The roles and perspectives of school mental health professionals in promoting adolescent mental health. Treating and preventing adolescent mental health disorders: What we know and what we don't know 2005, 598-615. Oxford University Press, New York, NY. D. Evans, E. Foa, R. Gur, H. Hendin, C. O'Brien, M. Seligman, B. Walsh (Eds.).
33. Schlientz M.D., Riley-Tillman T.C., Briesch A.M., Walcott C.M., Chafouleas S.M. The impact of training on the accuracy of Direct Behavior Ratings (DBRs). School Psychology Quarterly 2009, 24:73-83.
34. Sonis W.A., Costello A.J. Evaluation of differential data sources: Application of the diagnostic process in child psychiatry. Journal of the American Academy of Child Psychiatry 1981, 22:375-392.
35. Spool M.D. Training programs for observers or behavior: A review. Personnel Psychology 1978, 31:853-888.
36. Stamoulis D.T., Hauenstein N.M.A. Rater training and rater accuracy: Training for dimensional accuracy versus training for rating differentiation. Journal of Applied Psychology 1993, 78:994-1003.
37. Sterling-Turner H.E., Watson T.S., Wildmon M., Watkins C., Little E. Investigating the relationship between training type and treatment integrity. School Psychology Quarterly 2001, 16:56-67.
38. Sulsky L.M., Balzer W.K. Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology 1988, 73:497-506.
39. Sulsky L.M., Day D.V. Frame-of-reference training and cognitive categorization: An empirical investigation of memory issues. Journal of Applied Psychology 1992, 77:501-510.
40. Sulsky L.M., Day D.V. Effects of frame-of-reference training on rater accuracy under alternative time delays. Journal of Applied Psychology 1994, 79:535-543.
41. Sulsky L.M., Skarlicki D.P., Keown J.L. Frame-of-reference training: Overcoming the effects of organizational citizenship behavior on performance rating accuracy. Journal of Applied Social Psychology 2002, 32:1224-1240.
42. Tabachnick B.G., Fidell L.S. Using multivariate statistics 2007, Allyn & Bacon, Boston, MA. 5th ed.
43. Tapp J. MOOSES (multi-option observation system for experimental studies) Retrieved from. http://kc.vanderbilt.edu/mooses/mooses.html.
44. Thornton G., Zorich S. Training to improve observer accuracy. Journal of Applied Psychology 1980, 65:351-354.
45. Vikberg, S. (2006). A comparison of students' daily behavior report cards to teachers' daily behavior report cards and systematic direct observations. Unpublished master's thesis, East Carolina University, Greenville, North Carolina.
46. Volpe R.J., DiPerna J.C., Hintze J.M., Shapiro E.S. Observing students in classroom settings: A review of seven coding schemes. School Psychology Review 2005, 34:454-474.
47. Wexley K.N., Youtz M.A. Rater beliefs about others: Their effects on rater errors and rater accuracy. Journal of Occupational Psychology 1985, 58:265-275.
48. Woehr D.J. Understanding frame-of-reference training: The impact of training on the recall of performance information. Journal of Applied Psychology 1994, 79:525-534.
49. Wolery M., Bailey D.B., Sugai G. Effective teaching: Principles and procedures of applied behavior analysis with exceptional students 1988, Allyn Bacon, Boston, MA.