An analysis of the ratings and interrater reliability of high school band contests

Journal of Research in Music Education

Volumn 60, Issue 1, 2012, Pages 81-100

  Hash, Phillip M ^a  

^a CALVIN COLLEGE   (United States)

Author keywords

assessment; band; contests; festivals; interrater reliability

Indexed keywords

EID: 84858677287     PISSN: 00224294     EISSN: None     Source Type: Journal    
DOI: 10.1177/0022429411434932     Document Type: Article

References (67)

1. Adler, S. A., & Clark, R. (2008). How it's done: An invitation to social research (3rd ed.). Belmont, CA: Thompson Wadsworth. American Recovery and Reinvestment Act, Section 14005-6, Title XIV, Public Law 111-5 (2009).
2. Baker, V. (2004). The effect of repertoire selection on university interscholastic league choral concert ratings. Texas Music Education Research. Retrieved from http://www.tmea.org/assets/pdf/research/Bak2004.pdf
3. Barnes, G. V., & McCashin, R. (2005). Practices and procedures in state adjudicated orchestra festivals. Update: Applications of Research in Music Education, 23(2), 34-41. doi:10.1177/87551233050230020105
4. Batey, A. L. (2002). Take the terror out of adjudication. Teaching Music, 10(3), 40-46.
5. Bergee, M. J. (2003). Faculty interjudge reliability of music performance evaluation. Journal of Research in Music Education, 51, 137-150. doi:10.2307/3345847
6. Bergee, M. J. (2007). Performer, rater, occasion, and sequence as sources of variability in music performance assessment. Journal of Research in Music Education, 55, 344-358. doi:10.1177/0022429408317515
7. Bergee, M. J., & McWhirter, J. L. (2005). Selected influences on solo and small-ensemble festival ratings: Replication and extension. Journal of Research in Music Education, 53, 177-190. doi:10.1177/002242940505300207
8. Bergee, M. J., & Platt, M. C. (2003). Influence of selected variables on solo and small-ensemble festival ratings. Journal of Research in Music Education, 51, 342-353. doi:10.2307/3345660
9. Bergee, M. J., & Westfall, C. R. (2005). Stability of a model explaining selected extramusical influences on solo and small-ensemble festival ratings. Journal of Research in Music Education, 53, 358-374. doi:10.1177/002242940505300407
10. Boeckman, J. (2002). Grade inflation in band contest ratings: A trend study. Journal of Band Research, 38(1), 25-36.
11. Boyle, J. D. (1992). Program evaluation for secondary school music programs. NASSP Bulletin, 76(544), 63-68.
12. Brakel, T. D. (2006). Inter-judge reliability of the Indiana State School Music Association high school instrumental festival. Journal of Band Research, 42(1), 59-69.
13. Brown, J. D. (2001). Can we use the Spearman-Brown prophecy formula to defend low reliability? Shiken: JALT Testing and Evaluation SIG Newsletter, 4(3), 4-7. Retrieved from http://jalt.org/test/bro-9.htm
14. Burnsed, V., Hinkle, D., & King, S. (1985). Performance evaluation reliability at selected concert festivals. Journal of Band Research, 21(1), 22-29.
15. Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Beverly Hills, CA: Sage.
16. Cassidy, J. W., & Sims, W. L. (1991). Effects of special education labels on peers' and adults' evaluations of a handicapped youth choir. Journal of Research in Music Education, 39, 23-34. doi:10.2307/3344606
17. Cavitt, M. E. (1997). Effects of expectations on evaluators' judgments of music performance. Texas Music Education Research. Retrieved from http://www.tmea.org/assets/pdf/research/Cav1997.pdf
18. Cavitt, M. E. (2002). Differential expectation effects as factors in evaluations and feedback of musical performance. Texas Music Education Research. Retrieved from http://www.tmea .org/assets/pdf/research/Cav2002.pdf
19. Ciorba, C. R., & Smith, N. Y. (2009). Juries using a multidimensional assessment rubric measurement of instrumental and vocal undergraduate performance juries. Journal of Research in Music Education, 57, 5-15. doi:10.1177/0022429409333405
20. Conrad, D. (2003). Judging the judges: Improving rater reliability at music contests. NFHS Music Association Journal, 20(2), 27-31
21. Elliott, C. A. (1995/1996). Race and gender as factors in judgments of musical performance. Bulletin of the Council for Research in Music Education, 127, 50-55.
22. Ellis, M. C. (2007). An analysis of taped comments from a high school jazz band festival. Contributions to Music Education, 34, 35-49.
23. Fautley, M. (2010). Assessment in music education. New York, NY: Oxford University Press.
24. Fiske, H. E. (1975). Judge-group differences in the rating of secondary school trumpet performances. Journal of Research in Music Education, 23, 186-189. doi:10.2307/3344643
25. Fiske, H. E. (1978). The effect of a training procedure in music performance evaluation on judge reliability. Ontario Educational Research Council Report, Canada.
26. Fiske, H. E. (1983). Judging musical performance: Method or madness? Update: Applications of Research in Music Education, 1(3), 7-10.
27. Flores, R. G., & Ginsburgh, V. A. (1996). The Queen Elisabeth musical competition: How fair is the final ranking? The Statistician, 45, 97-104.
28. Forbes, G. W. (1994). Evaluative music festivals and contests-Are they fair? Update: Applications of Research in Music Education, 12(2), 16-20. doi:10.1177/875512339401200203
29. Freelon, D. (2009). ReCal: Reliability calculation for the masses [Online calculator]. Retrieved from http://dfreelon.org/utils/recalfront/
30. Garman, B. R., Boyle, J. D., & DeCarbo, N. J. (1991). Orchestra festival evaluations: Interjudge agreement and relationships between performance categories and final ratings. Research Perspectives in Music Education, 2, 19-24.
31. Hassel, B. E., & Hassel, E. A. (2009, December). Race to the top: Accelerating college and career readiness in states. Retrieved from http://www.achieve.org/RTTT-TeacherEffectiveness
32. Hunter, D., & Russ, M. (1996). Peer assessment in performance studies. British Journal of Music Education, 13, 67-78.
33. Illinois High School Association. (2009). 2009-2010 music rule book. Bloomington, IL: Author.
34. Killian J. N. (1998). Characteristics of successful choirs in a contest setting. Texas Music Education Research, 39-43. Retrieved from http://www.tmea.org/assets/pdf/research/Kil1998.pdf
35. Killian J. (1999). Music selection of successful choirs at UIL and non-UIL contests. Texas Music Education Research, 51-56. Retrieved from http://www.tmea.org/assets/pdf/research/Kil1999.pdf
36. Killian, J. (2000). Effect of music selection on contest ratings: Year three of a continuing study. Texas Music Education Research. Retrieved from http://www.tmea.org/assets/pdf/research/Kil2000.pdf
37. King, S. E., & Burnsed, V. (2009). A study of the reliability of adjudicator ratings at the 2005 Virginia band and orchestra directors association state marching band festivals. Journal of Band Research, 45(1), 27-32.
38. Kinney, D. W. (2009). Internal consistency of performance evaluations as a function of music expertise and excerpt familiarity. Journal of Research in Music Education, 56, 322-337. doi:10.1177/0022429408328934
39. Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA: Sage.
40. Latimer, M. E., Bergee, M. J., & Cohen, M. L. (2010). Performance assessment rubric reliability and perceived pedagogical utility of a weighted music performance assessment rubric. Journal of Research in Music Education, 58, 168-183. doi:10.1177/0022429410369836
41. Lee, K. (2009, December 10). Districts cautious with state's "race to the top." Illinois Statehouse News. Retrieved from http://illinois. statehousenewsonline.com/426/districts-cautious-withstate" race-to-the-top"
42. McPherson, G. E., & Thompson, W. F. (1998). Assessing music performance: Issues and influences. Research Studies in Music Education, 10, 12-24. doi:10.1177/1321103X9801000102
43. MENC: The National Association for Music Education. (n.d.). [Band adjudication form]. Retrieved from http://www.menc.org/documents/nmac/addforms/ band.pdf
44. Moore, J. E. (1968). National school band contests between 1926 and 1931 (Unpublished doctorial dissertation). University of Michigan, Ann Arbor.
45. Morrison, S. J., Price, H. E., Geiger, C. G., & Cornacchio, R. A. (2009). The effect of conductor expressivity on ensemble performance evaluation. Journal of Research in Music Education, 57, 37-49. doi:10.1177/0022429409332679
46. Music in the Parks. (2010). Festival performance guidelines. Retrieved from http://www.festivals ofmusic.net/ext/MIP/DisplayFile.aspx?pfil=82
47. National Association for Music Education. (2011). NAfME position statement on teacher evaluation. Retrieved from http://www.menc.org/about/view/ teacher-evaluation.
48. Norris, C. E., & Borst, J. D. (2007). An examination of the reliabilities of two choral festival adjudication forms. Journal of Research in Music Education, 55, 237-251. doi:10.1177/002242940705500305
49. Orman, E. K., Yarbrough, C., Neill, S., & Whitaker, J. A. (2007). Time usage of middle and high school band directors in sight-reading adjudication. Update: Applications of Research in Music Education, 25(2), 36-46. doi:10.1177/8755123307250020105
50. Phillips, K. H. (2008). Exploring research in music education and music therapy. New York, NY: Oxford University Press.
51. Presley, D. (n.d.). The mentor project. Retrieved from http://www.scmea.net/mentoring.htm
52. Price, H. E., & Chang, E. C. (2005). Conductor and ensemble performance expressivity and state festival ratings. Journal of Research in Music Education, 53, 66-77. doi:10.1177/002242940505300106
53. Radocy, R. E. 1976 Effects of authority figure biases on changing judgments of musical events Journal of Research in Music Education 24 119-128 doi:10.2307/3345155
54. Rickels D. A. (2009). A multivariate analysis of nonperformance variables as predictors of marching band contest results (Doctoral dissertation). Retrieved from Dissertations and Theses database. (UMI No. 3353883)
55. Rohrer, T. P. 2002 The debate on competition in music in the twentieth century Update: Applications of Research in Music Education 21(1) 38-47 doi:10.1177/875512330202 10010501
56. Saunders, T. C., & Holahan, J. M. 1997 Criteria-specific rating scales in the evaluation of high school instrumental performance Journal of Research in Music Education 45 259-272 doi:10.2307/3345585
57. Silver, N C. & Dunlap W P. 1987 Averaging correlation coefficients: Should Fischer's z transformation by used? Journal of Applied Psychology, 72, 146-148
58. Silvey, B. A. 2009 The effects of band labels on evaluators' judgments of musical performance Update: Applications of Research in Music Education 28(1) 47-52 doi:10.1177/8755123309344111
59. South Carolina Band Directors Association. (2009). SCBDA handbook. Author. South Carolina Department of Education. (2010). Quick facts: Education of South Carolina. Columbia SC: Author
60. Stamer, R. A. 2006 Changes in choral student perceptions of the music contest experience Update: Applications of Research in Music Education 25(1) 46-55 doi:10.1177/875512 33060250010106
61. Stemler, S. E. 2004 A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability Practical Assessment Research and Evaluation 9(4) Retrieved from http://PAREonline.net/getvn.asp?v= 9&n=4
62. Sullivan T. M. (2003). Factors influencing participation of Arizona high school marching bands in regional and state festivals (Doctorial dissertation). Retrieved from Dissertations and Theses database. (UMI No. 3080892)
63. University Interscholastic League. (2010). Constitution and contest rules. Austin, TX: Author. Retrieved from http://www.uil.utexas.edu/policy/ constitution/index.html
64. VanWeelden, K., & McGee, I. R. (2007). The influence of music style and conductor race on perceptions of ensemble and conductor performance. International Journal of Music Education, 25, 7-17. doi:10.1177/0255761407074886
65. Williams, J. (2009, December 14). Who would have guesses that the race would look like this? [Web log post] Retrieved from http://www.dfer.org/2009/12/ who-would-have.php
66. Winter, N. (1993). Music performance assessment: A study of the effects of training and experience on the criteria used by music examiners. International Journal of Music Education, 22, 34-39.
67. Yarbrough, C., Orman, E. K., & Neill, S. (2007). Time usage by choral directors prior to sightsinging adjudication. Update: Applications of Research in Music Education, 25(1), 27-35.