Inter-expert and intra-expert reliability in sleep spindle scoring

Clinical Neurophysiology

Volumn 126, Issue 8, 2015, Pages 1548-1556

  Wendt, Sabrina L ^a,b   Welinder, Peter ^c   Sorensen, Helge B D ^d   Peppard, Paul E ^e   Jennum, Poul ^b   Perona, Pietro ^c   Mignot, Emmanuel ^a   Warby, Simon C ^a,f  

^a STANFORD UNIVERSITY   (United States)

^b GLOSTRUP UNIVERSITY HOSPITAL   (Denmark)

^c CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^d TECHNICAL UNIVERSITY OF DENMARK   (Denmark)

^e UNIVERSITY OF WISCONSIN SCHOOL OF MEDICINE AND PUBLIC HEALTH   (United States)

^f HÔPITAL DU SACRÉ COEUR DE MONTRÉAL   (Canada)

Author keywords

Agreement; Electroencephalography; Event detection; Inter expert; Inter rater; Intra expert; Intra rater; Polysomnography; Reliability; Sleep scoring; Sleep spindles; Sleep staging

Indexed keywords

ADULT; AGED; ARTICLE; CORRELATION COEFFICIENT; ELECTROENCEPHALOGRAM; FALSE NEGATIVE RESULT; FALSE POSITIVE RESULT; FEMALE; HUMAN; HUMAN EXPERIMENT; INTEREXPERT RELIABILITY; INTRAEXPERT RELIABILITY; MALE; NORMAL HUMAN; OSCILLATION; PRIORITY JOURNAL; RELIABILITY; SLEEP SPINDLE; SLEEP STAGE; STATISTICAL SIGNIFICANCE; AROUSAL; ELECTROENCEPHALOGRAPHY; MIDDLE AGED; OBSERVER VARIATION; PHYSIOLOGY; POLYSOMNOGRAPHY; PROCEDURES; REPRODUCIBILITY;

ADULT; AGED; AROUSAL; ELECTROENCEPHALOGRAPHY; FEMALE; HUMANS; MALE; MIDDLE AGED; OBSERVER VARIATION; POLYSOMNOGRAPHY; REPRODUCIBILITY OF RESULTS; SLEEP STAGES;

EID: 84929486830     PISSN: 13882457     EISSN: 18728952     Source Type: Journal    
DOI: 10.1016/j.clinph.2014.10.158     Document Type: Article

References (69)

1. Anderer P., Gruber G., Parapatics S., Woertz M., Miazhynskaia T., Klosch G., et al. An E-health solution for automatic sleep classification according to Rechtschaffen and Kales: validation study of the Somnolyzer 24×7 utilizing the Siesta database. Neuropsychobiology 2005, 51:115-133.
2. Bazhenov M., Timofeev I., Steriade M., Sejnowski T.J. Model of thalamocortical slow-wave sleep oscillations and transitions to activated states. J Neurosci 2002, 22:8691-8704.
3. Bergmann T.O., Molle M., Diedrichs J., Born J., Siebner H.R. Sleep spindle-related reactivation of category-specific cortical regions after learning face-scene associations. Neuroimage 2012, 59:2733-2742.
4. Bodizs R., Kis T., Lazar A.S., Havran L., Rigo P., Clemens Z., et al. Prediction of general mental ability based on neural oscillation measures of sleep. J Sleep Res 2005, 14:285-292.
5. Bonjean M., Baker T., Bazhenov M., Cash S., Halgren E., Sejnowski T. Interactions between core and matrix thalamocortical projections in human sleep spindle synchronization. J Neurosci 2012, 32:5250-5263.
6. Bonjean M., Baker T., Lemieux M., Timofeev I., Sejnowski T., Bazhenov M. Corticothalamic feedback controls sleep spindle duration in vivo. J Neurosci 2011, 31:9124-9134.
7. Bremer G., Smith J.R., Karacan I. Automatic detection of the K-complex in sleep electroencephalograms. IEEE Trans Biomed Eng 1970, 17:314-323.
8. Brown W. Some experimental results in the correlation of mental abilities. Br J Psychol 1910, 3:296-322.
9. Campbell K., Kumar A., Hofman W. Human and automatic validation of a phase-locked loop spindle detection system. Electroencephalogr Clin Neurophysiol 1980, 48:602-605.
10. Christensen J.A., Kempfner J., Zoetmulder M., Leonthin H.L., Arvastson L., Christensen S.R., et al. Decreased sleep spindle density in patients with idiopathic REM sleep behavior disorder and patients with Parkinson's disease. Clin Neurophysiol 2014, 125:512-519.
11. Comella C.L., Tanner C.M., Ristanovic R.K. Polysomnographic sleep measures in Parkinson's disease patients with treatment-induced hallucinations. Ann Neurol 1993, 34:710-714.
12. Crowley K., Trinder J., Kim Y., Carrington M., Colrain I.M. The effects of normal aging on sleep spindle and K-complex production. Clin Neurophysiol 2002, 113:1615-1622.
13. Danker-Hopfe H., Anderer P., Zeitlhofer J., Boeck M., Dorn H., Gruber G., et al. Interrater reliability for sleep scoring according to the Rechtschaffen & Kales and the new AASM standard. J Sleep Res 2009, 18:74-84.
14. Danker-Hopfe H., Kunz D., Gruber G., Klosch G., Lorenzo J.L., Himanen S.L., et al. Interrater reliability between scorers from eight European sleep laboratories in subjects with different sleep disorders. J Sleep Res 2004, 13:63-69.
15. De Gennaro L., Ferrara M. Sleep spindles: an overview. Sleep Med Rev 2003, 7:423-440.
16. Devuyst S., Dutoit T., Stenuit P., Kerkhofs M. Automatic sleep spindles detection - overview and development of a standard proposal assessment method. Conf Proc IEEE Eng Med Biol Soc 2011, 2011:1713-1716.
17. Emser W., Brenner M., Stober T., Schimrigk K. Changes in nocturnal sleep in Huntington's and Parkinson's disease. J Neurol 1988, 235:177-179.
18. Eschenko O., Molle M., Born J., Sara S.J. Elevated sleep spindle density after learning or after retrieval in rats. J Neurosci 2006, 26:12914-12920.
19. Feinstein A.R., Cicchetti D.V. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990, 43:543-549.
20. Ferrarelli F., Huber R., Peterson M.J., Massimini M., Murphy M., Riedner B.A., et al. Reduced sleep spindle activity in schizophrenia patients. Am J Psychiatry 2007, 164:483-492.
21. Ferrarelli F., Peterson M.J., Sarasso S., Riedner B.A., Murphy M.J., Benca R.M., et al. Thalamic dysfunction in schizophrenia suggested by whole-night deficits in slow and fast spindles. Am J Psychiatry 2010, 167:1339-1348.
22. Fisher R.A. Intraclass correlations and the analysis of variance. statistical methods for research workers 1925, Oliver and Boyd, Edinburgh, p. 177-210.
23. Fogel S., Martin N., Lafortune M., Barakat M., Debas K., Laventure S., et al. NREM sleep oscillations and brain plasticity in aging. Front Neurol 2012, 3:176.
24. Fogel S.M., Nader R., Cote K.A., Smith C.T. Sleep spindles and learning potential. Behav Neurosci 2007, 121:1-10.
25. Forest G., Poulin J., Daoust A.M., Lussier I., Stip E., Godbout R. Attention and non-REM sleep in neuroleptic-naive persons with schizophrenia and control participants. Psychiatry Res 2007, 149:33-40.
26. Fuentealba P., Timofeev I., Bazhenov M., Sejnowski T.J., Steriade M. Membrane bistability in thalamic reticular neurons during spindle oscillations. J Neurophysiol 2005, 93:294-304.
27. Gais S., Molle M., Helms K., Born J. Learning-dependent increases in sleep spindle density. J Neurosci 2002, 22:6830-6834.
28. Geiger A., Huber R., Kurth S., Ringli M., Jenni O.G., Achermann P. The sleep EEG as a marker of intellectual ability in school age children. Sleep 2011, 34:181-189.
29. Genzel L., Kiefer T., Renner L., Wehrle R., Kluge M., Grozinger M., et al. Sex and modulatory menstrual cycle effects on sleep related memory consolidation. Psychoneuroendocrinology 2012, 37:987-998.
30. Gruber R., Wise M.S., Frenette S., Knaauper B., Boom A., Fontil L., et al. The association between sleep spindles and IQ in healthy school-age children. Int J Psychophysiol 2013, 89:229-240.
31. Halford J.J., Schalkoff R.J., Zhou J., Benbadis S.R., Tatum W.O., Turner R.P., et al. Standardized database development for EEG epileptiform transient detection: EEGnet scoring system and machine learning analysis. J Neurosci Methods 2013, 212:308-316.
32. Hostetler W.E., Doller H.J., Homan R.W. Assessment of a computer program to detect epileptiform spikes. Electroencephalogr Clin Neurophysiol 1992, 83:1-11.
33. Huupponen E., Gomez-Herrero G., Saastamoinen A., Varri A., Hasan J., Himanen S.L. Development and comparison of four sleep spindle detection methods. Artif Intell Med 2007, 40:157-170.
34. Iber C., Ancoli-Israel S., Chesson A., Quan S.F. The AASM manual for the scoring of sleep and associated events: rules, terminology, and technical specifications 2007, American Academy of Sleep Medicine, Westchester, IL.
35. Kaestner E.J., Wixted J.T., Mednick S.C. Pharmacologically increasing sleep spindles enhances recognition for negative and high-arousal memories. J Cogn Neurosci 2013, 25:1597-1610.
36. Kottner J., Audige L., Brorson S., Donner A., Gajewski B.J., Hrobjartsson A., et al. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J Clin Epidemiol 2011, 64:96-106.
37. Kraemer H.C. Ramifications of a population model for κ as a coefficient of reliability. Psychometrika 1979, 44:461-472.
38. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33:159-174.
39. Limoges E., Mottron L., Bolduc C., Berthiaume C., Godbout R. Atypical sleep architecture and the autism phenotype. Brain 2005, 128:1049-1061.
40. Magalang U.J., Chen N.H., Cistulli P.A., Fedson A.C., Gislason T., Hillman D., et al. Agreement in the scoring of respiratory events and sleep among international sleep centers. Sleep 2013, 36:591-596.
41. Malinowska U., Klekowicz H., Wakarow A., Niemcewicz S., Durka P.J. Fully parametric sleep staging compatible with the classical criteria. Neuroinformatics 2009, 7:245-253.
42. Martin N., Lafortune M., Godbout J., Barakat M., Robillard R., Poirier G., et al. Topography of age-related changes in sleep spindles. Neurobiol Aging 2013, 34:468-476.
43. Mednick S.C., McDevitt E.A., Walsh J.K., Wamsley E., Paulus M., Kanady J.C., et al. The critical role of sleep spindles in hippocampal-dependent memory: a pharmacology study. J Neurosci 2013, 33:4494-4504.
44. Miano S., Bruni O., Leuzzi V., Elia M., Verrillo E., Ferri R. Sleep polygraphy in Angelman syndrome. Clin Neurophysiol 2004, 115:938-945.
45. Molle M., Bergmann T.O., Marshall L., Born J. Fast and slow spindles during the sleep slow oscillation: disparate coalescence and engagement in memory processing. Sleep 2011, 34:1411-1421.
46. Montplaisir J., Petit D., Lorrain D., Gauthier S., Nielsen T. Sleep in Alzheimer's disease: further considerations on the role of brainstem and forebrain cholinergic populations in sleep-wake mechanisms. Sleep 1995, 18:145-148.
47. Myslobodsky M., Mintz M., Ben-Mayor V., Radwan H. Unilateral dopamine deficit and lateral EEG asymmetry: sleep abnormalities in hemi-Parkinson's patients. Electroencephalogr Clin Neurophysiol 1982, 54:227-231.
48. Nicolas A., Petit D., Rompre S., Montplaisir J. Sleep spindle characteristics in healthy subjects of different age groups. Clin Neurophysiol 2001, 112:521-527.
49. Rechtschaffen A, Kales A. A manual of standardized terminology, techniques and scoring system for sleep stages of human subjects. U.S. National Institute of Neurological Diseases and Blindness, Neurological Information Network. Bethesda, MD, 1968.
50. Sherif O., Pagurek B., Mahmoud S., Broughton R. Automatic Detection of K-Complex in the Sleep EEG. Int Electrical and Electronic Conf and Exp 1977, 81:204-205.
51. Olbrich E., Achermann P. Analysis of the temporal organization of sleep spindles in the human sleep EEG using a phenomenological modeling approach. J Biol Phys 2008, 34:241-249.
52. Peppard P.E., Ward N.R., Morrell M.J. The impact of obesity on oxygen desaturation during sleep-disordered breathing. Am J Respir Crit Care Med 2009, 180:788-793.
53. Peppard P.E., Young T., Barnet J.H., Palta M., Hagen E.W., Hla K.M. Increased prevalence of sleep-disordered breathing in adults. Am J Epidemiol 2013, 177:1006-1014.
54. Peters K.R., Ray L., Smith V., Smith C. Changes in the density of stage 2 sleep spindles following motor learning in young and older adults. J Sleep Res 2008, 17:23-33.
55. Pittman S.D., MacDonald M.M., Fogel R.B., Malhotra A., Todros K., Levy B., et al. Assessment of automated scoring of polysomnographic recordings in a population with suspected sleep-disordered breathing. Sleep 2004, 27:1394-1403.
56. Plante D.T., Goldstein M.R., Landsness E.C., Peterson M.J., Riedner B.A., Ferrarelli F., et al. Topographic and sex-related differences in sleep spindles in major depressive disorder: a high-density EEG investigation. J Affect Disord 2013, 146:120-125.
57. Schabus M., Hodlmoser K., Gruber G., Sauter C., Anderer P., Klosch G., et al. Sleep spindle-related activity in the human EEG and its relation to general cognitive and learning abilities. Eur J Neurosci 2006, 23:1738-1746.
58. Schabus M., Hoedlmoser K., Pecherstorfer T., Anderer P., Gruber G., Parapatics S., et al. Interindividual sleep spindle differences and their relation to learning-related enhancements. Brain Res 2008, 1191:127-135.
59. Seeck-Hirschner M., Baier P.C., Sever S., Buschbacher A., Aldenhoff J.B., Goder R. Effects of daytime naps on procedural and declarative memory in patients with schizophrenia. J Psychiatr Res 2010, 44:42-47.
60. Silvestri R., Raffaele M., De Domenico P., Tisano A., Mento G., Casella C., et al. Sleep features in Tourette's syndrome, neuroacanthocytosis and Huntington's chorea. Neurophysiol Clin 1995, 25:66-77.
61. Sim J., Wright C.C. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005, 85:257-268.
62. Spearman C. Correlation calculated from faulty data. Br J Psychol 1910, 3:271-295.
63. Tamminen J., Payne J.D., Stickgold R., Wamsley E.J., Gaskell M.G. Sleep spindle activity is associated with the integration of new memories and existing knowledge. J Neurosci 2010, 30:14356-14360.
64. Wamsley E.J., Tucker M.A., Shinn A.K., Ono K.E., McKinley S.K., Ely A.V., et al. Reduced sleep spindles and spindle coherence in schizophrenia: mechanisms of impaired memory consolidation?. Biol Psychiatry 2012, 71:154-161.
65. Warby S.C., Wendt S.L., Welinder P., Munk E.G., Carrillo O., Sorensen H.B., et al. Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. Nat Methods 2014, 11:385-392.
66. Webber W.R., Litt B., Lesser R.P., Fisher R.S., Bankman I. Automatic EEG spike detection: what should the computer imitate?. Electroencephalogr Clin Neurophysiol 1993, 87:364-373.
67. Westerberg C.E., Mander B.A., Florczak S.M., Weintraub S., Mesulam M.M., Zee P.C., et al. Concurrent impairments in sleep and memory in amnestic mild cognitive impairment. J Int Neuropsychol Soc 2012, 18:490-500.
68. Wiegand M., Moller A.A., Lauer C.J., Stolz S., Schreiber W., Dose M., et al. Nocturnal sleep in Huntington's disease. J Neurol 1991, 238:203-208.
69. Zygierewicz J., Blinowska K.J., Durka P.J., Szelenberger W., Niemcewicz S., Androsiuk W. High resolution study of sleep spindles. Clin Neurophysiol 1999, 110:2136-2147.