Reliability analysis of observational data: Problems, solutions, and software implementation

Behavior Research Methods, Instruments, and Computers

Volumn 35, Issue 3, 2003, Pages 391-399

  Jansen, Roland G ^a   Wiertz, Leon F ^a   Meyer, Erik S ^a   Noldus, Lucas P J J ^a  

^a Noldus Information Technology bv   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0348162593     PISSN: 07433808     EISSN: None     Source Type: Journal    
DOI: 10.3758/BF03195516     Document Type: Conference Paper

References (22)

1. BAKEMAN, R., & QUERA, V. (1995). Analyzing interaction: Sequential analysis with SDIS and GSEQ. New York: Cambridge University Press.
2. BLOOR, R. N. (1983). A computer program to determine interrater reliability for dichotomous-ordinal rating scales. Behavior Research Methods & Instrumentation, 15, 615.
3. BRAVAIS, A. (1846). Analyse mathématique sur les probabilités des erreurs de situation d'un point. Mémoires présentés par divers savants à l'Académie royale des sciences de l 'Institut de France, 9, 255-332.
4. BURNS, E., & CAVALLARO, C. (1982). A computer program to determine interobserver reliability statistics. Behavior Research Methods & Instrumentation, 14, 42.
5. CHAN, T. S. C. (1987). A DBASE III program that performs significance testing for the Kappa coefficient. Behavior Research Methods, Instruments, & Computers, 19, 53-54.
6. COHEN, J. (1960). A coefficient of agreement for nominal scales. Educational & Psychological Measurement, 20, 37-46.
7. GALTON, F. (1888). Co-relations and their measurement, chiefly from anthropometric data. Proceedings of the Royal Society of London, 45, 135-145.
8. HACCOU, P., & MEELIS, E. (1992). Statistical analysis of behavioural data: An approach based on time-structured models. Oxford: Oxford University Press.
9. HOLLENBECK, A. R. (1978). Problems of reliability in observational data. In G. P. Sackett (Ed.), Observing behavior: Vol. 2, Data collection and analysis methods (pp. 79-98). Baltimore: University Park Press.
10. JOHNSON, S. M., & BOLSTAD, O. D. (1973). Methodological issues in naturalistic observation: Some problems and solutions for field research. In L. A. Hamerlynck, L. C. Handy, & E. J. Mash (Eds.), Behavior change: Methodology, concepts and practice (pp. 7-67). Champaign, IL: Research Press.
11. MACLEAN, W. E., TAPP, J. T., & JOHNSON, W. L. (1985). Alternate methods and software for calculating interobserver agreement for continuous observation data. Journal of Psychopathology & Behavioral Assessment, 7, 65-73.
12. NOLDUS, L. P. J. J. (1991). The Observer: A software system for collection and analysis of observational data. Behavior Research Methods, Instruments, & Computers, 23, 415-429.
13. NOLDUS, L. P. J. J., TRIENES, R. J. H., HENDRIKSEN, A. H. M., JANSEN, H., & JANSEN, R. G. (2000). The Observer Video-Pro: New software for the collection, management, and presentation of time-structured data from videotapes and digital media files. Behavior Research Methods, Instruments, & Computers, 32, 197-206.
14. PEARSON, K. (1920). Notes on the history of correlation. Biometrika, 13, 25-45.
15. REPP, A. C., HARMAN, M. L., FELCE, D., VAN ACKER, R., & KARSH, K. G. (1989). Conducting behavioral assessments on computer-collected data. Behavioral Assessment, 11, 249-268.
16. ROBINSON, B. F., & BAKEMAN, R. (1998). ComKappa: A Windows '95 program for calculating kappa and related statistics. Behavior Research Methods, Instruments, & Computers, 30, 731-734.
17. STRUBE, M. J. (1989). A general program for the calculation of the kappa coefficient. Behavior Research Methods, Instruments, & Computers, 21, 643-644.
18. SUEN, H. K., & ARY, D. (1989). Analyzing quantitative behavioral observation data. Hillsdale, NJ: Erlbaum.
19. THOMANN, B. (2001). Observation and judgment in psychology: Assessing agreement among markings of behavioral events. Behavior Research Methods, Instruments, & Computers, 33, 339-348.
20. VALIQUETTE, C. A. M., LESAGE, A. D., CYR, M., & TOUPIN, J. (1994). Computing Cohen's kappa coefficients using SPSS MATRIX. Behavior Research Methods, Instruments, & Computers, 26, 60-61.
21. VAN DER VLUGT, M. J., KRUK, M. R., VAN ERP, A. M. M., & GEUZE, R. H. (1992). CAMERA: A system for fast and reliable acquisition of multiple ethological records. Behavior Research Methods, Instruments, & Computers, 24, 147-149.
22. WATKINS, M. W., & LARIMER, L. D. (1980), Interrater agreement statistics with the microcomputer. Behavior Research Methods & Instrumentation, 12, 466.