A Primer on Observational Measurement

Assessment

Volumn 23, Issue 4, 2016, Pages 404-413

  Girard, Jeffrey M ^a   Cohn, Jeffrey F ^a  

^a UNIVERSITY OF PITTSBURGH   (United States)

Author keywords

affective computing; computational behavioral science; contemporary validity theory; interobserver reliability; observational measurement

Indexed keywords

AUTOMATION; COMPUTER; FORECASTING; HUMAN; OBSERVATIONAL STUDY; OBSERVER VARIATION; PROCEDURES; REPRODUCIBILITY; STATISTICS;

AUTOMATION; COMPUTERS; FORECASTING; HUMANS; OBSERVATIONAL STUDIES AS TOPIC; OBSERVER VARIATION; REPRODUCIBILITY OF RESULTS; STATISTICS AS TOPIC;

EID: 84978481309     PISSN: 10731911     EISSN: 15523489     Source Type: Journal    
DOI: 10.1177/1073191116635807     Document Type: Article

References (56)

1. J.G.Adair (1984). The Hawthorne effect: A reconsideration of the methodological artifact. Journal of Applied Psychology, 69, 334-345.
2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: Author.
3. Z.AmbadarJ.F.CohnL.I.Reed (2009). All smiles are not created equal: Morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. Journal of Nonverbal Behavior, 33, 17-34.
4. R.BakemanJ.R.Brownlee (1980). The strategic use of parallel play: A sequential analysis. Child Development, 51, 873-878.
5. R.BakemanV.Quera (2011). Squential analysis and observational methods for the behavioral sciences. New York, NY: Cambridge University Press.
6. J.K.BakerJ.D.HaltiganR.BrewsterJ.JaccardD.S.Messinger (2010). Non-expert ratings of infant and parent emotion: Concordance with expert coding and relevance to early autism risk. International Journal of Behavioral Development, 34, 88-95.
7. E.M.BennettR.AlpertA.C.Goldstein (1954). Communication through limited response questioning. Public Opinion Quarterly, 18, 303-308.
8. R.Boice (1983). Observational skills. Psychological Bulletin, 93, 3-29.
9. J.T.CacioppoL.G.Tassinary (1990). Inferring psychological significance from physiological signals. American Psychologist, 45, 16-28.
10. D.T.CampbellJ.C.Stanley (1966). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
11. G.J.Cizek (2015). Validating test score meaning and defending test score use: Different aims, different methods. Assessment in Education: Principles, Policy & Practice, 1-14.
12. J.A.CoanJ.M.Gottman (2007). The specific affect coding system (SPAFF). In (Eds.), Handbook of emotion elicitation and assessment (pp. 267-285). Oxford, England: Oxford University Press.
13. J.F.CohnF.De la Torre (2014). Automated face analysis for affective computing. In (Eds.), Handbook of affective computing (pp. 131-150). New York, NY: Oxford University Press.
14. J.F.CohnP.Ekman (2005). Measuring facial action by manual coding, facial EMG, and automatic facial image analysis. In (Eds.), The new handbook of nonverbal behavior research (pp. 9-64). New York, NY: Oxford University Press.
15. R.CowieG.McKeownE.Douglas-Cowie (2012). Tracing emotion: An overview. International Journal of Synthetic Emotions, 3, 1-17.
16. F.De la TorreT.SimonZ.AmbadarJ.F.Cohn (2011). Fast-FACS: A computer-assisted system to increase speed and reliability of manual FACS coding. Affective Computing and Intelligent Interaction, 6974 LNCS(Pt. 1), 57-66.
17. H.DibekliogluZ.HammalY.YangJ.F.Cohn (2015). Multimodal detection of depression in clinical interviews. Proceedings of the ACM International Conference on Multimodal Interaction, Seattle, Washington.
18. P.EkmanW.V.Friesen (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.
19. P.EkmanW.V.FriesenJ.Hager (2002). Facial action coding system: A technique for the measurement of facial movement. Salt Lake City, UT: Research Nexus.
20. G.C.Feng (2013a). Factors affecting intercoder reliability: A Monte Carlo experiment. Quality & Quantity, 47, 2959-2982.
21. G.C.Feng (2013b). Intercoder reliability indices: Disuse, misuse, and abuse. Quality & Quantity, 48, 1803-1815.
22. J.L.Fleiss (1981). Statistical methods for rates and proportions (2nd ed.). New York, NY: John Wiley.
23. K.F.Geisinger (1992). The metamorphosis of test validation. Educational Psychologist, 27, 197-222.
24. J.M.Girard (2014). CARMA: Software for continuous affect rating and media annotation. Journal of Open Research Software, 2(1), e5.
25. J.M.GirardJ.F.CohnL.A.JeniM.A.SayetteF.De la Torre (2015). Spontaneous facial expression in unscripted social interactions can be measured automatically. Behavior Research Methods, 47, 1136-1147.
26. J.M.GirardJ.F.CohnM.H.MahoorS.M.MavadatiZ.HammalD.P.Rosenwald (2014). Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses. Image and Vision Computing, 32, 641-647.
27. J.M.GottmanR.W.Levenson (1985). A valid procedure for obtaining self-report of affect in marital interaction. Journal of Consulting and Clinical Psychology, 53, 151-160.
28. H.GunesB.W.Schuller (2013). Categorical and dimensional affect analysis in continuous input: Current trends and future directions. Image and Vision Computing, 31, 120-136.
29. K.L.Gwet (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.
30. A.M.HayesC.Yasinski (2015). Pattern destabilization and emotional processing in cognitive therapy for personality disorders. Frontiers in Psychology, 23, 107.
31. A.O.HorvathR.P.Bedi (2002). The alliance. In (Ed.), Psychotherapy relationships that work: Therapists contributions and responsiveness to patients (pp. 37-69). New York, NY: Oxford University Press.
32. G.JuckelR.MerglA.PrasslP.MavrogiorgouH.WitthausH.J.MollerU.Hegerl (2008). Kinematic analysis of facial behaviour in patients with schizophrenia under emotional stimulation by films with “Mr. Bean.”European Archives of Psychiatry and Clinical Neuroscience, 258, 186-191.
33. L.M.Knobloch-FeddersK.L.CritchfieldT.BoissonN.WoodsR.BitmanC.E.Durbin (2014). Depression, relationship quality, and couples’ demand/withdraw and demand/submit sequential interactions. Journal of Counseling Psychology, 61, 264-279.
34. P.Kosso (2011). A summary of the scientific method. Berlin, Germany: Springer Science & Business Media.
35. I.G.G.KreftJ.de Leeuw (1998). Introducing multilevel modeling. Thousand Oaks, CA: Sage.
36. R.W.LevensonA.M.Ruef (1992). Empathy: A physiological substrate. Journal of Personality and Social Psychology, 63, 234-246.
37. I.LizdekP.SadlerE.WoodyN.EthierG.Malet (2012). Capturing the stream of behavior: A computer-joystick method for coding interpersonal behavior continuously over time. Social Science Computer Review, 30, 513-521.
38. W.MasonS.Suri (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1-23.
39. K.O.McGrawS.P.Wong (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
40. R.MerglP.MavrogiorgouU.HegerlG.Juckel (2005). Kinematical analysis of emotionally induced facial expressions: A novel tool to investigate hypomimia in patients suffering from depression. Journal of Neurology, Neurosurgery, and Psychiatry, 76, 138-140.
41. S.Messick (1989). Validity. In (Ed.), Educational measurement (3rd ed., pp. 13-103). New York, NY: Macmillan.
42. K.D.O’LearyR.N.Kent (1973). Behavior modification for social action: Research tactics and problems. In (Eds.), Behavior change: Methodology, concepts and practice (pp. 69-96). Champaign, IL: Research Press.
43. M.PanticL.J.M.Rothkrantz (2003). Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91, 1370-1390.
44. F.RamseyerW.Tschacher (2011). Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome. Journal of Consulting and Clinical Psychology, 79, 284-295.
45. S.W.RaudenbushA.S.Bryk (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
46. J.B.Reid (1970). Reliability assessment of observation data: A possible methodological problem. Child Development, 41, 1143-1150.
47. S.SalvatoreW.Tschacher (2012). Time dependency of psychotherapeutic exchanges: The contribution of the Theory of Dynamic Systems in analyzing process. Frontiers in Psychology, 3(253), 1-14.
48. L.W.Scheffler (1977). Being is believing: Playing the psychiatric patient. Journal of Psychiatric Education, 1(1), 63-67.
49. K.L.SchmidtZ.AmbadarJ.F.CohnL.I.Reed (2006). Movement differences between deliberate and spontaneous facial expressions: Zygomaticus major action in smiling. Journal of Nonverbal Behavior, 30, 37-52.
50. N.SchwarzB.KnauperH.-J.HipplerE.Noelle-NeumannL.Clark (1991). Rating scales: Numeric valuesmaychange the meaning of scale labels. Public Opinion Quarterly, 55, 570-582.
51. P.E.ShroutJ.L.Fleiss (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
52. S.S.Stevens (1946). On the theory of scales of measurement. Science, 103, 677-680.
53. A.A.StoneJ.S.TurkkanC.A.BachrachJ.B.JobeH.S.KurtzmanV.S.Cain (2000). The science of self-report: Implications for research and practice. Mahwah, NJ: Lawrence Erlbaum.
54. M.F.ValstarT.AlmaevJ.M.GirardG.MckeownM.MehuL.YinJ.F.. Cohn (2015, May4-8). FERA 2015: Second facial expression recognition and analysis challenge. Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, Ljubljana.
55. M.W.van SomerenY.F.BarnardJ.A.C.Sandberg (1994). The think aloud method: A practical guide to modelling cognitive processes. London, England: Academic Press.
56. X.ZhaoJ.S.LiuK.Deng (2012). Assumptions behind inter-coder reliability indices. In (Ed.), Communication yearbook (pp. 418-480). London, England: Routledge.