Conducting Judgment Studies: Some Methodological Issues

The New Handbook of Methods in Nonverbal Behavior Research

Volumn , Issue , 2008, Pages

  Rosenthal, Robert ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Judges; Judgment study; Metrics; Reliability; Variable

Indexed keywords

EID: 84921259406     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1093/acprof:oso/9780198529620.003.0005     Document Type: Chapter

References (94)

1. Allport, F.H. (1924). Social psychology. Boston: Houghton-Mifflin.
2. Ambady, N. & Rosenthal, R. (1992). Thin slices of expressive behavior as predictors of interpersonal consequences: a metaanalysis. Psychological Bulletin, 111, 256-74.
3. Ambady, N. & Rosenthal, R. (1993). Half a minute: predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology, 64, 431-41.
4. Ambady, N., LaPlante, D., Nguyen, T., Rosenthal, R., Chaumeton, N., & Levinson, W. (2002). Surgeon's tone of voice: a clue to malpractice history. Surgery, 132(1), 5-9.
5. American Psychological Association (2001). Publication manual of the American Psychological Association (5th edn). Washington, DC: American Psychological Association.
6. Armor, D.J. (1974). Theta reliability and factor scaling. In Sociological methodology 1973-1974 (ed. H.L. Costner). San Francisco: Jossey-Bass.
7. Babad, E. (1992). Teacher expectancies and nonverbal behavior. In Applications of nonverbal behavioral theories and research (ed. R.S. Feldman), pp. 167-90. Hillsdale, NJ: Erlbaum.
8. Barnett, V. & Lewis, T. (1978). Outliers in statistical data. New York: John Wiley.
9. Blanck, P.D. & Rosenthal, R. (1984). Mediation of interpersonal expectancy effects: counselor's tone of voice. Journal of Educational Psychology, 76, 418-26.
10. Blanck, P.D., Buck, R.W., & Rosenthal, R. (ed.) (1986). Nonverbal communication in the clinical context. University Park, PA:Pennsylvania State University Press.
11. Blanck, P.D., Rosenthal, R., Hart, A.J., & Bernieri, F. (1990). The measure of the judge: an empirically-based framework for exploring trial judges' behavior. Iowa Law Review, 75, 653-84.
12. Blanck, P.D., Rosenthal, R., & Vanicelli, M. (1986). Talking to and about patients: the therapist's tone of voice. In Nonverbal communication in the clinical context (ed. P.D. Blanck, R.W. Buck, & R. Rosenthal), pp. 99-143. University Park, PA:Pennsylvania State University Press.
13. Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296-322.
14. Brunswik, E. (1956). Perception and the representative design of psychological experiments. Berkeley: University of California Press.
15. Buck, R. (1979). Individual differences in nonverbal sending accuracy and electrodermal responding: the externalizinginternalizing dimension. In Skill in nonverbal communication: individual differences (ed. R. Rosenthal), pp. 140-70. Cambridge, MA: Oelgeschlager, Gunn and Hain.
16. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
17. Cohen, J. (1968). Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-20.
18. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd edn). Hillsdale, NJ: Erlbaum.
19. Cronbach, L.J. (1951). Coefficient alpha and the internal consistency of tests. Psychometrika, 16, 297-334.
20. Davitz, J.R. (1964). The communication of emotional meaning. New York: McGraw-Hill.
21. DePaulo, B.M. & Rosenthal, R. (1979a). Ambivalence, discrepancy, and deception in nonverbal communication. In Skill in nonverbal communication: individual differences (ed. R. Rosenthal), pp. 204-48. Cambridge, MA: Oelgeschlager, Gunn and Hain.
22. DePaulo, B.M. & Rosenthal, R. (1979b). Telling lies. Journal of Personality and Social Psychology, 37, 1713-22.
23. DiMatteo, M.R. (1979). Nonverbal skill and the physician-patient relationship. In Skill in nonverbal communication: individual differences (ed. R. Rosenthal), pp. 104-34. Cambridge, MA: Oelgeschlager, Gunn and Hain.
24. Eden, D. (1990). Pygmalion in management: productivity as a self-fulfilling prophecy. Lexington, MA: Heath.
25. Ekman, P. (1965). Communication through nonverbal behavior : a source of information about an interpersonal relationship. In Affect, cognition, and personality (ed. S.S. Tomkins & C. Izard), pp. 390-442. New York: Springer.
26. Ekman, P. (1973). Cross-cultural studies of facial expression. In Darwin and facial expression: a century of research in review (ed. P. Ekman), pp. 169-222. New York: Academic Press.
27. Ekman, P., Friesen, W.V., O'sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., et al. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 53, 712-17.
28. Friedman, H.S. (1976). About face: the role of facial expressions of emotion in the verbal communication of meaning. Unpublished doctoral dissertation, Harvard University.
29. Friedman, H.S. (1978). The relative strength of verbal versus nonverbal cues. Personality and Social Psychology, 4, 147-50.
30. Friedman, H.S. (1979a). The interactive effects of facial expressions of emotion and verbal messages on perceptions of affective meaning. Journal of Experimental Social Psychology, 15, 453-69.
31. Friedman, H.S. (1979b). The concept of skill in nonverbal communication: implications for understanding social interaction. In Skill in nonverbal communication: individual differences (ed. R. Rosenthal), pp. 2-27. Cambridge, MA: Oelgeschlager, Gunn and Hain.
32. Guilford, J.P. (1954). Psychometric methods (2nd edn). New York: McGraw-Hill.
33. Hall, J.A. (1984). Nonverbal sex differences. Baltimore, MD: The Johns Hopkins University Press.
34. Hall, J.A. & Bernieri, F.J. (ed.) (2001). Interpersonal sensitivity: theory and measurement. Mahwah, NJ: Erlbaum.
35. Halverson, A.M., Hallahan, M., Hart, A.J., & Rosenthal, R. (1997). Reducing the biasing effects of judges' nonverbal behavior with simplified jury instruction. Journal of Applied Psychology, 82, 590-8.
36. Harrigan, J.A. & Rosenthal, R. (1983). Physicians' head and body positions as determinants of perceived rapport. Journal of Applied Social Psychology, 13, 496-509.
37. Harrigan, J.A., Harrigan, K.M., Sale, B.A., & Rosenthal, R. (1996). Detecting anxiety and defensiveness from visual and auditory cues. Journal of Personality, 64, 675-709.
38. Harris, M.J. & Rosenthal, R. (1985). The mediation of interpersonal expectancy effects: 31 meta-analyses. Psychological Bulletin, 97, 363-86.
39. Harris, M.J. & Rosenthal, R. (1986). Four factors in the mediation of teacher expectancy effects. In The social psychology of education (ed. R.S. Feldman), pp. 91-114. New York: Cambridge University Press.
40. Hedges, L.V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-28.
41. Huber, P.J. (1981). Robust statistics. New York: John Wiley.
42. Li, H., Rosenthal, R., & Rubin, D.B. (1996). Reliability of measurement in psychology: from Spearman-Brown to maximal reliability. Psychological Methods, 1, 98-107.
43. Milmoe, S., Novey, M.S., Kagan, J., & Rosenthal, R. (1968). The mother's voice: postdictor of aspects of her baby's behavior. Proceedings of the 76th Annual Convention of APA, pp. 463-464.
44. Milmoe, S., Rosenthal, R., Blane, H. T., Chafetz, M. E., & Wolf, I. (1967). The doctor's voice: postdictor of successful referral of alcoholic patients. Journal of Abnormal Psychology, 72, 78-84.
45. Mosteller, F. & Rourke R.E.K. (1973). Sturdy statistics. Reading, MA: Addison-Wesley.
46. Mosteller, F. & Tukey, J.W. (1977). Data analysis and regression. Reading, MA: Addison-Wesley.
47. Nasby, W., Hayden, B., & DePaulo, B.M. (1980). Attributional bias among aggressive boys to interpret unambiguous social stimuli as displays of hostility. Journal of Abnormal Psychology, 89, 459-68.
48. Puccinelli, N.M., Tickle-Degnen, L., & Rosenthal, R. (2003). Effect of dyadic context on judgments of rapport: dyad task and partner presence, Journal of Nonverbal Behavior, 27, 211-36.
49. Puccinelli, N.M., Tickle-Degnen, L., & Rosenthal, R. (2004). Effect of target position and target task on judge sensitivity to felt rapport. Journal of Nonverbal Behavior, 28, 211-20.
50. Puccinelli, N.M., Tickle-Degnen, L., & Rosenthal, R. (2005). Stage left, stage right? Position effects on perception of an agent. Manuscript submitted for publication.
51. Rogers, P.L., Scherer, K.R., & Rosenthal, R. (1971). Content-filtering human speech: a simple electronic system. Behavior Research Methods and Instrumentation, 3, 16-18.
52. Rosenthal, R. (1966). Experimenter effects in behavioral research. New York: Appleton-Century-Crofts.
53. Rosenthal, R. (1967). Covert communication in the psychological experiment. Psychological Bulletin, 67, 356-67.
54. Rosenthal, R. (1969). Interpersonal expectations. In Artifact in behavioral research (ed. R. Rosenthal & R.L. Rosnow), pp. 181-277. New York: Academic Press.
55. Rosenthal, R. (1973). Estimating effective reliabilities in studies that employ judges' ratings. Journal of Clinical Psychology, 29, 342-5.
56. Rosenthal, R. (1974). On the social psychology of the self-fulfilling prophecy: further evidence for Pygmalion effects and their mediating mechanisms, Module 53. New York: MSS Modular Publications.
57. Rosenthal, R. (1976). Experimenter effects in behavioral research (enlarged edn). New York: Irvington Publishers.
58. Rosenthal, R. (ed.) (1979). Skill in nonverbal communication: individual differences. Cambridge, MA: Oelgeschlager, Gunn and Hain.
59. Rosenthal, R. (1982). Conducting judgment studies. In Handbook of methods in nonverbal behavior research (ed. K.R. Scherer & P. Ekman), pp. 287-361. New York: Cambridge University Press.
60. Rosenthal, R. (1985). From unconscious experimenter bias to teacher expectancy effects. In Teacher expectancies (ed. J.B. Dusek, V.C. Hall, & W.J. Meyer). Hillsdale, NJ: Erlbaum.
61. Rosenthal, R. (1987). Judgment studies: design, analysis, and meta-analysis. New York: Cambridge University Press.
62. Rosenthal, R. (1991). Some indices of the reliability of peer review. The Behavioral and Brain Sciences, 14, 160-1.
63. Rosenthal, R. (2002a). Covert communication in classrooms, clinics, courtrooms, and cubicles. American Psychologist, 57, 838-49.
64. Rosenthal, R. (2002b). The Pygmalion effect and its mediating mechanisms. In Improving academic achievement (ed. J. Aronson), pp. 25-36. New York: Academic Press.
65. Rosenthal, R. (2003). Covert communication in laboratories, classrooms, and the truly real world. Current Directions in Psychological Science, 12, 151-4.
66. Rosenthal, R. & DePaulo, B.M. (1979a). Sex differences in accommodation in nonverbal communication. In Skill in nonverbal communication: individual differences (ed. R. Rosenthal), pp. 68-103. Cambridge, MA: Oelgeschlager, Gunn and Hain.
67. Rosenthal, R. & DePaulo, B.M. (1979b). Sex differences in eavesdropping on nonverbal cues. Journal of Personality and Social Psychology, 37, 273-85.
68. Rosenthal, R. & Jacobson, L. (1968). Pygmalion in the classroom: teacher expectation and pupils' intellectual development. New York: Holt, Rinehart & Winston.
69. Rosenthal, R. & Rosnow, R.L. (1975a). A primer of methods for the behavioral sciences. New York: John Wiley.
70. Rosenthal, R. & Rosnow, R.L. (1975b). The volunteer subject. New York: Wiley-Interscience.
71. Rosenthal, R. & Rosnow, R.L. (1991). Essentials of behavioral research: methods and data analysis (2nd edn). New York:McGraw-Hill.
72. Rosenthal, R. & Rubin, D.B. (1978). Interpersonal expectancy effects: the first 345 studies. The Behavioral and Brain Sciences, 3, 377-86.
73. Rosenthal R. & Rubin, D.B. (1979). A note on percent variance explained as a measure of the importance of effects. Journal of Applied Social Psychology, 9, 395-6.
74. Rosenthal, R. & Rubin, D.B. (1982). A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology, 74, 166-9.
75. Rosenthal, R. & Rubin, D.B. (2003). requivalent: a simple effect size indicator. Psychological Methods, 8, 492-6.
76. Rosenthal, R., Blanck, P.D., & Vannicelli, M. (1984). Speaking to and about patients: predicting therapists' tone of voice. Journal of Consulting and Clinical Psycology, 52, 679-86.
77. Rosenthal, R., Hall, J.A., DiMatteo, M.R., Rogers, P.L., & Archer, D. (1979). Sensitivity to nonverbal communication: The PONS test. Baltimore, MD: The Johns Hopkins University Press.
78. Rosenthal, R., Rosnow, R.L., & Rubin, D.B. (2000). Contrasts and effect sizes in behavioral research: a correlational approach. New York: Cambridge University Press.
79. Scherer, K.R. (1970). Non-verbale Kommunikation. Hamburg: Helmut Buske Verlag.
80. Scherer, K.R. (1971). Randomized-splicing: a note on a simple technique for masking speech content. Journal of Experimental Research in Personality, 5, 155-9.
81. Scherer, K.R. (1978). Personality inference from voice quality: the loud voice of extraversion. European Journal of Social Psychology, 8, 467-87.
82. Scherer, K.R. (1979a). Acoustic concomitants of emotional dimensions: judging affect from synthesized tone sequences. In Nonverbal communication: readings with commentary (2nd edn) (ed. S. Weitz), pp. 249-53. New York: Oxford University Press.
83. Scherer, K.R. (1979b). Voice and speech correlates of perceived social influence in simulated juries. In Language and social psychology (ed. H. Giles and R. St.Clair), pp. 88-120. Oxford: Basil Blackwell.
84. Scherer, K.R. (1982). Methods of research on vocal communication: paradigms and parameters. In Handbook of methods in nonverbal behavior research (ed. K.R. Scherer and P. Ekman), pp. 136-98. New York: Cambridge University Press.
85. Scherer, K.R. & Oshinsky, J.S. (1977). Cue utilization in emotion attribution from auditory stimuli. Motivation and Emotion, 1, 331-46.
86. Scherer, K.R., Koivumaki, J.H., & Rosenthal, R. (1972). Minimal cues in the vocal communication of affect: judging emotions from content-masked speech. Journal of Psycholinguistic Research, 1, 269-85.
87. Scherer, K.R., Scherer, U., Hall, J.A., & Rosenthal, R. (1977). Differential attribution of personality based on multi-channel presentation of verbal and nonverbal cues. Psychological Research, 39, 221-47.
88. Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 171-95.
89. Tukey, J.W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
90. Uno, Y., Koivumaki, J.H., & Rosenthal, R. (1972). Unintended experimenter behavior as evaluated by Japanese and American observers. Journal of Social Psychology, 88, 91-106.
91. Weitz, S. (1979). Commentary. In Skill in nonverbal communication: individual differences (ed. R. Rosenthal), pp. 28-30. Cambridge, MA: Oelgeschlager, Gunn and Hain.
92. Westen, D. & Rosenthal, R. (2003). Quantifying construct validity: two simple measures. Journal of Personality and Social Psychology, 84, 608-18.
93. Wilkinson, L. & Task Force on Statistical Inference (1999). Statistical methods in psychology journals: guidelines and explanations. American Psychologist, 54, 594-604.
94. Zuckerman, M. & Larrance, D.T. (1979). Individual differences in perceived encoding and decoding abilities. In Skill in nonverbal communication: individual differences (ed. R. Rosenthal), pp. 171-203. Cambridge, MA: Oelgeschlager, Gunn and Hain.