Testing Intergroup Concordance in Ranking Experiments With Two Groups of Judges

Psychological Methods

Volumn 13, Issue 1, 2008, Pages 58-71

  Dekle, Dawn J ^a   Leung, Denis H Y ^a   Zhu, Min ^b  

^a SINGAPORE MANAGEMENT UNIVERSITY   (Singapore)

^b CSIRO   (Australia)

Author keywords

concordance; intergroup; Kendall's W; missing data; ranking experiment

Indexed keywords

ARTICLE; DECISION MAKING; EMPIRICAL RESEARCH; HUMAN; PSYCHOLOGICAL MODEL;

EMPIRICAL RESEARCH; HUMANS; JUDGMENT; MODELS, PSYCHOLOGICAL;

EID: 43049135865     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/1082-989X.13.1.58     Document Type: Article

References (53)

1. Alvo M., and Cabilio P. Rank correlation methods for missing data. The Canadian Journal of Statistics 23 (1995) 345-358
2. Arvesen J.N. Jackknifing U-statistics. Annals of Mathematical Statistics 40 (1969) 2076-2100
3. Beckett J., and Schucany W.R. Concordance among categorized groups of judges. Journal of Educational Statistics 4 (1979) 125-137
4. Beran R. Prepivoting test statistics: A bootstrap view of asymptotic refinements. Journal of the American Statistical Association 83 (1988) 687-697
5. Bonner B.L. Expertise in group problem solving: Recognition, social combination, and performance. Group Dynamics: Theory, Research, and Practice 8 (2004) 277-290
6. Castel A., Miró J., and Rull M. Validation of the faces pain scale in a sample of elderly Spanish individuals. European Journal of Psychological Assessment 21 (2005) 265-270
7. Critchlow D.E. Metric methods for analyzing partially ranked data (1985), Springer-Verlag, Berlin, Germany
8. Daniels H.E. The relation between measures of correlation in the universe of sample permutations. Biometrika 33 (1944) 129-135
9. Diaconis P., and Graham R.L. Spearman's footrule as a measure of disarray. Journal of the Royal Statistical Society: Series B 39 (1977) 262-268
10. Ehrenberg A.S.C. On sampling from a population of rankers. Biometrika 39 (1952) 82-87
11. Elstein A.S., Chapman G.B., and Knight S.J. Patients' values and clinical substituted judgments: The case of localized prostate cancer. Health Psychology 24 4, Suppl. (2005) S85-S92
12. Feigin P.D., and Alvo M. Intergroup diversity and concordance for ranking data: An approach via metrics for permutations. Annals of Statistics 14 (1986) 691-707
13. Fisher S.G., Macrosson W.D.K., and Yusuff M.R. Team performance and human values. Psychological Reports 79 (1996) 1019-1024
14. Hays W.L. A note on average tau as a measure of concordance. Journal of the American Statistical Association 55 (1960) 331-341
15. Hollander M., and Sethuraman J. Testing for agreement between two groups of judges (with commentaries). Biometrika 65 (1978) 403-411
16. Kahneman D. Experienced utility and objective happiness: A moment-based approach. In: Kahneman D., and Tversky A. (Eds). Choices, values and frames (2000), Cambridge University Press, Cambridge, United Kingdom 673-692
17. Kendall M.G., and Gibbons J.D. Rank correlation methods. 5th ed. (1990), Griffin, London
18. Kendall M.G., and Smith B.B. The problem of m rankings. Annals of Mathematical Statistics 10 (1939) 275-287
19. Kraemer H.C. Intergroup concordance: Definition and estimation. Biometrika 68 (1981) 641-646
20. Kruskal W.H. Ordinal measures of association. Journal of the American Statistical Association 53 (1958) 814-861
21. Legendre P. Species associations: The Kendall coefficient of concordance revisited. Journal of Agricultural, Biological, and Environmental Statistics 10 2 (2005) 226-245
22. Legendre P., and Lapointe F.J. Assessing congruence among distance matrices: Single-malt scotch whiskies revisited. Australian & New Zealand Journal of Statistics 46 (2004) 615-629
23. Li L., and Schucany W.R. Some properties of a test for concordance of two groups of judges. Biometrika 62 (1975) 417-423
24. Linhart H. Approximate test for m rankings. Biometrika 47 (1960) 476-480
25. Little R.J.A., and Rubin D.B. Statistical analysis with missing data. 2nd ed. (2002), Wiley, Hoboken, NJ
26. Lohmann A., Delius J.D., Hollard V.D., and Friesel M.F. Discrimination of shape reflections and shape orientations by Columba livia. Journal of Comparative Psychology 102 (1988) 3-13
27. Lyerly S.B. The average Spearman rank correlation coefficient. Psychometrika 17 (1952) 421-428
28. Mardia K.V., Kent J.T., and Bibby J.M. Multivariate analysis (1979), Academic Press, New York
29. Marlowe C.M., Schneider S.L., and Nelson C.E. Gender and attractiveness biases in hiring decisions: Are more experienced managers less biased?. Journal of Applied Psychology 81 (1996) 11-21
30. McKnight J., and Hills A.M. Just how good are we at estimating attractiveness?. Psychology, Evolution and Gender 1 (1999) 213-238
31. Miró J., Huguet A., and Nieto R. Evaluation of reliability, validity, and preference for a pain intensity scale for use with the elderly. Journal of Pain 6 (2005) 727-735
32. Page E.B. Ordered hypothesis for multiple treatments: A significant test for linear ranks. Journal of the American Statistical Association 58 (1963) 216-230
33. Pope M., and Scott J. Do clinicians understand why individuals stop taking lithium?. Journal of Affective Disorders 74 (2003) 287-291
34. Quenouille M.H. Notes on bias in estimation. Biometrika 43 (1956) 353-360
35. Rubin D.B. Multiple imputation for nonresponse in surveys (1987), Wiley, New York
36. Rule B.G., Bisanz G.L., and Kohn M. Anatomy of a persuasion schema: Targets, goals, and strategies. Journal of Personality and Social Psychology 48 (1985) 1127-1140
37. Salama I.A., and Quade D. Agreement among censored rankings using Spearman's footrule. Communications in Statistics: Theory and Methods 33 (2004) 1837-1850
38. Schucany W.R. Comments on paper by M. Hollander and J. Sethuraman. Biometrika 65 (1978) 410-411
39. Schucany W.R., and Beckett J. Analysis of multiple sets of incomplete rankings. Communications in Statistics: Theory and Methods 5 (1976) 1327-1334
40. Schucany W.R., and Frawley W.H. A rank test for two group concordance. Psychometrika 38 (1973) 249-258
41. Sen P.K., Salama I.A., and Quade D. Spearman's footrule under progressive censoring. Journal of Nonparametric Statistics 15 (2003) 53-60
42. Shieh G.S. A weighted Kendall's tau statistic. Statistics & Probability Letters 39 (1998) 17-24
43. Shieh G.S., Bai Z., and Tsai W.Y. Rank tests for independence-with a weighted contamination alternative. Statistica Sinica 10 (2000) 577-593
44. Spearman C. The proof and measurement of association between two things. American Journal of Psychology 15 (1904) 72-101
45. Stephens L.J., Claypool P.L., and Buchalter B. Partial ordering of populations. Journal of Educational and Behavioral Statistics 2 (1977) 41-54
46. Stephens L.J., Claypool P.L., and Buchalter B. Partial ordering of populations [Correction notice]. Journal of Educational and Behavioral Statistics 3 (1978) 384
47. Stewart A.E., and Stewart E.A. A decision-making technique for choosing a psychology internship. Professional Psychology: Research and Practice 27 (1996) 521-526
48. Swanson J.S., Wigal S.B., and Udrea D. Evaluation of individual subjects in the analog classroom setting: I. Examples of graphical and statistical procedures for within-subject ranking of responses to different delivery patterns of methylphenidate. Psychopharmacology Bulletin 34 (1998) 825-832
49. Thurstone L.L. A law of comparative judgment. Psychological Review 79 (1927) 281-299
50. Thurstone L.L. Rank order as a psychological method. Journal of Experimental Psychology 14 (1931) 187-201
51. Vidmar G., and Cernigoj M. Studying norms in small groups by means of multi-group concordance analysis. Horizons of Psychology 13 4 (2004) 55-66
52. Wanschura R.G., and Dawson W.E. Regression effect and individual power functions over sessions. Journal of Experimental Psychology 102 (1974) 806-812
53. Yu P.L.H., Lam K.F., and Alvo M. Nonparametric rank tests for independence in opinion surveys. Australian Journal of Statistics 31 (2002) 279-290