Bias and controversy in evaluation systems

IEEE Transactions on Knowledge and Data Engineering

Volumn 20, Issue 11, 2008, Pages 1490-1504

  Lauw, Hady W ^a   Lim, Ee Peng ^b   Wang, Ke ^c  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b SINGAPORE MANAGEMENT UNIVERSITY   (Singapore)

^c SIMON FRASER UNIVERSITY   (Canada)

Author keywords

Bias; Controversy; Evaluation; Link analysis; Rating; Social network mining

Indexed keywords

REINFORCEMENT;

BIAS; BLOGGING; CONTROVERSY; DEGREE OF CONFIDENCE; EVALUATION; EVALUATION SYSTEMS; LINK ANALYSIS; ON-LINE EVALUATION; RATING; SOCIAL NETWORK MINING; SOCIAL NETWORKING; STATISTICAL APPROACHES; SYNTHETIC DATA SETS; WEB 2.0;

ARSENIC;

EID: 52949154707     PISSN: 10414347     EISSN: None     Source Type: Journal    
DOI: 10.1109/TKDE.2008.77     Document Type: Article

References (41)

1. A. McCook, "Is Peer Review Broken?" The Scientist, vol. 20, no. 2, p. 26, 2006.
2. H.M. Walker, W. Ma, and D. Mboya, "Variability of Referees' Ratings of Conference Papers," Proc. Seventh Ann. Conf. Innovation and Technology in Computer Science Education (ITiCSE 02), pp. 178-182, 2002.
3. J.A. Preston, "Evaluation Software: Improving Consistency and Reliability of Performance Rating," The Supplemental Proc. Conf. Integrating Technology into Computer Science Education: Working Group Reports and Supplemental Proc., pp. 132-134, 1997.
4. G. Adomavicius and A. Tuzhilin, "Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions," IEEE Trans. Knowledge and Data Eng., vol. 17, no. 6, pp. 734-749, June 2005.
5. J.L. Herlocker, J.A. Konstan, L.G. Terveen, and J.T. Riedl, "Evaluating Collaborative Filtering Recommender Systems," ACM Trans. Information Systems, vol. 22, no. 1, pp. 5-53, 2004.
6. P. Resnick and H.R. Varian, "Recommender Systems," Comm. ACM, vol. 40, no. 3, pp. 56-58, 1997.
7. T. Riggs and R. Wilensky, "An Algorithm for Automated Rating of Reviewers," Proc. First ACM/IEEE-CS Joint Conf. Digital Libraries (JCDL '01), pp. 381-387, 2001.
8. M. Chen and J.P. Singh, "Computing and Using Reputations for Internet Ratings," Proc. Third ACM Conf. Electronic Commerce (EC '01), pp. 154-162, 2001.
9. H.W. Lauw, E.-P. Lim, and K. Wang, "Summarizing Review Scores of "Unequal" Reviewers," Proc. Seventh SIAM Int'l Conf. Data Mining (SDM), 2007.
10. S. Dumais and J. Nielsen, "Automating the Assignment of Submitted Manuscripts to Reviewers," Proc. 15th Int'l ACM SIGIR Conf. Research and Development in Information Retrieval (SIGIR '92), pp. 233-244, 1992.
11. J. Geller and R. Scherl, "Challenge: Technology for Automated Reviewer Selection," Proc. 15th Int'l Joint Conf. Artificial Intelligence (IJCAI '97), pp. 55-61, 1997.
12. S. Hettich and M.J. Pazzani, "Mining for Proposal Reviewers: Lessons Learned at the National Science Foundation," Proc. 12th ACM SIGKDD Int'l Conf Knowledge Discovery and Data Mining (KDD '06), pp. 862-871, 2006.
13. J. Cohen, P. Cohen, S.G. West, and L.S. Aiken, Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, third ed. Lawrence Erlbaum Assoc, 2003.
14. J.L. Blackburn and M.D. Hakel, "An Examination of Sources of Peer-Review Bias," Psychological Science, vol. 17, no. 5, pp. 378-382, 2006.
15. A.G. Greenwald and G.M. Gillmore, "Grading Leniency is a Removable Contaminant of Student Ratings," Am. Psychologist, vol. 52, no. 11, pp. 1209-1217, 1997.
16. N. Franke, M. Gruber, D. Harhoff, and J. Henkel, "What You Are Is What You Like-Similarity Biases in Venture Capitalists' Evaluations of Start-Up Teams," J. Business Venturing, vol. 21, no. 6, pp. 802-826, 2006.
17. P. Resnick, R. Zeckhauser, E. Friedman, and K. Kuwabara, "Reputation Systems," Comm. ACM, vol. 43, no. 12, pp. 45-48, 2000.
18. R. Bhattacharjee and A. Goel, "Avoiding Ballot Stuffing in eBay-Like Reputation Systems," Proc. ACM SIGCOMM Workshop Economics of Peer-to-Peer Systems (P2PECON '05), pp. 133-137, 2005.
19. C. Dellarocas, "Mechanisms for Coping with Unfair Ratings and Discriminatory Behavior in Online Reputation Reporting Systems," Proc. 21st Int'l Conf. Information Systems (ICIS '00), pp. 520-525, 2000.
20. J. Zhang and R. Cohen, "Trusting Advice from Other Buyers in E-Marketplaces: The Problem of Unfair Ratings," Proc. Eighth Int'l Conf Electronic Commerce: The New E-Commerce: Innovations for Conquering Current Barriers, Obstacles and Limitations to Conducting Successful Business on the Internet (ICEC '06), pp. 225-234, 2006.
21. A. Arning, R. Agrawal, and P. Raghavan, "A Linear Method for Deviation Detection in Large Databases," Proc. Second ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD '96), pp. 164-169, 1996.
22. E. Knorr and R. Ng, "A Unified Notion of Outliers: Properties and Computation," Proc. Third ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD '97), pp. 219-222, 1997.
23. J. Han and M. Kamber, Data Mining: Concepts and Techniques, second ed. Morgan Kauffmann, 2006.
24. S. Wasserman and K. Faust, Social Network Analysis: Methods and Applications. Cambridge Univ. Press, 1994.
25. K. Faust, "Centrality in Affiliation Networks," Social Networks, vol. 19, no. 2, pp. 157-191, 1997.
26. J. Golbeck and J. Hendler, "Inferring Binary Trust Relationships in Web-Based Social Networks," ACM Trans. Internet Technology, vol. 6, no. 4, pp. 497-529, 2006.
27. L. Backstrom, C. Dwork, and J. Kleinberg, "Wherefore art thou r3579x?: Anonymized Social Networks, Hidden Patterns, and Structural Steganography," Proc. 16th Int'l Conf. World Wide Web (WWW '07), pp. 181-190, 2007.
28. S.P. Borgatti and M.G. Everett, "Network Analysis of 2-Mode Data," Social Networks, vol. 19, no. 3, pp. 243-269, 1997.
29. C. Tantipathananandh, T. Berger-Wolf, and D. Kempe, "A Framework for Community Identification in Dynamic Social Networks," Proc. 13th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD '07), pp. 717-726, 2007.
30. B. Yang, W. Cheung, and J. Liu, "Community Mining from Signed Social Networks," IEEE Trans. Knowledge and Data Eng., vol. 19, no. 10, pp. 1333-1348, Oct. 2007.
31. A. Borodin, G.O. Roberts, J.S. Rosenthal, and P. Tsaparas, "Link Analysis Ranking: Algorithms, Theory, and Experiments," ACM Trans. Interna Technology, vol. 5, no. 1, pp. 231-297, 2005.
32. T.H. Haveliwala, "Topic-Sensitive PageRank: A Context-Sensitive Ranking Algorithm for Web Search," IEEE Trans. Knowledge and Data Eng., vol. 15, no. 4, pp. 784-796, July/Aug. 2003.
33. L. Page, S. Brin, R. Motwani, and T. Winograd, The PageRank Citation Ranking: Bringing Order to the Web. Stanford Digital Library Technologies Project, 1998.
34. J.M. Kleinberg, "Authoritative Sources in a Hyperlinked Environment," /. ACM, vol. 46, no. 5, pp. 604-632, 1999.
35. D. Chakrabarti and C. Faloutsos, "Graph Mining: Laws, Generators, and Algorithms," ACM Computing Surveys, vol. 38, no. 1, p. 2, 2006.
36. X. Yan and J. Han, "CloseGraph: Mining Closed Frequent Graph Patterns," Proc. Ninth ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD '03), pp. 286-295, 2003.
37. Z. Zeng, J. Wang, L. Zhou, and G. Karypis, "Out-of-Core Coherent Closed Quasi-Clique Mining from Large Dense Graph Data-bases," ACM Trans. Database Systems, vol. 32, no. 2, p. 13, 2007.
38. H. Anton and C. Rorres, Elementary Linear Algebra with Applications. John Wiley and Sons, 1987.
39. G.H. Golub and C.F. Van Loan, Matrix Computations, third ed. Johns Hopkins Univ. Press, 1996.
40. C. Dwork, R. Kumar, M. Naor, and D. Sivakumar, "Rank Aggregation Methods for the Web," Proc. 10th Int'l Conf. World Wide Web (WWW '01), pp. 613-622, 2001.
41. R. Fagin, R. Kumar, and D. Sivakumar, "Comparing Top K Lists," SiAM J. Discrete Math., vol. 17, no. 1, pp. 134-160, 2003.