Design of randomized experiments in networks

Proceedings of the IEEE

Volumn 102, Issue 12, 2015, Pages 1940-1951

  Walker, Dylan ^a   Muchnik, Lev ^b  

^a BOSTON UNIVERSITY   (United States)

^b HEBREW UNIVERSITY OF JERUSALEM   (Israel)

Author keywords

Behavioral science; general; science; sociology; systems, man, and cybernetics

Indexed keywords

BEHAVIORAL RESEARCH; BIG DATA;

BEHAVIORAL SCIENCE; GENERAL; POPULATION LEVELS; RANDOMIZED CONTROLLED TRIAL; RANDOMIZED EXPERIMENTS; SCIENCE; SCIENTIFIC METHOD; SOCIOLOGY;

DESIGN OF EXPERIMENTS;

EID: 84914168650     PISSN: 00189219     EISSN: 15582256     Source Type: Journal    
DOI: 10.1109/JPROC.2014.2363674     Document Type: Article

References (85)

1. V. Mayer-Schonberger and K. Cukier, Big Data: A Revolution That Will Transform How We Live, Work, Think, 1st ed. Chicago, IL, USA: Eamon Dolan/Houghton Mifflin Harcourt, 2013, p. 256.
2. F. Provost and T. Fawcett, Data Science for Business. Sebastopol, CA, USA: O'Reilly Media, 2013, p. 414.
3. M. A. Hamburg and F. S. Collins, " The path to personalized medicine, " New England J. Med., vol. 363, pp. 301-304, 2010.
4. T. B. Murdoch and A. S. Detsky, " The inevitable application of big data to health care, " J. Amer. Med. Assoc., vol. 309, no. 13, pp. 1351-1352, Apr. 2013.
5. P. B. Jensen, L. J. Jensen, and S. Brunak, " Mining electronic health records: Towards better research applications and clinical care, " Nature Rev. Genetics, vol. 13, no. 6, pp. 395-405, Jun. 2012.
6. P. Brusilovsky, A. Kobsa, and W. Nejdl, The Adaptive Web: Methods and Strategies of Web Personalization. New York, NY, USA: Springer-Verlag, 2007, p. 766.
7. H. Chen and V. C. Storey, " Business intelligence and analytics: From big data to big impact, " MIS Quarterly, vol. 36, no. 4, pp. 1165-1188, 2012.
8. B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, " Analysis of recommendation algorithms for E-Commerce, " Proc. 2nd ACM Conf. Electron. Commerce, 2000, pp. 158-167.
9. J. Lampel and H. Mintzberg, " Customizing customization, " MIT Sloan Manag. Rev., vol. 38, no. 1, pp. 21-30, 1996.
10. G. Marcus and E. Davis, " Eight (no, nine!) problems with big data, " The New York Times, Apr. 6, 2014.
11. D. Boyd and K. Crawford, " Six provocations for big data, " Social Sci. Res. Netw. Electron J., 2011. [Online]. Available: Http://dx.doi.org/ 10.2139/ssrn.1926431.
12. FDA Clinical Trials Guidance Documents, 2014. [Online]. Available: Http://www.fda. gov/regulatoryinformation/guidances/ ucm122046.htm
13. R. Kohavi, " Online controlled experiments, " Proc. 1st Workshop User Engagement Optim., 2013, pp. 15-16.
14. H. O. Stolberg, G. Norman, and I. Trop, " Randomized controlled trials, " Amer J. Roentgenol., vol. 183, no. 6, pp. 1539-1544, 2014.
15. D. Lazer et al., " Computational social science, " Science, vol. 323, no. 5915, pp. 721-723, 2009.
16. D. Centola and M. Macy, " Complex contagions and the weakness of long ties, " Amer J. Sociol., vol. 113, no. 3, pp. 702-734, 2007.
17. L. Backstrom and J. Kleinberg, " Network bucket testing, " Proc. 20th Int. Conf. World Wide Web, 2011, pp. 615-624.
18. D. J. Watts and P. S. Dodds, " Influentials, networks, public opinion formation, " J. Consum. Res., vol. 34, no. 4, pp. 441-458, Dec. 2007.
19. P. Dodds and D.Watts, " Universal behavior in a generalized model of contagion, " Phys. Rev. Lett., vol. 92, no. 21, 2004, DOI: 10.1103/ PhysRevLett.92.218701.
20. R. Bond et al., " A 61-million-person experiment in social influence and political mobilization, " Nature, vol. 489, no. 7415, pp. 295-298, 2012.
21. S. Aral and D. Walker, " Identifying influential and susceptible members of social networks, " Science, vol. 337, no. 6092, pp. 337-341, Jul. 2012.
22. S. Aral and D. Walker, " Creating social contagion through viral product design: A randomized trial of peer influence in networks, " Manage. Sci., vol. 57, no. 9, pp. 1623-1639, Sep. 2011.
23. D. Godes, " Firm-created word-of-mouth communication: Evidence from a field test, " Marketing Sci., vol. 28, no. 4, pp. 721-739, 2009.
24. J. Goldenberg, S. Han, D. R. Lehmann, and J. W. Hong, " The role of hubs in the adoption process, " J. Marketing, vol. 73, no. 2, pp. 1-13, 2009.
25. R. Iyengar, C. Van den Bulte, and T. W. Valente, " Opinion leadership and social contagion in new product diffusion, " Marketing Sci., vol. 30, no. 2, pp. 195-212, 2010.
26. C. Van den Bulte and S. Wuyts, Social Networks and Marketing. Cambridge, MA, USA: Marketing Science Institute, 2007.
27. A. Goyal, Social Influence and Its Applications. Vancouver, BC, Canada: Univ. British Columbia Press, 2013.
28. E. Bakshy and J. Hofman, " Everyone's an influencer: Quantifying influence on twitter, " Proc. 4th ACM Int. Conf. Web Search Data Mining, 2011, pp. 65-74.
29. M.Webster and J. Sell, Laboratory Experiments in the Social Sciences. New York, NY, USA: Academic, 2007, p. 576.
30. A. Falk and J. Heckman, " Lab experiments are a major source of knowledge in the social sciences, " Science, vol. 326, no. 5952, pp. 535-538, 2009.
31. S. Levitt and J. List, " Field experiments in economics: The past, the present, the future, " Eur. Econ. Rev., vol. 53, no. 1, pp. 1-18, 2009.
32. M. Kearns, S. Suri, and N. Montfort, " An experimental study of the coloring problem on human subject networks, " Science, vol. 313, no. 5788, pp. 824-827, Aug. 2006.
33. S. Judd, M. Kearns, and Y. Vorobeychik, " Behavioral dynamics and influence in networked coloring and consensus, " Proc. Nat. Acad. Sci. USA, vol. 107, no. 34, pp. 14978-82, Aug. 2010.
34. M. Kearns, S. Judd, J. Tan, and J. Wortman, " Behavioral experiments on biased voting in networks, " Proc. Nat. Acad. Sci. USA, vol. 106, no. 5, pp. 1347-1352, Feb. 2009.
35. L. Qiu, H. Rui, and A. B. Whinston, " The impact of social network structures on prediction market accuracy in the presence of insider information, " J. Manage. Inf. Syst., vol. 31, no. 3, pp. 145-172, 2014.
36. L. Qiu, H. Rui, and A. Whinston, " Effects of social networks on prediction markets: Examination in a controlled experiment, " J. Manage. Inf. Syst., vol. 30, no. 4, pp. 235-268, 2014.
37. G. Paolacci, J. Chandler, and P. Ipeirotis, " Running experiments on Amazon Mechanical Turk, " Judgment Decision Making, vol. 5, no. 5, pp. 411-419, 2010.
38. M. J. Salganik, P. S. Dodds, and D. J. Watts, " Experimental study of inequality and unpredictability in an artificial cultural market, " Science, vol. 311, no. 5762, pp. 854-856, 2006.
39. S. Suri and D. J. Watts, " Cooperation and contagion in web-based, networked public goods experiments, " PLoS One, vol. 6, no. 3, Mar. 2011, e16836+.
40. W. Mason and S. Suri, " Conducting behavioral research on Amazon's Mechanical Turk, " Behav. Res. Methods, vol. 44, no. 1, pp. 1-23, 2012.
41. D. Centola, " The spread of behavior in an online social network experiment, " Science, vol. 329, no. 5996, pp. 1194-1197, Sep. 2010.
42. D. Centola, " An experimental study of homophily in the adoption of health behavior, " Science, vol. 334, no. 6060, pp. 1269-1272, Dec. 2011.
43. D. Centola and A. van de Rijt, " Choosing your network: Social preferences in an online health community, " Social Sci. Med., May 2014, DOI: 10.1016/j.socscimed.2014. 05.019.
44. R. Bapna, A. Gupta, S. Rice, and A. Sundararajan, " Trust, reciprocity and the strength of social ties: An online social network based field experiment, " working paper, Jul. 2011.
45. J. Wang, S. Suri, and D. J. Watts, " Cooperation and assortativity with dynamic partner updating, " Proc. Nat. Acad. Sci. USA, vol. 109, no. 36, pp. 14363-14368, Sep. 2012.
46. G. Harrison and J. List, " Field experiments, " J. Econ. Lit., vol. 42, no. 4, pp. 1009-1055, 2004.
47. D. Edwards, I'm Feeling Lucky: The Confessions of Google Employee Number 59, 1st ed. Chicago, IL, USA: Houghton Mifflin Harcourt, 2011, p. 432.
48. R. Kohavi et al., " Trustworthy online controlled experiments, " Proc. 18th ACM SIGKDD Int. Conf. Knowl. Disc. Data Mining, 2012, pp. 786-794.
49. E. Bakshy, D. Eckles, and M. Bernstein, " Designing and deploying online field experiments, " Proc. 23rd Int. Conf. World Wide Web, 2014, pp. 283-292.
50. A. D. I. Kramer, J. E. Guillory, and J. T. Hancock, " Experimental evidence of massive-scale emotional contagion through social networks, " Proc. Nat. Acad. Sci. USA, vol. 111, no. 24, pp. 8788-9790, Jun. 2014.
51. S. Harriman and J. Patel, " The ethics and editorial challenges of internet-based research, " BMC Med., vol. 12, no. 1, Jul. 2014, DOI:10.1186/s12916-014-0124-3.
52. R. Bapna, J. Ramaprasad, G. Shmueli, and A. Umyarov, " One-way mirrors and weak-signaling in online dating: A randomized field experiment, " working paper, 2013.
53. E. Bakshy, D. Eckles, R. Yan, and I. Rosenn, " Social influence in social advertising: Evidence from field experiments, " Proc. 13th ACM Conf. Electron. Commerce, 2012, pp. 146-161.
54. E. Bakshy, I. Rosenn, C. Marlow, and L. Adamic, " The role of social networks in information diffusion, " Proc. 21st Int. Conf. World Wide Web, 2012, pp. 519-528.
55. S. Taylor, E. Bakshy, and S. Aral, " Selection effects in online sharing: Consequences for peer adoption, " Proc. 14th ACM Conf. Electron. Commerce, 2013, pp. 821-836.
56. S. Taylor, L. Muchnik, and S. Aral, " Identity and opinion: A randomized experiment, " working paper, 2014.
57. S. Aral, L. Muchnik, and A. Sundararajan, " Engineering social contagions: Optimal network seeding and incentive strategies, " Netw. Sci., vol. 1, no. 2, pp. 125-153, Feb. 2013.
58. S. E. Asch, " Effects of group pressure upon the modification and distortion of judgment Groups, Leadership and Men. Pittsburgh, PA, USA: Carnegie Press, 1951, pp. 177-190.
59. S. Milgram, L. Bickman, and L. Berkowitz, " Note on the drawing power of crowds of different size, " J. Personality Social Psychol., vol. 13, no. 2, pp. 79-82, 1969.
60. M. Salganik and D. Watts, " Leading the herd astray: An experimental study of self-fulfilling prophecies in an artificial cultural market, " Social Psychol. Quarterly, vol. 71, no. 4, pp. 338-355, 2008.
61. L. Muchnik, S. Aral, and S. Taylor, " Social influence bias: A randomized experiment, " Science, vol. 341, no. 6146, pp. 647-651, 2013.
62. M. Godinho de Matos, P. Ferreira, M. D. Smith, and R. Telang, " Culling the herding: Using real world randomized experiments to measure social bias with known costly goods, " Social Science Research Network (SSRN) working paper, May 2014.
63. R. Bapna and A. Umyarov, " Are paid subscriptions on music social networks contagious? A randomized field experiment, " Carlson School Manage., Univ. Minnesota, Minneapolis, MN, USA, SOBACO working paper, 2012.
64. S. Aral and D. Walker, " Tie strength, embeddedness, social influence: A large-scale networked experiment, " Manage. Sci., vol. 60, no. 6, pp. 1352-1370, Jun. 2014.
65. C. F. Manski, " Identification of endogenous social effects: The reflection problem, " Rev. Econ. Studies, vol. 60, no. 3, pp. 531-542, 1993.
66. S. Aral, L. Muchnik, and A. Sundararajan, " Distinguishing influence-based contagion from homophily-driven diffusion in dynamic networks, " Proc. Nat. Acad. Sci USA, vol. 106, no. 51, pp. 21544-21549, 2009.
67. C. R. Shalizi and A. C. Thomas, " Homophily and contagion are generically confounded in observational social network studies, " Sociol. Methods Res., vol. 42, no. 2, pp. 211-239, 2011.
68. D. R. Cox, Planning of Experiments. New York, NY, USA: Wiley-Interscience, 1958, p. 320.
69. D. Rubin, " Bayesian inference for causal effects: The role of randomization, " Ann. Stat., vol. 6, no. 1, pp. 34-58, 1978.
70. C. F. Manski, " Identification of treatment response with social interactions, " Econom J., vol. 16, no. 1, pp. S1-S23, Feb. 2013.
71. P. M. Aronow and C. Samii, " Estimating average causal effects under general interference, " Proc. Summer Meeting Soc. Political Methodol., 2012, pp. 19-21.
72. D. Eckles, B. Karrer, and J. Ugander, " Design and analysis of experiments in networks: Reducing bias from interference, " working paper, 2014, p. 29.
73. A. Coppock and N. Sircar, " Design of field experiments under unknown interference structures, " working paper, 2013.
74. L. Katzir, E. Liberty, and O. Somekh, " Framework and algorithms for network bucket testing, " Proc. 21st Int. Conf. World Wide Web, 2012, pp. 1029-1036.
75. J. Ugander, B. Karrer, L. Backstrom, and J. Kleinberg, " Graph cluster randomization: Network exposure to multiple universes, " Proc. 19th ACM SIGKDD Int. Conf. Knowl. Disc. Data Mining, 2013, pp. 329-337.
76. E. Airoldi, P. Toulis, E. Kao, and D. B. Rubin, " Estimation of causal peer influence effects, " J. Mach. Learn. Res., vol. 28, no. 3, pp. 1489-1497, 2013.
77. A. C. Thomas and M. Finegold, " Protocols for randomized experiments to identify network contagion, " working paper, 2013, p. 13.
78. J. Nishimura and J. Ugander, " Restreaming graph partitioning: Simple versatile algorithms for advanced balancing, " Proc. 19th ACM SIGKDD Int. Conf. Knowl. Disc. Data Mining, 2013, pp. 1106-1114.
79. D. Eckles and E. Bakshy, " Bias and high-dimensional adjustment in observational studies of peer effects, " working paper, 2014.
80. M. J. Salganik and D. D. Heckathorn, " Sampling and estimation in hidden populations using respondent-driven sampling, " Sociol. Methodol., vol. 34, no. 1, pp. 193-240, Dec. 2004.
81. M. Gjoka, M. Kurant, C. T. Butts, and A. Markopoulou, " A walk in Facebook: Uniform sampling of users in online social networks, " working paper, May 2009.
82. D. Godes et al., " The firm's management of social interactions, " Marketing Lett., vol. 16, no. 3/4, pp. 415-428, 2005.
83. S. Aral and S. Taylor, " Viral incentive systems: A randomized field experiment, " Stern School Business, New York Univ., working paper, 2014.
84. G. Pickard et al., " Time-critical social mobilization, " Science, vol. 334, no. 6055, pp. 509-512, Oct. 2011.
85. E. Bakshy and D. Eckles, " Uncertainty in online experiments with dependent data: An evaluation of bootstrap methods, " Proc. 19th ACM SIGKDD Int. Conf. Knowl. Disc. Data Mining, 2013, pp. 1303-1311.