Crowdsourced network measurements: Benefits and best practices

Computer Networks

Volumn 90, Issue , 2015, Pages 85-98

  Hirth, Matthias ^a   Hoßfeld, Tobias ^b   Mellia, Marco ^c   Schwartz, Christian ^a   Lehrieder, Frank ^a  

^a UNIVERSITY OF WÜRZBURG   (Germany)

^b UNIVERSITY OF DUISBURG ESSEN   (Germany)

^c POLITECNICO DI TORINO   (Italy)

Author keywords

Best practices; Crowdsourcing; Methodology; Network measurements

Indexed keywords

INTERNET; INTERNET SERVICE PROVIDERS;

BEST PRACTICES; CROWDSOURCING; CROWDSOURCING PLATFORMS; DESIGN AND EVALUATIONS; INTERNET SERVICE PROVIDERS (ISP); METHODOLOGY; NETWORK MEASUREMENT; TRADITIONAL MEASUREMENT TECHNIQUES;

COMPUTER AIDED NETWORK ANALYSIS;

EID: 84945474213     PISSN: 13891286     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.comnet.2015.07.003     Document Type: Article

References (66)

1. M. Vukovic, R. Das, and S. Kumara From sensing to controlling: the state of the art in ubiquitous crowdsourcing Int. J. Commun. Netw. Distrib. Syst. 11 1 2013 11 25
2. V.K. Adhikari, S. Jain, and Z.-L. Zhang YouTube traffic dynamics and its interplay with a tier-1 ISP: An ISP Perspective Proceedings of SIGCOMM Conference 2010 New Delhi, India
3. B. Ager, N. Chatzis, A. Feldmann, N. Sarrar, S. Uhlig, and W. Willinger Anatomy of a large European IXP Proceedings of SIGCOMM Conference 2012 Helsinki, Finland
4. A. Finamore, M. Mellia, M. Meo, M. Munafo, and D. Rossi Experiences of internet traffic monitoring with Tstat IEEE Netw. 25 3 2011 8 14
5. P.M. Santiago del Rio, D. Rossi, F. Gringoli, L. Nava, L. Salgarelli, and J. Aracil Wire-speed statistical classification of network traffic on commodity hardware Proceedings of Internet Measurement Conference 2012 Boston, Massachusetts, USA
6. B. Chun, D. Culler, T. Roscoe, A. Bavier, L. Peterson, M. Wawrzoniak, and M. Bowman Planetlab: an overlay testbed for broad-coverage services Comput. Commun. Rev. 33 3 2003 3 12
7. C. Dovrolis, K. Gummadi, A. Kuzmanovic, and S.D. Meinrath Measurement lab: overview and an invitation to the research community Comput. Commun. Rev. 40 3 2010 53 56
8. GENI, GENI - Global Environment for Network Innovations, mbox. http://www.geni.net/ (accessed February, 2015).
9. GLab, GLab - German Lab. http://www.german-lab.de (accessed February, 2015.
10. Y. Shavitt, and E. Shir DIMES: let the internet measure itself Comput. Commun. Rev. 35 5 2005 71 74
11. H.V. Madhyastha, T. Isdal, M. Piatek, C. Dixon, T. Anderson, A. Krishnamurthy, and A. Venkataramani iPlane: an information plane for distributed services Proceedings of the Symposium on Operating Systems Design and Implementation 2006 Seattle, Washington, USA
12. Z. Wen, S. Triukose, and M. Rabinovich Facilitating focused internet measurements Proceedings of the SIGMETRICS Conference 2007 San Diego, California, USA
13. M. Dhawan, J. Samuel, R. Teixeira, C. Kreibich, M. Aollman, N. Weaver, and V. Paxson Fathom: a browser-based network measurement platform Proceedings of Internet Measurement Conference 2012 Boston, Massachusettes, USA
14. D.R. Choffnes, F.E. Bustamante, and Z. Ge Crowdsourcing Service-level network event monitoring Proceedings of the SIGCOMM Conference 2010 New Delhi, India
15. M.A. Sánchez, J.S. Otto, Z.S. Bischof, D.R. Choffnes, F.E. Bustamante, B. Krishnamurthy, and W. Willinger DASU: pushing experiments to the Internet's edge Proceedings of the Symposium on Networked Systems Design and Implementation 2013 Lombard, Illinois, USA
16. SamKnows Ltd, SamKnows - A free broadband masurement service for consumers. http://www.samknows.com/ (accessed February, 2015).
17. Ripe NCC, Ripe atlas. http://atlas.ripe.net/ (accessed February, 2015).
18. Conviva Inc., Conviva - intelligent control platform. http://www.conviva.com (accessed February, 2015).
19. D.R. Choffnes, and F.E. Bustamante Taming the torrent: a practical approach to reducing cross-ISP traffic in peer-to-peer systems Comput. Commun. Rev. 38 4 2008 362 374
20. T. Hoßfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt, K. Diepold, and P. Tran-Gia Best practices for QoE crowdtesting: QoE assessment with crowdsourcing Trans. Multimed. 16 2 2014 541 558
21. J. Howe The rise of crowdsourcing Wired Mag. 14 6 2006
22. T. Hoßfeld, M. Hirth, and P. Tran-Gia Modeling of crowdsourcing platforms and granularity of work organization in future internet Proceedings of the International Teletraffic Congress 2011 San Francisco, California, USA
23. A.J. Quinn, and B.B. Bederson Human computation: a survey and taxonomy of a growing field Proceedings of the Conference on Human Factors in Computing Systems 2011 Vancouver, Canada
24. L. Von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum reCAPTCHA: human-based character recognition via web security measures Science 321 5895 2008 1465 1468
25. 99designs Pty Ltd, 99designs. http://www.99designs.com (accessed February, 2015).
26. InnoCentive Inc., InnoCentive challenges programms. http://www.innocentive.com (accessed February, 2015).
27. Amazon Inc., Amazon Mechanical Turk. http://www.mturk.com (accessed February, 2015).
28. Weblabcenter Inc., Microworkers. http://www.microworkers.com (accessed February, 2015).
29. Gigwalk Inc., Gigwalk. http://www.gigwalk.com (accessed February, 2015).
30. M. Allahbakhsh, B. Benatallah, A. Ignjatovic, H.R. Motahari-Nezhad, E. Bertino, and S. Dustdar Quality control in crowdsourcing systems: issues and directions IEEE Internet Comput. 17 2 2013 76 81
31. M. Hirth, T. Hoßfeld, and P. Tran-Gia Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms Math. Comput. Modell. 57 2013 2918 2932
32. CrowdFlower Inc., CrowdFlower. http://www.crowdflower.com. (accessed February, 2015).
33. B. Staehle, M. Hirth, R. Pries, F. Wamser, and D. Staehle YoMo: a YouTube application comfort monitoring tool Proceedings of the Workshop on New Dimensions in the Assessment and Support of Quality of Experience for Multimedia Applications, Tampere, Finland 2010
34. K. Thompson, G.J. Miller, and R. Wilder Wide-area internet traffic patterns and characteristics IEEE Netw. 11 6 1997 10 23
35. R. Van De Meent Network Link Dimensioning: A Measurement & Modeling Based Approach (Ph.D. thesis) 2006 online available (open access) http://doc.utwente.nl/56434/
36. A.-J. Su, D.R. Choffnes, A. Kuzmanovic, and F.E. Bustamante Drafting behind Akamai (travelocity-based detouring) Proceedings of the SIGCOMM Conference 2006 Pisa, Italy
37. S. Banerjee, T.G. Griffin, and M. Pias The interdomain connectivity of PlanetLab nodes Proceedings of the Workshop on Passive and Active Network Measurementm 2004 Juan-les-Pins, France
38. Ookla, Speedtest.net. http://www.speedtest.net (accessed February, 2015).
39. Matt Martz, Speedtest.net commandline interface. http://github.com/sivel/speedtest-cli (accessed February, 2015).
40. V. Burger, M. Hirth, C. Schwartz, and T. Hoßfeld Increasing the coverage of vantage points in distributed active network measurements by crowdsourcing Proceedings of the Conference on Measurement, Modelling and Evaluation of Computing Systems 2014 Bamberg, Germany
41. M. Varela, T. Mäki, L. Skorin-Kapov, and T. Hoßfeld Increasing payments in crowdsourcing: don't look a gift horse in the mouth Proceedings of the Workshop on Perceptual Quality of Systems 2013 Vienna, Austria
42. J. Redi, T. Hoßfeld, P. Korshunov, F. Mazza, I. Povoa, and C. Keimel Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal Proceedings of the CrowdMM Workshop 2013 Barcelona, Spain
43. ITU-T, Recommendation P.800.1 - Mean Opinion Score (MOS) terminology. http://www.itu.int/, 2006 (accessed February, 2015).
44. T. Hoßfeld, R. Schatz, E. Biersack, and L. Plissonneau Internet video delivery in YouTube: from traffic measurements to quality of experience E. Biersack, C. Callegari, M. Matijasevic, Data Traffic Monitoring and Analysis: From Measurement, Classification and Anomaly Detection to Quality of Experience (Computer Communications and Networks Series) vol. 77 2013 Springer
45. P. Amrehn, K. Vandenbroucke, T. Hoßfeld, K. de Moor, M. Hirth, R. Schatz, and P. Casas Need for speed? On quality of experience for file storage services Proceedings of the Workshop on Perceptual Quality of Systems 2013 Vienna, Austria
46. I. Drago, M. Mellia, M. M.Munafo, A. Sperotto, R. Sadre, and A. Pras Inside Dropbox: understanding personal cloud storage services Proceedings of the Internet Measurement Conference 2012 Boston, Massachusetts, USA
47. B. Gardlo, M. Ries, and T. Hoßfeld Impact of screening technique on crowdsourcing QoE assessments Proceedings of the Radioelektronika Confernce 2012 Brno, Czech Republic
48. T. Hoßfeld, R. Schatz, M. Seufert, M. Hirth, T. Zinner, and P. Tran-Gia Quantification of YouTube QoE via crowdsourcing Proceedings of the Workshop on Multimedia Quality of Experience - Modeling, Evaluation, and Directions 2011 Dana Point, California, USA
49. J. Ross, L. Irani, M. Silberman, A. Zaldivar, and B. Tomlinson Who are the crowdworkers?: Shifting demographics in mechanical turk Proceedings of the Conference on Human Factors in Computing Systems 2010 Atlanta, Georgia, USA
50. M. Hirth, T. Hoßfeld, and P. Tran-Gia Anatomy of a crowdsourcing platform - using the example of Microworkers.com Proceedings of the Workshop on Future Internet and Next Generation Network 2011 Seoul, Korea
51. D. Vakharia, and M. Lease Beyond mechanical turk: an analysis of paid crowd work platforms Proceedings of the iConference 2015 Newport Beach, California, USA
52. J.M. Rzeszotarski, and A. Kittur Instrumenting the Crowd: using implicit behavioral measures to predict task performance Proceedings of the Symposium on User Interface Software and Technology 2011 Santa Barbara, California, USA
53. R.K. Mok, W. Li, and R.K. Chang A user behavior based cheat detection mechanism for crowdtesting Proceedings of the SIGCOMM Conference 2014 Chicago, Illinois, USA
54. M. Hirth, S. Scheuring, T. Hoßfeld, C. Schwartz, and P. Tran-Gia Predicting result quality in crowdsourcing using application layer monitoring Proceedings of the International Conference on Communications and Electronics 2014 Da Nang, Vietnam
55. M. Sabou, K. Bontcheva, and A. Scharl Crowdsourcing research opportunities: lessons from natural language processing Proceedings of the Conference on Knowledge Management and Knowledge Technologies 2012 Graz, Austria
56. A. Kittur, J.V. Nickerson, M. Bernstein, E. Gerber, A. Shaw, J. Zimmerman, M. Lease, and J. Horton The Future of Crowd Work Proceedings of the Conference on Computer Supported Cooperative Work 2013 San Antonio, Texas, USA
57. Y. Zhao, and Q. Zhu Evaluation on crowdsourcing research: current status and future direction Inf. Syst. Front. 16 3 2014 417 434
58. J.P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R.C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, and et al. VizWiz: nearly real-time answers to visual questions Proceedings of the Symposium on User Interface Software and Technology 2010 New York, USA
59. M.S. Bernstein, J. Brandt, R.C. Miller, and D.R. Karger Crowds in two seconds: enabling realtime crowd-powered interfaces Proceedings of the Symposium on User Interface Software and Technology 2011 Santa Barbara, California, USA
60. M.S. Bernstein, D.R. Karger, R.C. Miller, and J. Brandt Analytic methods for optimizing realtime crowdsourcing Proceedings of the Collective Intelligence Conference 2012 Cambrige, Massachusetts, USA
61. D. Yang, G. Xue, X. Fang, and J. Tang Crowdsourcing to smartphones: incentive mechanism design for mobile phone sensing Proceedings of the Conference on Mobile Computing and Networking 2012 Istanbul, Turkey
62. M. Lease, J. Hullman, J.P. Bigham, M.S. Bernstein, J. Kim, W. Lasecki, S. Bakhshi, T. Mitra, R.C. Miller, Mechanical turk is not anonymous. Available at SSRN: http://ssrn.com/abstract=2228728; http://dx.doi.org/10.2139/ssrn.2228728, 2013.
63. S. Faradani, B. Hartmann, and P.G. Ipeirotis What's the right price? Pricing tasks for finishing on time Proceedings of the Workshop on Human Computation 2011 San Francisco, California, USA
64. Y. Singer, and M. Mittal Pricing mechanisms for crowdsourcing markets Proceedings of the Conference on World Wide Web 2013 Rio de Janeiro, Brazil
65. J. Wang, P.G. Ipeirotis, and F. Provost Quality-based pricing for crowdsourced workers NYU Working Paper 2013
66. M. Vukovic Crowdsourcing for enterprises Proceedings on the Conference on Services-I 2009 Los Angeles, California, USA