Scaling requirements extraction to the crowd: Experiments with privacy policies

2014 IEEE 22nd International Requirements Engineering Conference, RE 2014 - Proceedings

Volumn , Issue , 2014, Pages 163-172

  Breaux, Travis D ^a   Schaub, Florian ^a  

^a Carnegie Mellon University   (United States)

Author keywords

crowdsourcing; natural language; requirements extraction

Indexed keywords

CROWDSOURCING; EXTRACTION;

DATA COLLECTION; LABOR INTENSIVE; MANUAL ANALYSIS; NATURAL LANGUAGE TEXT; NATURAL LANGUAGES; NEW APPROACHES; PRIVACY POLICIES; TASK DECOMPOSITION;

DATA MINING;

EID: 84909950534     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/RE.2014.6912258     Document Type: Conference Paper

References (38)

1. L. von Ahn, B. Maurer, C. McMillen, D. Abraham, M. Blum. "reCAPTCHA: Human-Based Character Recognition via Web Security Measures." Science 321 (5895): 1465-68, 2008.
2. M. Allahbakhsh, B. Benatallah, A. Ignjatovic, H.R. Motahari-Nezhad, E. Bertino, S. Dustdar. "Quality Control in Crowdsourcing Systems: Issues and Directions." IEEE Int. Comp. 17 (2): 76-81, 2013.
3. E. Alpaydin, Introduction to Machine Learning, MIT Press, 2009.
4. V. Ambati, S. Vogel, J. Carbonell. "Collaborative Workflow for Crowdsourcing Translation." ACM Conf. Comp. Sup. Coop. Work, pp. 1191-94, 2012.
5. T.D. Breaux, Legal Requirements Acquisition for the Specification of Legally Compliant Information Systems. Ph.D. Thesis, N.C. State Univ., Apr. 2009
6. T.D. Breaux, A.I. Antón, J. Doyle. "Semantic Parameterization: A Process for Modeling Domain Descriptions." ACM Trans. Soft. Engr. Method., 18(2): 5, Nov. 2008
7. T.D. Breaux, H. Hibshi, A. Rao. "Eddy, A Formal Language for Specifying and Analyzing Data Flow Specifications for Conflicting Privacy Requirements." To Appear: Req'ts Engr. J., 2014.
8. T.D. Breaux, M.W. Vail, A.I. Antón: "Towards Regulatory Compliance: Extracting Rights and Obligations to Align Requirements with Regulations", IEEE Int'l Req'ts Engr. Conf., pp. 46-55, 2006.
9. P. Caire, N. Genon, P. Heymans, D.L. Moody. "Visual notation design 2.0: Towards user comprehensible requirements engineering notations," IEEE Int'l Req'ts Engr. Conf., pp. 115-224, 2013.
10. L. B. Chilton, G. Little, D. Edge, D. S. Weld, J. A. Landay. "Cascade: Crowdsourcing Taxonomy Creation." Conf. Human Factors in Comp. Sys., pp. 1999-2008, 2013, ACM.
11. J. Cleland-Huang, A. Czauderna, M. Gibiec, J. Emenecker "A machine learning approach for tracing regulatory codes to product specific requirements," IEEE Int'l Conf. Soft. Engr., pp. 155-164, 2010.
12. J. Corbin, A. Strauss, Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, SAGE Pubs., 2007.
13. A. Doan, R. Ramakrishnan, A. Y. Halevy. "Crowdsourcing Systems on the World-Wide Web." Commun. ACM 54 (4): 86-96, 2011.
14. C. Eickhoff, A. P. de Vries. "Increasing Cheat Robustness of Crowdsourcing Tasks." Information Retrieval 16 (2): 121-37, 2013.
15. A. Finnerty, P. Kucherbaev, S. Tranquillini, G. Convertino. "Keep It Simple: Reward and Task Design in Crowdsourcing." Biannual Conf. Italian Chapter of SIGCHI, 2014, ACM.
16. D.G. Gordon, T.D. Breaux, "Assessing regulatory change through legal requirements coverage modeling." IEEE Int'l Req'ts Engr. Conf., pp. 145-154, 2013.
17. J. J. Horton, L. B. Chilton. "The Labor Economics of Paid Crowdsourcing."ACM Conf. Electronic Commerce, pp. 209-18, 2010.
18. P.-Y. Hsueh, P. Melville, V. Sindhwani. "Data Quality from Crowdsourcing: A Study of Annotation Selection Criteria." NAACL HLT 2009 W'shp Active Learning for NLP, pp. 27-35. 2009.
19. N. Q. V. Hung, N. T. Tam, L. N. Tran, K. Aberer. "An Evaluation of Aggregation Techniques in Crowdsourcing." Web Info. Sys. Engr., 2013, Springer Berlin Heidelberg.
20. P. G. Ipeirotis, F. Provost, J. Wang. "Quality Management on Amazon Mechanical Turk." SIGKDD Workshop on Human Computation, pp. 64-67. 2010, ACM.
21. E. Kamar, S. Hacker, E. Horvitz. "Combining Human and Machine Intelligence in Large-Scale Crowdsourcing." Conf. Auto. Agents & Multiagent Sys., pp. 467-74, 2012.
22. G. Kaur, "Analyzing Email Archives to Better Understand Legal Requirements," W'shp Req'ts Engr. & Law, 2009, pp. 21-26.
23. G. Kazai. "An Exploration of the Influence That Task Parameters Have on the Performance of Crowds." CrowdConf 2010.
24. A. Kittur, E. H. Chi, B. Suh. "Crowdsourcing User Studies with Mechanical Turk." SIGCHI Conf. Human Factors in Comp. Sys., pp. 453-56, 2008, ACM.
25. A. Kittur, B. Smus, S. Khamkar, R. E. Kraut. "CrowdForge: Crowdsourcing Complex Work." ACM Symp. User Interface Software and Technology, pp. 43-52, 2011, ACM.
26. A. Kulkarni, M. Can, B. Hartmann. "Collaboratively Crowdsourcing Workflows with Turkomatic." Conf. Comp. Sup. Coop. Work, pp. 1003-12, 2012, ACM.
27. S. L. Lim, A. Finkelstein. "StakeRare: Using Social Networks and Collaborative Filtering for Large-Scale Requirements Elicitation," IEEE Trans. Soft. Engr., 38(3): 707-735, 2012.
28. G. Little, L. B. Chilton, M. Goldman, R. C. Miller. "TurKit: Human Computation Algorithms on Mechanical Turk." ACM Symp. UI Soft. & Tech., pp. 57-66, 2010.
29. A.K. Massey, J. Eisenstein, A. Anton. "Automated text mining for requirements analysis of policy documents." IEEE Int'l Req'ts Engr. Conf., pp. 4-13, 2013.
30. S. Messick, D.N. Jackson. "Acquiescence and the factorial interpretation of the MMPI." Psych. B., 58(4): 299-304, 1961.
31. M. Negri, L. Bentivogli, Y. Mehdad, D. Giampiccolo, A. Marchetti. "Divide and Conquer: Crowdsourcing the Creation of Cross-Lingual Textual Entailment Corpora." Conf. Emp. Meths. in NLP, pp. 670-79, 2011, ACL.
32. G. Paolacci, J. Chandler, P. G. Ipeirotis. "Running Experiments on Amazon Mechanical Turk." Judge. & Dec. Making 5 (5): 411-19, 2010.
33. A. J. Quinn, B. B. Bederson. "Human Computation: A Survey and Taxonomy of a Growing Field." Conf. Human Factors in Comp. Sys., pp. 1403-12, 2011, ACM.
34. K. Ryan, "The Role of Natural Language in Requirements Engineering," Int'l Symp. Req'ts Engr., 1993, pp. 240-242.
35. J. Saldana, The Coding Manual for Qualitative Researchers, SAGE Pubs, 2012.
36. R. Snow, B. O'Connor, D. Jurafsky, A. Y. Ng.. "Cheap and Fast - but Is It Good?: Evaluating Non-Expert Annotations for Natural Language Tasks." Conf. Emp. Meths. in NLP, pp. 254-63, 2008, ACL.
37. A. Sheshadri, M. Lease. "SQUARE: A Benchmark for Research on Computing Crowd Consensus." AAAI Conf. Human Comp. & Crowdsourcing, 2013, AAAI.
38. O. Zaidan, C. Callison-Burch. "Crowdsourcing Translation: Professional Quality from Non-Professionals."ACL, pp. 1220-29, 2011.