A focused crawler for dark web forums

Journal of the American Society for Information Science and Technology

Volumn 61, Issue 6, 2010, Pages 1213-1231

  Fu, Tianjun ^a   Abbasi, Ahmed ^b   Chen, Hsinchun ^a  

^a UNIVERSITY OF ARIZONA   (United States)

^b UNIVERSITY OF WISCONSIN MILWAUKEE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONTENT ANALYSIS; CYBERCRIME; DARK WEB; DARK WEB FORUMS; FOCUSED CRAWLER; IMPROVEMENT MECHANISM; ONLINE COMMUNITIES; SYSTEM USE; WEB CRAWLING;

COMPUTER CRIME; FEATURE EXTRACTION;

INTERNET;

EID: 77952994598     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.21323     Document Type: Article

References (51)

1. Abbasi, A., & Chen, H. (2005). Identification and comparison of extremistgroup Web forum messages using authorship analysis. IEEE Intelligent Systems, 20(5), 67-75.
2. Aggarwal, C.C., Al-Garawi, F., & Yu, P.S. (2001). Intelligent crawling on the World Wide Web with arbitrary predicates. In Proceedings of the Tenth World Wide Web Conference (pp. 96-105). New York: ACM Press.
3. Baeza-Yates, R. (2003). Information retrieval in the Web: Beyond current search engines. International Journal of Approximate Reasoning, 34, 97-104.
4. Barbosa, L., & Freire, J. (2004). Siphoning hidden-Web data through keyword-based interfaces. In Proceedings of the 19th Brazilian Symposium on Databases (pp. 309-321). Brasilia, Brazil:SBBD.
5. Bergman, M.K. (2000). The deep Web: Surfacing hidden value. Retrieved March 3, 2010, from http://quod.lib.umich.edu/cgi/t/text/textidx?c=jep;view= text;rgn=main;idno=3336451.0007.104
6. Burris, V., Smith, E., & Strahm, A. (2000). White supremacist networks on the Internet. Sociological Focus, 33(2), 215-235.
7. Chakrabarti, S., Punera, K., & Subramanyam, M. (2002). Accelerated focused crawling through online relevance feedback. In Proceedings of the 11th International World Wide Web Conference (pp. 148-159). New York: ACM Press.
8. Chakrabarti, S., Van. Den Berg, M., & Dom, B. (1999). Focused crawling: A new approach to topic-specific resource discovery. In Proceedings of the Eighth World Wide Web Conference (pp. 1623-1640). New York: ACM Press.
9. Chalmers, M., & Chitson, P. (1992). Bead: Explorations in information visualization. In Proceedings of the 15th Annual International ACM/SIGIR Conference (pp. 330-337). New York: ACM Press.
10. Chau, M., & Chen, H. (2003). Comparison of three vertical search spiders. IEEE Computer, 36(5), 56-62.
11. Chen, H. (2006). Intelligence and security informatics for international security: Information sharing and data mining. London: Springer Press.
12. Chen, H., & Chau, M. (2003). Web mining: Machine learning for Web applications. Annual Review of Information Science and Technology, 37, 289-329.
13. Chen, H., Chung, Y., Ramsey, M., & Yang, C. (1998a). A smart itsy bitsy spider for the Web. Journal of the American Society for Information Science, 49(7), 604-619. (Pubitemid 128612358)
14. Chen, H., Chung, Y., Ramsey, M., &Yang, C. (1998b). An intelligent personal spider (agent) for dynamic internet/intranet searching. Decision Support Systems, 23(1), 41-58.
15. Cheong, F.C. (1996). Internet agents: Spiders, wanderers, brokers, and bots. Indianapolis, IN: New Riders.
16. Cho, J., & Garcia-Molina, H. (2000). The evolution of the Web and implications for an incremental crawler. In Proceedings of the 26th International Conference on Very Large Databases (pp. 200-209). New York: ACM Press.
17. Cho, J., Garcia-Molina, H., & Page, L. (1998). Efficient crawling through URL ordering. Computer Networks and ISDN Systems, 30 (1-7), 161-172. (Pubitemid 128666934)
18. Crilley, K. (2001). Information warfare: New battle fields terrorists, propaganda, and the Internet. In Proceedings of the Association for Information Management, 53(7), 250-264.
19. De Bra, P.M.E. & Post, R.D.J. (1994). Information retrieval in the WorldWide Web: Making client-based searching feasible. In Proceedings of the First World-Wide Web Conference (pp. 183-192). New York: ACM Press.
20. Diligenti, M., Coetzee, F.M., Lawrence, S., Giles, C.L., & Gori, M. (2000). Focused crawling using context graphs. In Proceedings of the 26th. Conference on Very Large Databases (pp. 527-534). New York: ACM Press.
21. Ester, M., Grob, M., & Kriegel, H. (2001). Focused Web crawling: A generic framework for specifying the user interest and for adaptive crawling strategies. Retrieved March 3, 2010, from http://www.dbs. informatik.uni- muenchen.de/~ester/papers/VLDB2001.Submitted.pdf
22. Florescu, D., Levy, A.Y., & Mendelzon, A.O. (1998). Database techniques for the World-Wide Web: A Survey. SIGMOD Record, 27 (3), 59-74. (Pubitemid 128456040)
23. Fu, T., Abbasi, A., & Chen, H. (2008). A hybrid approach, to Web forum interactional coherence analysis. Journal of the American Society for Information Science and Technology, 59(8), 1195-1209.
24. Glance, N., Hurst, M., Nigam, K., Siegler, M., Stockton, R., & Tomokiyo, T. (2005a). Analyzing online discussion for marketing intelligence. In Proceedings of the 14th International World Wide Web Conference (pp. 1172-1173). NewYork: ACM Press. (Pubitemid 43218304)
25. Glance, N., Hurst, M., Nigam, K., Siegler, M., Stockton, R., & Tomokiyo, T. (2005b). Deriving market intelligence from online discussion. In Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (pp. 419-428), Chicago. (Pubitemid 43218304)
26. Glance, N., Hurst, M., & Tomokiyo, T. (2004, May). BlogPulse: Automated trend discovery for weblogs. Paper presented at the 13th International World Wide Web Conference Workshop on Weblogging Ecosystem: Aggregation, Analysis, and Dynamics, New York, NY. Retrieved March 3, 2010, from http://www.blogpulse.com/papers/www2004glance.pdf
27. Glaser, J., Dixit, J., & Green, D.P. (2002). Studying hate crime with, the Internet: What makes racists advocate racial violence? Journal of Social Issues, 58(1), 177-193.
28. Guo, Y., Li, K., Zhang, K., & Zhang, G. (2006). Board forum, crawling: A Web crawling method for Web forum. In Proceedings of the Conference on Web Intelligence (pp. 745-748). Washington, DC: IEEE.
29. Gustavson, A.T., & Sherkat, D.E. (2004, August). Elucidating the Web of hate: The ideological structuring of network ties among White supremacist groups on the Internet. Paper presented at Annual Meeting of American Sociological Association, San Francisco, CA.
30. Heydon, A., & Najork, M. (1999). Mercator: A scalable, extensible Web crawler. In Proceedings of the International Conference on the World Wide Web (pp. 219-229). NewYork: ACM Press.
31. Lage, J.P., Da Silva, A.S., Golgher, P.B., & Laender, A.H.F. (2002). Collecting hidden Web pages for data extraction. In Proceedings of the Fourth International Workshop on Web Information and Data Management (pp. 69-75). New York: ACM Press.
32. Lawrence, S., & Giles, C.L. (1999). Searching the World Wide Web. Science, 280(5360), 98.
33. Leuski, A., & Allan, J. (2000). Lighthouse: Showing the way-relevant information. In Proceedings of the IEEE Symposium on Information Visualization (125-130). Washington, DC: IEEE. (Pubitemid 32072849)
34. Li, Y., Meng, X., Wang, L., & Li, Q. (2006). RecipeCrawler: Collecting recipe data from WWW incrementally. In Proceedings of the 7th International Conference on Web-Age Information Management (pp. 263-274). Washington, DC: IEEE.
35. Limanto, H.Y., Giang, N.N., Trung, V.T., Huy, N.Q., & He, J.Z.Q. (2005). An information extraction engine for Web discussion forums. In Special Interest Tracks and Posters of the 14th International Conference on the World Wide Web (pp. 978-979). New York: ACM Press.
36. Lin, K., & Chen, H. (2002). Automatic information discovery from the "Invisible Web." In Proceedings of the International Conference on Information Technology: Coding and Computing (p. 332). Washington, DC: IEEE.
37. Menczer, F. (2004). Lexical and semantic clustering by Web links. Journal of the American. Society for Information Science and Technology, 55(14), 1261-1269.
38. Menczer, F., Pant, G., & Srinivasan, P. (2004). Topical Web crawlers: Evaluating adaptive algorithms. ACM Transactions on Internet Technology, 4(4), 378-419. (Pubitemid 40009828)
39. Najork, M., & Wiener, J.L. (2001). Breadth-first search crawling yields high-quality pages. In Proceedings of the World Wide Web Conference (pp. 114-118). New York: ACM Press.
40. Ntoulas, A., Zerfos, P., & Cho, J. (2005). In Proceedings of the Fifth ACM/IEEE-CS Joint Conference on Digital Libraries (pp. 100-109). New York: ACM Press.
41. Pant, G., & Srinivasan, P. (2005). Learning to crawl: Comparing classification schemes. ACM Transations on Information Systems, 23(4), 430-462. (Pubitemid 43909019)
42. Pant, G., & Srinivasan, P. (2006). Link contexts in classifier-guided topical crawlers. IEEE Transactions on Knowledge and Data Engineering, 18(1), 107-122. (Pubitemid 43145092)
43. Pant, G., Srinivasan, P., & Menczer, F. (2002, May). Exploration versus exploitation in topic driven, crawlers. Paper presented at the Second World Wide Web Workshop on Web Dynamics, Honolulu, Hawaii. Retrieved March 2, 2010, from http://www.dcs.bbk.ac.uk/webDyn2/proceedings/pant-topic-driven- crawlers.pdf
44. Raghavan, S., & Garcia-Molina, H. (2001). Crawling the hidden Web. In Proceedings of the 27th International Conference on Very Large Databases (pp. 129-138). New York: ACM Press.
45. Schafer, J. (2002). Spinning the Web of hate: Web-based hate propagation by extremist organizations. Journal of Criminal Justice and Popular Culture, 9(2), 69-88.
46. Sizov, S., Graupmann, J., & Theobald, M. (2003). From focused crawling to expert information: An application framework for Web exploration and portal generation. In Proceedings of the 29th International Conference on Very Large Databases (pp. 1105-1108). New York: ACM Press.
47. Smith, M. (2002). Tools for navigating large social cyberspaces. Communications of the ACM, 45(4), 51-55.
48. Srinivasan, P., Mitchell, J., Bodenreider, O., Pant, G., & Menczer, F. (2002, July). Web crawling agents for retrieving biomedical information. Paper presented at International Workshop on Agents in Bioinformatics (NETTAB), Bologna, Italy. Retrieved March 3, 2010, from http:// citeseerx.ist.psu.edu/ viewdoc/download?doi= 10.1.1.16.8948&rep=rep 1 &type=pdf
49. Whine, M. (1997). The governance of cyberspace: Politics, technology, and global restructuring. London: Routledge.
50. Yih, W., Chang, P., & Kim, W. (2004). Mining online deal forums for hot deals. In Proceedings of the Web Intelligence Conference (pp. 384-390). Washington, DC: IEEE.
51. Zhou, Y., Reid, E., Qin, J., Chen, H., & Lai, G. (2005). U.S. extremist groups on the Web: Link and content analysis. IEEE Intelligent Systems, 20(5), 44-51.