Scalable centralized bayesian spam mitigation with bogofilter

LISA 2004 - 18th Large Installation System Administration Conference

Volumn , Issue , 2004, Pages 1-20

  Blosser, Jeremy ^a   Josephsen, David ^a  

^a VHA

Author keywords

[No Author keywords available]

Indexed keywords

BAYESIAN CLASSIFICATION; CONTENT-BASED FILTERS; IMPLEMENTATION STRATEGIES; LEGITIMATE MESSAGES; LONG-TERM SOLUTIONS; MANAGEMENT ISSUES; MISCLASSIFICATIONS; SPAM FILTERING TECHNIQUE;

LIME;

EID: 79952052472     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. Allman, Eric, The State of Spam, https://db. usenix.org/events/usenix04/audio/allman.mp3, June 2004.
2. Baard, Mark, Going Upstream to Fight Spam, http://www.wired.com/news/print/0, 1294, 61971, 00.html, January 2004.
3. Bowers, Jeremy, Spam Filtering's Last Stand, http://www.jerf.org/iri/2002/11/18.html, November 2002.
4. Domingos, Pedro and Michael Pazzani, Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier, http:// www.cs.washington.edu/homes/pedrod/papers/ mlc96.pdf, 1996.
5. Dreyfus, Emmanuel, Mail-Filtering Techniques, http://www.onlamp.com/pub/a/onlamp/2004/05/ 20/mail_filtering.html, May 2004.
6. Raymond, Eric S., 2003 MIT Spam Conference: Lessons from Bogofilter, http://www.usenix.org/ publications/login/2003-06/openpdfs/spam.pdf, January, 2003. In this talk Raymond explained his rationale for developing Bogofilter; he was primarily interested in putting a tool in the hands of end users since it was believed that Bayesian methods would not work in a centralized fashion. This point was summarized by Chris Devers in the June, 2003 issue of ;login magazine as follows: "As good as Graham's Bayesian algorithm is, ESR felt - as did many of the other speakers - that the nature of your spam/ham corpus is much more significant than the relative difference among any handful of reasonably good algorithms. (Back to the oft-repeated point about how corpus effectiveness falls apart when used for a group of users, as opposed to individuals.)"
7. Graham, Paul, A Plan For Spam, http:// www.paulgraham.com/spam.html, August 2002.
8. Graham, Paul, So Far, So Good, http:// www.paulgraham.com/sofar.html, August 2003.
9. Jacob, Philip, The Spam Problem: Moving Beyond RBLs, http://theory.whirlycott.com/~phil/ antispam/rbl-bad/rbl-bad.html, January 2003.
10. Pollard, Jonathan de Boyne, SPF is harmful. Adopt it., http://homepages.tesco.net/~J.deBoyne Pollard/FGA/smtp-spf-is-harmful.html, 2004
11. Pollard, Jonathan de Boyne, No anti-UBM measure for SMTP-based Internet mail works, http://homepages.tesco.net/~J.deBoynePollard/ FGA/smtp-anti-ubm-dont-work.html#Bayesian, 2004.
12. Graham-Cumming, John, 2004 MIT Spam Conference: How to beat an adaptive spam filter, http://www. jgc.org/SpamConference011604.pps, January 2004.
13. Graham-Cumming, John, Fooling and poisoning adaptive spam filters, http://www.sophos. com/sophos/docs/eng/papers/WP_PMFool_US. pdf, November 2003.
14. Knowles, Brad, Considered Harmful: SPF . . ., http://bradknowles.typepad.com/considered_ harmful/2004/05/spf.html, May 2004.
15. Louis, Greg, Tuning Bogofilter's Robinson- Fisher Method - a HOWTO, http://www.bgl.nu/ bogofilter/tuning.html, April 2003.
16. Louis, Greg, Bogofilter Training: Comparing Full Training with Training-on-error, http:// www.bgl.nu/bogofilter/training.html, April 2003.
17. Louis, Greg, Bogofilter Training: Comparing Full Training with Training-on-error, part 2 http:// www.bgl.nu/bogofilter/training2.html, April 2003.
18. Louis, Greg, Bogofilter Parameters: Effect of varying s and mindev, continued: Comparison of results from different email sources, http://www. bgl.nu/bogofilter/smindev3.html, April 2003.
19. Louis, Greg, Is Bogofilter Scalable? http:// www.bgl.nu/bogofilter/scale.html, November 2002.
20. Louis, Greg, Token Redundancy and the Effective Size Factor, http://www.bgl.nu/bogofilter/ esf.html, May 2004.
21. Mertz, David, Spam Filtering Techniques: Comparing a Half-Dozen Approaches to Eliminating Unwanted Email, http://gnosis.cx/publish/ programming/filtering-spam.html, August 2002.
22. Paganini, Marco, ASK: Active Spam Killer http:// www.usenix.org/events/usenix03/tech/freenix03/ full_papers/paganini/paganini_html/node2.html# SECTION00022000000000000000, April 2003.
23. Robinson, Gary, Spam Detection, http://radio. weblogs.com/0101454/stories/2002/09/16/spam Detection.html, September 2002.
24. Robinson, Gary, Handling Redundancy in Email Token Probabilities, http://garyrob.blogs. com/handlingtokenredundancy94.pdf, May 2004.
25. Robinson, Gary, Spam Filtering: Training to Exhaustion, http://www.garyrobinson.net/2004/02/ spam_filtering_.html, February 2004.
26. Self, Karsten M., Challenge-Response Anti- Spam Systems Considered Harmful, http://linux mafia.com/faq/Mail/challenge-response.html, December 2003.
27. Ward, Mark, How to Make Spam Unstoppable, http://news.bbc.co.uk/1/hi/technology/3458457. stm, February 2004. This article references John Graham-Cumming's work in defeating Bayesian filters using other Bayesian filters. While Graham- Cumming found this method can work, his conclusion was that it is very costly and quickly blocked. Nevertheless, many administrators and weblogs continue to point to this experiment (and specifically this article) as "proof " that Bayesian filters can be easily defeated.
28. Zdziarski, Jonathan A., Bayesian Noise Reduction: Progressive Noise Logic for Statistical Language Analysis, http://www.nuclearelephant.com/ projects/dspam/bnr.html, February 2004.