e-scribe.com

database-driven web development

This is a PNG image. If you're running an OS X browser that can't display it, please e-mail me.

Using Mailfilter on Mac OS X

Update (May 2003): After happily using mailfilter for many months and thousands of killed spam messages, I've moved on. While mailfilter is a great tool, I now believe that Bayesian filtering is our best bet in the spam war. Rule-based systems like mailfilter are much easier to game. Witness the decreasing effectiveness of the excellent and widely used SpamAssassin -- spammers are specifically tuning their messages to beat its rules. Bayesian systems are very effective, and are also much less hassle to maintain, because they constantly tune themselves with little or no human tweaking. Easier is important because, after all, the whole point of filtering spam is to save time and hassle. I'm leaving this page up so that people using mailfilter can still benefit from the information and links, but I am no longer a good target for mailfilter questions.

Update (November 2003): Recently I've started running my own mailserver. This has given me yet another turn in my thoughts on fighting spam. One factor is volume -- whereas before I was dealing with a few hundred pieces of spam per week, now with multiple domains and users it's a thousand a day and rising. I wanted to start rejecting spam at the earliest possible point, rather than accepting it and filtering later. I now block most spam attempts right at the door. There are technical details and fun stats on the e-scribe antispam page. I still find Bayesian filtering to the be the best solution on the client-side, and use bogofilter for end-user mail screening of messages that make it through the first lines of defense.

Update (July 2006): Daily spam attempts on my server are now in the 10,000 to 15,000 range. Thank goodness for Postfix and real-time blacklists.

Introduction

OS X opens up a lot of spam-killing options for the Mac user -- Unix-based tools that go beyond the filtering capabilities of most e-mail clients. Unfortunately for the Unix neophyte, many of these are rather daunting to install and configure.

Mailfilter, by Andreas Bauer, is one of the simplest of these tools, but is remarkably effective when set up right. It's essentially a command-line POP3 mail client whose only job is to kill spam. It connects to your POP account, compares the headers on waiting messages to its configurable list of rules, and deletes messages accordingly -- before your e-mail client even downloads it. You don't have to change any settings in your e-mail client or on the server where your POP account resides.

While you can of course write your own rules, you can also use those created and tested by others. My mailfilter setup, with about 300 rules, kills about 600 pieces of spam per week, and that number is rising. With my current rule-set, Mailfilter catches around 95% of the spam sent to me. I look at the stuff that slips through, and add appropriate new rules when possible. My sample ruleset is large, but fairly conservative. There are few Subject: filters, and I don't filter mail from major ISPs that happen to be on spam crusaders' shitlists. Many of my rules rely on the stupid crap that spammers put in the To: and From: lines, and miscellaneous other screwy headers.

(I suppose I should insert some sort of disclaimer here for those who want to use my sample rules: I have been careful to write rules that are unlikely to kill legitimate messages, but I cannot be held responsible if when using them you somehow delete a Very Important Message from Ed McMahon or your Aunt Mildred.)

Installing Mailfilter

If all this hasn't scared you off, here's how to proceed. It's not hard, but it does require that you are comfortable using the Terminal.
  1. Download the mailfilter software. (I used the binary version myself)
  2. Install mailfilter in /usr/local/bin per the included instructions
  3. Optional: Instead of starting from scratch making your own filtering rules, download my .mailfilterrc_sample file, add the necessary info for your POP account, rename it (mv .mailfilterrc_sample .mailfilterrc), and place it in your Home directory.
  4. Optional: Add a cron job that triggers a script like my spamcheck AppleScript, which runs mailfilter and immediately thereafter tells Eudora to check for mail. Edit your crontab (crontab -e) and add this line: 
    */30    *       *       *       *       osascript /Users/paul/Library/Scripts/spamcheck
    This will execute the "spamcheck" script every 30 minutes. (Replace the bit after "osascript" with the actual path to the script on your system.)
  5. Optional, for Entourage users (courtesy of Didier Barret): Entourage can execute an Applescript before checking mail. Create a one-line Applescript such as do shell script "/usr/local/bin/mailfilter", save it, and apply it like this (what, you don't speak French?)
  6. Optional, for Apple Mail users (courtesy of Helmut Reininger): You can use this version of the Spamcheck script.
  7. Optional: Download my "spamwatch" script and put it somewhere in your $path (for example, ~/bin). Make sure it's executable (chmod ugo+x spamwatch). Click the link to see the script and its brief self-description.
  8. Optional: Use the following hairy command line to produce a handsomely sorted report telling you which filters are seeing the most action. It assumes you are using a level of logging that records which filter was applied to a given message. 
    cat Library/Logs/mailfilter.log | grep "Deleted" | sed "s/.*Deleted.*\[Applied filter: '//g" | sed "s/' to '.*//g" | sort | uniq -c | grep -v "   1" | sort -r
    If you run this mini-script on your own logs, e-mail me the results!


Please let me know if you have trouble with these instructions, so I can improve them.

Life with Mailfilter

At the beginning of the day I launch the "spamwatch" script in its own Terminal window, then minimize it and leave it running all day. I have the "spamcheck" script running via a cron job every thirty minutes. Periodically I check the spamwatch output (much easier to scan than the raw mailfilter.log because of linewrap issues) to see if any real mail has been killed. If a legit message does get caught, I edit the .mailfilterrc file to fix the offending rule, or add an ALLOW= exception, then e-mail the sender and ask them to send the message again.

There are a couple rules I did not include in the sample ruleset because they need to be adapted to your name and e-mail address. The first one looks like this: DENY=^(To|CC):.*(pb@.*){3,} -- this kills anything that gets sent to a long list of generated addresses like "pb@e-scribe.com, pb@aol.com, pb@hotmail.com..." The second is this rule, which helps prevent false positives: ALLOW=^(To|Cc):.*paul\ bissex -- it is a rare spammer who actually knows your full name.

If you get anywhere near the amount of spam I do (or, god forbid, more), consider it an investment in your own cardiovascular health. Life with Mailfilter is good.

Other options

Also see

questions or comments?

home • blog