×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Ask Slashdot: Speeding Up Personal Anti-Spam Filters?

timothy posted about 8 months ago | from the backup-your-email-in-case-of-rogue-filter dept.

Software 190

New submitter hmilz writes "I've been using procmail for years to filter my incoming mail, and over time a long list of spam patterns was created. The good thing about the patterns is, there are practically no false positives, and practically no false negatives, i.e. I see each new spam exactly once, and lose no legit mail. This works by using an external spam-patterns file, containing one pattern per line, and running an 'egrep -F' against it. As simple as this is, with a long pattern list this becomes rather slow and CPU consuming. An average mail currently needs about 15 seconds to be grepped. In other words, this has become quite clumsy over time, and I would like to replace it by a more (CPU, hence energy) efficient method. I was thinking about a small indexed database or something. What would you recommend and use if you were me? Is sqlite something to look at?"

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

190 comments

spamassassin (5, Insightful)

mdaitc (619734) | about 8 months ago | (#44721401)

have you tried spamassassin?

Re:spamassassin (2)

Scutter (18425) | about 8 months ago | (#44721681)

Latest News: 2011-06-16: SpamAssassin 3.3.2 has been released, a minor new release primarily to support perl-5.12 and later. Visit the downloads page to pick it up, and for more info.

Last update was more than two years ago. I know you can refresh your rule sets periodically, but is the software even still maintained?

Re:spamassassin (4, Informative)

dbIII (701233) | about 8 months ago | (#44721757)

There is still stuff going on in the dev version with an svn commit listed on August 30 2013.
http://spamassassin.markmail.org/search/?q=#query:%20list%3Aorg.apache.spamassassin.commits+page:1+state:facets

Re:spamassassin (4, Insightful)

wvmarle (1070040) | about 8 months ago | (#44721829)

Maybe the software is pretty much finished? In that case there's not much more to do - no new features to add, and sooner or later you'll run out of bugs to fix.

Re:spamassassin (2)

Scutter (18425) | about 8 months ago | (#44722037)

I don't think there's any such thing as "pretty much finished", especially with a piece of software involved in the arms race that is spam vs. filtering. There's only so much you can do with rules before you need to revisit your engine. Also, it's not just the software that's been stagnant for two years. The website itself hasn't been updated in as long. Not a single news item since 2011. The other respondent mentioned that dev is still active, but dev is not production. Dev is dev. Ever since Spamassassin moved to Apache, it's been pretty much dead.

Re:spamassassin (4, Informative)

bill_mcgonigle (4333) | about 8 months ago | (#44722053)

The rules sets are updated pretty frequently - that's where the front lines of the battle are. As others have said, the engine is pretty mature.

The question, I guess, is what do you want spamassassin to do that can't be expressed with the current rules language?

Re:spamassassin (1)

jgrahn (181062) | about 8 months ago | (#44722579)

I don't think there's any such thing as "pretty much finished",

There is; software designed according to "do one thing and do it well" ... for example the Unix cat(1) command is probably pretty stable by now. Same with fgrep(1).

especially with a piece of software involved in the arms race that is spam vs. filtering.

... but yeah, well, I don't know Spamassassin but I suspect it has broader and more loosely-defined goals.

Re:spamassassin (0)

Anonymous Coward | about 8 months ago | (#44721741)

Sendmail, MIMEdefang and spamassassin. Hell MIMEdefang alone is quite powerful.

Also, I'd like to point out to the guy that the fact that he submitted this question to slashdot, makes it look like he's asking "How do I shot web?"

Re:spamassassin (4, Informative)

wvmarle (1070040) | about 8 months ago | (#44721823)

Add greylisting to the mix. For me it stops approx. 90% of junk at the gate. That alone saves >90% of your server's spam workload (90% of the spam checker; a bit extra due to the mail server not having to process the mail at all).

Of course I don't know about legitimate mail but if someone is trying to send legitimate mail trough a spam-type minimised mail server that doesn't retry, that's their problem...

Re:spamassassin (2)

FridayBob (619244) | about 8 months ago | (#44721841)

On the mail servers I maintain, I employ SpamAssassin only a last resort because it is resource-intensive. Submitter hmilz's approach is not only resource-intensive, but also labor intensive, so I would never recommend it.

I've used Exim for my MTA since 2001 and my main defense against spam has always been to filter it out before SpamAssassin comes into play based on analysis of header information and checking against DNS black lists. Actually, the first thing I do is look for obvious fakes from a limited number of well-known domains: gmail messages that are not sent from a Google server, eBay messages not from an ebay.com server, etc. Such messages are rejected immediately. However, the bulk of the filters I've collected and developed over the years check a number of items: whether the sender's reverse DNS address is in order, the HELO is correct, whether the sending IP address or any domains mentioned in the header lines are blacklisted, whether the callout works and any DKIM signature is valid, if an RFC-compliant date and To are included, whether any attachments are included with file types that I consider risky (e.g. ,bat, .btm, .cmd, .com, .cpl, .dat, .dll, .exe, etc.), if the message headers contain non-ASCII or characters from some unspecified character set, whether any SPF record says that the sending server really is authorized as an MX for the sender's domain, and finally if the incoming message is using one of my domains in its message ID. For all of these types of checks I often have multiple filter statements.

In the past I would usually reject messages that matched any of these filters. I would hardly ever receive any spam, but would see lots of false-positives, so I had to maintain very long white lists. When I finally got tired of that, I modified the above filtering system so that each filter was categorized. Each category has a variable that starts out as zero, but gets changed to a one with a match for any of the filters in that category. Later in the process the system counts the number of category variables that equal one. Generally, I figure "three strikes and you're out" is a good rule to apply.

Moreover, my MTA configuration works with a spambox system. For instance, if an incoming message scores only one or two category matches, and/or the message scores less than a certain number of SpamAssassin points, then it gets deposited in the user's spambox instead of their inbox. I've been running the four MTA's in my care like this for the last three years and they've been very reliable and almost totally free of maintenance. They don't require very much in the way of resources either. But best of all, I've had no more complaints from the users at all.

Re:spamassassin (0)

Anonymous Coward | about 8 months ago | (#44722219)

But best of all, I've had no more complaints from the users at all.

That's a clear warning sign. Users always complain. Better recheck your rules to see if you're dropping all mails from your users.

Re:spamassassin (1)

whoever57 (658626) | about 8 months ago | (#44721959)

have you tried spamassassin?

Indeed. I just looked at logs on a server that acts as an incoming mail filter for a small company. The range of times for spamassassin (spamd) to filter the incoming emails was about 1 to 7 seconds. with most being in the range of 2-4 seconds. This is without bypassing spamd for large emails (spam can be relied upon to be small)

Re:spamassassin (0)

pongo000 (97357) | about 8 months ago | (#44722155)

have you tried spamassassin?

Don't follow this advice. SA has become so slow that it's almost useless. On a VM with 1GB RAM, it takes anywhere from 15-60 seconds to process a single e-mail, and is an incredible resource hog. I've been running SA for years, run the latest stuff, and have pretty much done every tweak imaginable. And the default rules are about useless now as well: The scores are set so low that you have to set a low threshold, increasing your false positive rate. About 50% of the mail on my mail server (personal use, maybe 200-300 inbound messages a day, 90% spam) just gets passed due to spamd timing out.

Unfortunately, there appear to be no decent alternatives out there. Greylisting is nice, but spammers are wising up to it, and simply resend spam. There was a time about 3-4 years ago that zero spam came through (same inbound volume)...now, it's more like 5-10 a day. Not that I'm complaining. My point being that switching over to SA will not solve any of the submitter's resource woes with procmail.

Re:spamassassin (1)

xrayspx (13127) | about 8 months ago | (#44722339)

There's a lot to do to SA to make it "good". I shared your opinion a year ago. I run a relatively low volume personal mail server for a few domains and a few users. I had SA, but it didn't do much, and I had bigger fish to fry dealing with much larger mail sites than my stupid personal nonsense. I typically get about 300-500 spams a day, and very few legit mails. I was getting false positives, so I'd just never see the mail, and tons of false negatives. About 20% of the daily spam was hitting my inbox, making it unlikely that I'd ever even check my personal mail. If you mailed me, and I didn't have an existing filter from you, there was maybe a 60% chance I would notice your mail in time for it to matter.

I decided one day to fix all this, regardless of what that entailed. I lowered the threshold for SA to a score of 4 (which they bark at you not to do, but fuck 'em, I've seen maybe 6 legit mails with a score higher than 4.5, in my world anyway). The key components were: enabling remote checks, RAZOR and DCC, and having SA train its filters off of my false negatives. I use the Train SA [crazysquirrel.com] script, so I drop any false negatives in a Train Spam folder, and this picks them up and runs them through SA's filters to train it.

My false negative rate dropped pretty much immediately from 20% to ~3% to 5% on weekdays, and zero to 1% on weekends, which I can live with. In the year or so since I actually put my back into fixing this, I've gotten maybe 2 false positives.

I don't see long processing times, mail comes through pretty much as I send it in my tests on my VPS, but again, I only get a few hundred mails/day. If I had volume over a few dozen thousand/day, I'd probably just bite the bullet and pay Google (Postini) to make it go away.

Or... (0)

bmo (77928) | about 8 months ago | (#44721405)

You could route everything through gmail and wash out the spam.

Gmail's spam detection is spectacular.

inb4 gmail hate.

--
BMO

Re:Or... (0)

Anonymous Coward | about 8 months ago | (#44721415)

Then why isn't Postini's? We use it at work, and it can't seem to deal with any sort of Chinese spam or most phishing attempts.

Re:Or... (4, Informative)

bmo (77928) | about 8 months ago | (#44721573)

Well, the OP wasn't exactly clear if it was just his personal account or whether it's a corporate server. My "least amount of work" thing is to forward every email address I have to gmail, and pull mail from there via imap. I get a few hundred spams a day just on one mail account, and I haven't lost any real mail due to Gmail's filtering.

>postini

You do realize that is being EOLed, yes?

http://postini-transition.googleapps.com/ [googleapps.com]

>why gmail filters better than postini

Probably separate spam databases. Stuff like that happens. Gmail probably gets orders of magnitude more spam to "teach" the system.

YMMV.

Using grep and procmail is the stone-knives-and-bearskins approach to filtering. There are a lot of other filtering systems that will be much more efficient on Unix systems. He can begin by using greylisting to filter out the non-compliant "fire and forget" spambots and then filter the winnowed pile o' crap. At least greylisting's not server intensive (it throws the load back to the sender) since 5xx and 4xx errors are cheap.

Also blocking mail from dynamic IPs is a good idea.

At this point he can then run the mail through a series of weighted RBLs. Reach a certain score and it's tossed. That's the processor intensive bit, but it's at the end and non-intensive filtering has already happened.

--
BMO

Re:Or... (4, Interesting)

dbIII (701233) | about 8 months ago | (#44721821)

and I haven't lost any real mail due to Gmail's filtering.

Email from people at one site I look after used to vanish into a black hole at gmail until I convinced them to replace the GIF of their corporate logo attached to all their emails with a PNG version. That's some real mail lost due to gmail's filtering.

IMHO it's better to do the filtering somewhere where you have access to the stuff that is discarded. False positives may be rare now but they still happen. That's why I like stuff such as MailScanner (open source wrapper for spamassassin+your choice of commercial antivirus and/or clamav+other open source stuff+distributed updating rulesets) run on site. There's plenty of others that give you this function including some of the commercial "appliances" and outsourced email filtering.

Also blocking mail from dynamic IPs is a good idea.

It used to be the case that one IP address I have a mail server on would get blocked for a couple of days every year because some idiot at a blacklist would load in an obsolete list of dynamic IP addresses from what is now a decade ago. As IPv4 addresses diminish expect the lists of dynamic addresses to become outdated very quickly.

Re:Or... (0)

Anonymous Coward | about 8 months ago | (#44722127)

Email from people at one site I look after used to vanish into a black hole at gmail until I convinced them to replace the GIF of their corporate logo attached to all their emails with a PNG version. That's some real mail lost due to gmail's filtering.

Bullshit. Google either refuses the mail during the SMTP transaction or accepts it and queue's it, then it either delivers it to your inbox or delivers it to your spam folder. Gmail does not black hole emails. It's more likely that the senders were ignoring/deleting the bounces, the sender's mail server administrator was too lazy or incompetent to look at server logs, or you were too stupid to check your spam folder.

It used to be the case that one IP address I have a mail server on would get blocked for a couple of days every year because some idiot at a blacklist would load in an obsolete list of dynamic IP addresses from what is now a decade ago. As IPv4 addresses diminish expect the lists of dynamic addresses to become outdated very quickly.

I'll bet that IP has a generic hostname in rDNS, too. If you're gonna run a mail server but can't be bothered to set and maintain a non-generic hostname, don't expect your IP to stay off dynamic IP lists. How do you think those IPs ended up on the list in the first place?

Re:Or... (1)

dbIII (701233) | about 8 months ago | (#44722517)

OK then - maybe dumped them into the recipient's spam folder, all I know is the recipients never became aware of the emails until the GIF files were changed to PNG. Maybe gmail policy has changed since or maybe they only said they looked in their spam folders but never actually did. Either way the emails were not getting read if they had a GIF on them.

How do you think those IPs ended up on the list in the first place

All your insulting bets are wrong, all that happened is some IP addresses got reassigned quite a few years ago. What used to be dynamic with one ISP became static with the company that bought them out.

Re:Or... (2)

pongo000 (97357) | about 8 months ago | (#44722171)

At this point he can then run the mail through a series of weighted RBLs.

Fuck you and your RBLs. RBLs are a draconian solution that do immeasurable damage to those of us who (1) aren't spammers, and (2) choose to run our own mailservers on business-class IPs. I can't tell you how many times various IPs I use for outbound mail (I run several mailing lists) end up on an RBL for absolutely no fucking reason.

Oh, because someone in the same /24 block sent spam? Really? That's a good reason to block an entire /24 subnet?

RBLs are a solution in search of a problem. Some of them are nothing more than moneymakers for the people that run them: In order to get off their list, they blackmail you into paying money.

Want to do the world a favor? Don't use RBLs. You'll just end up finding yourself blacklisted at some point anyway.

Re:Or... (1)

Anonymous Coward | about 8 months ago | (#44721505)

GMail's false positive rate is absurdly high.

Re:Or... (1)

YukariHirai (2674609) | about 8 months ago | (#44721693)

Is it? I've never had a false positive in all the years I've been using GMail.

Re:Or... (3, Interesting)

Count Fenring (669457) | about 8 months ago | (#44721711)

Is it? I've never had a false positive in all the years I've been using GMail.

That you noticed. There's a fairly high bias inherent there; it just has to not have hit something that was both noticeable and that you knew was incoming.

Re: Or... (0)

Anonymous Coward | about 8 months ago | (#44721759)

Um... there's a "spam" folder that one can check, too. I have, and I do, but I'm not noticing any false positives.

Re:Or... (1)

Nemyst (1383049) | about 8 months ago | (#44722149)

In years of using the service I've never once been told that I'd missed an email by anyone I know. I'll take that as a sufficient confirmation that I haven't missed anything important. Things that I have missed which people haven't followed up on or notified me of are just about as good as junk mail anyway.

Re:Or... (2)

CaptQuark (2706165) | about 8 months ago | (#44722431)

Last night I sent my Gmail account an email from my ISP email system, then waited for it to show up. Nothing. So I resent it. Second time nothing.

The email contained two screen captures I needed at the office. The subject line was "Steve on telework". Nothing obvious that would trip Postini's spam filter. It is now 24 hours later and neither has shown up. I wonder how many other emails I don't get.

~~

Re:Or... (0)

Anonymous Coward | about 8 months ago | (#44721525)

Outsourcing the management and storage of your emails to Google's big ears when you are capable of doing it yourself is definitely stupid.

Re:Or... (1)

FridayBob (619244) | about 8 months ago | (#44721861)

And allow Google and the US government to scan all of my mail? No thanks. The same goes for Hotmail, Yahoo and any other commercial email service provider (and certainly those based in the US). These days it makes more sense than ever to maintain your own MTA.

Re:Or... (1)

vux984 (928602) | about 8 months ago | (#44722103)

And allow Google and the US government to scan all of my mail?

Well, routing it outside of gmail keep gmails hands off it, well the half that doesn't originate with them to start with. But i think you'll need to do quite a bit more to keep the us govt out of it.

Re:Or... (1)

macraig (621737) | about 8 months ago | (#44722151)

Gmail's spam detection is spectacular.

Are you new here? Its false positives are equally spectacular. Some days I could swear there's some mean bored Google sysadmin who happens to be a less-than-friendly coworker from my past who's just sitting there randomly applying the Spam label to messages in my Inbox.

CRM114 (1)

Anonymous Coward | about 8 months ago | (#44721413)

Look up CRM114.

You could speed up your current solution (5, Interesting)

russotto (537200) | about 8 months ago | (#44721419)

Write something that uses a regular expression library (RE2 would be ideal, if your expressions are actually regular), and keeps the compiled patterns resident. Most of your time is likely spent parsing the patterns.

Re:You could speed up your current solution (5, Informative)

PetiePooo (606423) | about 8 months ago | (#44721541)

...Most of your time is likely spent parsing the patterns.

I second that. And as your rules have built up, there are likely some that have never been used beyond when they were first put in. I'd instrument your next solution to identify outliers and cull them over time so your parser doesn't have to work so hard.

Re:You could speed up your current solution (1)

grcumb (781340) | about 8 months ago | (#44722101)

Write something that uses a regular expression library (RE2 would be ideal, if your expressions are actually regular), and keeps the compiled patterns resident. Most of your time is likely spent parsing the patterns.

I'm probably going to get shat on by kids who don't know any better, but....

Use Perl. If a complex set of regular expressions is taking 15 seconds per email, then there's clearly something wrong with the implementation. I suspect you're doing too much backtracking [regular-expressions.info] . I've been guilty of the same in the past. In one case, simply anchoring my regular expressions to the start and end of the string reduced running time literally by two orders of magnitude. Just glom the whole message into a string and go nuts.

And before someone makes a 'write-only' joke about Perl regular expressions, I'd suggest you take a look at Perl 6 regex grammars [wikipedia.org] , which provide you with the ability to lay out complex rulesets with ease - and makes them vastly easier to read.

As with any programming issue, it's horses for courses [wiktionary.org] , and when it comes to parsing text with regular expressions, Perl is still at the head of its class.

Re:You could speed up your current solution (1)

niftymitch (1625721) | about 8 months ago | (#44722211)

Write something that uses a regular expression library (RE2 would be ideal, if your expressions are actually regular), and keeps the compiled patterns resident. Most of your time is likely spent parsing the patterns.

Yes but a more resource friendly set of tools might begin with the OP's procmail to move the mail
onto a local machine quickly. Filters inside of procmail are hobbled. Do this as one message
per file (http://unix.stackexchange.com/questions/62563/savings-emails-as-individual-files-using-procmail).
Procmail locks and gates are OS dependent but still slow.

Next test each message with one or more simple "grep expressions" that then pass it or gate it
to more complex expressions. On a multi core machine with a SSD disk this might be quick.

Now move or pull the files to a location for a human reader in folders or dirs or what ever
the reader expects.

Better filters do exist and are well recommended. You can teach them honey pot style with a public account on gmail
and slurp the service discovered spam into a training file. To some degree there is a need to isolate but not delete
trouble files so you can retrain the filter.

Do watch out for mime attachments where the content differs by reader/ tool. A safe text reader can
be fooled because an unsafe pile of poo is wrapped in a mime that does not speak. Then a rich visual tool
handy dandy mail tool will open the trouble payload based on the text content.

Current solution is awful (1)

goombah99 (560566) | about 8 months ago | (#44722371)

Here's several things you can do to make this faster.
1) first don't keep invoking egrep. this has to parse the command line and then re-load the egrep command itself every time. Instead do this from within a loaded program. Perl is a very good choice for this
2) the perl command can pre-compile the regular expression. So you can leave the perl program running as a process then simply feed it new data to analyse.
3) given you are searching for words, you probably want to split the incoming stream on white space one-time not every time.
4) even better than that, take the e-mail, parse it to words, then parse each word into all 3,4,5,6,7,8 consecutive strings. Then just look these up in a hash table.
5) if you are only trying to match from the start of the word, (not interior word strings) then this hashing becomes trivial.

Database? (2, Insightful)

K. S. Kyosuke (729550) | about 8 months ago | (#44721421)

What would the database achieve? I'm not sure what is the exact nature of the patterns (an example would really help here), but perhaps writing a compiler from the patterns into some decision procedure in something reasonably efficient yet featuring quick start, such as SBCL or Gambit, could help.

bogofilter (4, Informative)

jon787 (512497) | about 8 months ago | (#44721441)

http://bogofilter.sourceforge.net/ [sourceforge.net]

I haven't timed it to see how well its been doing in the 6 years I've had it though.

Re:bogofilter (1)

SigmundFloyd (994648) | about 8 months ago | (#44722571)

http://bogofilter.sourceforge.net/

Seconded. Procmail + bogofilter + spam.mbox = no problem.
I keep - and periodically review - a "spam" mbox for the rare false positive.

I haven't timed it to see how well its been doing in the 6 years I've had it though.

It's written in C, so it's very likely much faster and leaner than Spamassassin.

tokens and a lookup table (0)

Anonymous Coward | about 8 months ago | (#44721449)

a big one. 15s per email???
holy smokes you are so fired.

What a forking awful solution... (2, Informative)

Anonymous Coward | about 8 months ago | (#44721455)

Sorry, couldn't resist the pun.

Your problem (besides not using existing Bayesian tools...) is that every single egrep is a fork. As others have pointed out, you should rewrite your script in something like Python and use the native regex libraries. Even if you have to read and 'compile' the regex list every time, you're saving a *massive* amount of OS-level overhead.

Re:What a forking awful solution... (1)

Anonymous Coward | about 8 months ago | (#44721835)

Or you could use a more pleasant language for regexp, like perl. It's also faster.

Distribute (1)

manu0601 (2221348) | about 8 months ago | (#44721463)

It seems you could easily distribute the load on multiple machines, each doing a subset of the regex.

Re:Distribute (1)

Anonymous Coward | about 8 months ago | (#44721499)

There's something hilarious about having to distribute email filtering across several machines.

Re:Distribute (1)

Cryacin (657549) | about 8 months ago | (#44721867)

Yeah, just use "the cloud" to "drag and drop" your email into, so that you can "sanitise your synergy" and "reclaim your potential".

Phew, enough markitechture for one day. I have to go shower now.

ragel (2, Interesting)

Anonymous Coward | about 8 months ago | (#44721471)

Try compiling your patterns using Ragel: http://www.complang.org/ragel/

Union them all together and you'll see orders of magnitude improvement in performance (e.g. 10x - 100x) over other regular expression engines, although GNU grep is using Aho–Corasick with the -F switch, so you're likely to see less of an improvement.

Many people use re2c, but it has nowhere near the performance or capabilities of Ragel. Ragel has a steep learning curve, but it's well worth the effort to master. It's well maintained, and has been for years.

Re:ragel (0)

Anonymous Coward | about 8 months ago | (#44721567)

I should mention that I include RE2 in that "orders of magnitude" improvement statement. RE2 falls down pretty quickly under memory pressure when you try to union too many expressions together. You can union thousands of patterns together using Ragel, whereas RE2 can blow up after dozens.

Grey Listing and zen.spamhaus.org (0)

Anonymous Coward | about 8 months ago | (#44721479)

15 seconds per email? That must be one heck of a pattern list. I used to rely on procmail for filtering. In simpler times it did everything I needed.

First of all, setup grey listing. 99.99% of the emails you're receiving never make it past grey listing. You can nearly forget about filtering again once grey listing is enabled.

Second add a reject client like zen.spamhaus.org to your mail server to stop the emails that make it past grey listing.

You can continue to filter anything that makes it past those two barriers, but I think you'll find your filters are redundant at that point. In fact you can probably cut procmail from the process entirely, unless you use it do other stuff with the mail.

Re:Grey Listing and zen.spamhaus.org (1)

sumdumass (711423) | about 8 months ago | (#44721703)

I second grey listing and using spamhause filter lists.

It is well worth it. I had a 30 user environment receiving about 100 to 300 spam emails a day per user go to approximately 10 a piece making it through. Then when I activated the spam filtering it went to about 10 a week for about 1/3 of the users. The biggest problem is third party user machines being compromised and the spam being sent through their internet's email servers (grey listing doesn't stop legitimate servers and most big ISP's don't make it on the spamhaus lists). Usually this contains a virus attached to it and the antivirus on the mail server catches it.

PS.. the reason the spam was so high is because they sign up for all sorts of crap from their work computers and use their work email almost as a personal email. The partner's don't mind as long as their work is getting done. The biggest offenders were the partners (read owners) themselves.

Re:Grey Listing and zen.spamhaus.org (1)

dbIII (701233) | about 8 months ago | (#44721939)

Yes, used greylisting for a couple of days and now am well aware of an inherent flaw that could cost the people who use it their jobs. Consider how it works and then consider that people at the top of organisations like to think of email as a nearly instant communications system and really don't like it when that last minute tender they've been working on all night gets delayed for half an hour (as is typical, or several hours with insane greylisting settings I've seen used) just because the person they sent it to has never had email from them before. It's a nice idea from an IT perspective but from a business perspective it sucks dog balls. A lot of places have far too many of those edge cases to make it worthwhile. Other places are more slow moving and it just doesn't matter if the most urgent email arrives a few hours late, but good luck trying to explain why if you're not in one of those places :)
Reducing the time in greylisting does reduce the potential damage but then it's a balance between the patience of the people that expect instant communications and the patience of spammers. While most spambots only tried once in 2005 things have moved on since - greylisting is well behind the spam arms race.

grey listing doesn't stop legitimate servers

It stops them long enough for it to be a problem in enough cases that I kept getting a lot of "why doesn't X have my email yet" phone calls when greylisting first became popular. I then ran it myself for a while to see what was going on and to see where some of those who were using it were applying frankly insane settings, and how even less tight settings were problematic on occasion.
Sometimes it's better to look at entire systems to resolve problems instead of a tightly focused technical only approach. If you guys are going to call yourselves "engineers" you should act like them and consider entire systems instead of single bolts or what the manual tells you to do. Cute tricks that fuck around with communication policy shouldn't be used unless you can take the consequences of changing communication policy. If it's going to put your boss on the carpet in front of the CEO you have a duty to your boss of explaining to them why you are doing it.

Re:Grey Listing and zen.spamhaus.org (1)

sumdumass (711423) | about 8 months ago | (#44722027)

Yes, used greylisting for a couple of days and now am well aware of an inherent flaw that could cost the people who use it their jobs. Consider how it works and then consider that people at the top of organisations like to think of email as a nearly instant communications system and really don't like it when that last minute tender they've been working on all night gets delayed for half an hour (as is typical, or several hours with insane greylisting settings I've seen used) just because the person they sent it to has never had email from them before. It's a nice idea from an IT perspective but from a business perspective it sucks dog balls

The default resend interval on most mail systems is between 2 and 15 minutes that I know of. If you are using it, it doesn't impact anyone you are sending to. If who you are sending it to is using it, it is their problem not yours.

I'm also surprised by the comment of " never had email from them before". First, are you confusing grey listing with white listing and a challenge response? Second, I'm not sure I would be sending something I worked on all night to someone I never communicated with before by email. It might be possible that the specific person is a different person, but the email should work domain wide (if I emailed your secretary I should be able to email you without the grey listing).

It stops them long enough for it to be a problem in enough cases that I kept getting a lot of "why doesn't X have my email yet" phone calls when greylisting first became popular. I then ran it myself for a while to see what was going on and to see where some of those who were using it were applying frankly insane settings, and how even less tight settings were problematic on occasion.

If grey listing stops the email from being received for hours, there is something wrong with the server sending it. Most default times are minutes. I'm not sure I know of a scenario that would require specific minute by minute communications that wouldn't warrant a phone call or something more instantaneous. Even without grey listing, the emails can be delayed for several minutes. Most email clients do not update more then every couple minutes anyways.

The longest I have seen an email delayed from grey listing is about 5 minutes. Anyways, I'm not sure we are talking about the same things. Grey listing simply drops the first connection attempt by a server not in a white list and requires the smtp server to retry at their set interval. There aren't a whole lot of setting you can do besides specifically allowing domains or servers and specifically denying them. The person sending the email might have their servers jacked around but that is their problem, not yours. If a route goes down somehow and the communication is interrupted, it will retry the communications anyways- this is no different except it's on purpose.

Sometimes it's better to look at entire systems to resolve problems instead of a tightly focused technical only approach. If you guys are going to call yourselves "engineers" you should act like them and consider entire systems instead of single bolts or what the manual tells you to do. Cute tricks that fuck around with communication policy shouldn't be used unless you can take the consequences of changing communication policy. If it's going to put your boss on the carpet in front of the CEO you have a duty to your boss of explaining to them why you are doing it.

I'm not convinced we are talking about the same things here. Like I said, the longest I have seen an email delayed is about 5 minutes. I say about because the log measures seconds between connection attempts when I check them and I have never seen them go over 300-350 seconds unless something was wrong with the route (traceroute fails to complete).

Re:Grey Listing and zen.spamhaus.org (1)

dbIII (701233) | about 8 months ago | (#44722501)

Please stop roleplaying someone stupid with your current game of presenting the incorrect suggestion that the greylisting time set on the recieving server doesn't matter. You're misleading the newbies that haven't worked out from your handle and posting history that you like to pretend to be dumb as shit and offer utterly stupid suggestions as if they are viable. People playing games like yours make it difficult to have an honest and factual discussion in this place.

Re:Grey Listing and zen.spamhaus.org (0)

Anonymous Coward | about 8 months ago | (#44722549)

Sometimes it's better to look at entire systems to resolve problems instead of a tightly focused technical only approach. If you guys are going to call yourselves "engineers" you should act like them and consider entire systems instead of single bolts or what the manual tells you to do. Cute tricks that fuck around with communication policy shouldn't be used unless you can take the consequences of changing communication policy. If it's going to put your boss on the carpet in front of the CEO you have a duty to your boss of explaining to them why you are doing it.

You sound like the kind of prick who has jacked around their MS Exchange server with 'cute tricks' to the point that it can't properly respond to standard mail communication protocols anymore. Or maybe you just don't know how to do anything beyond clicking the 'start mail server' button and left the settings on defaults so that it only ever plays nice with other Exchange servers. Then when your boss misses an important email you blame it on greylisting so you don't get your ass wiped up and down the hallway for being the incompetent ass that you are. You clearly don't understand how grey listing works any more than you understand the how the 'entire system' works.

Go back to designing web pages and stop posing as some kind of IT admin.

Short circuit (1)

Ben Thul (2851611) | about 8 months ago | (#44721489)

Does your process require that all of the regexes are tried in turn or is it the case that if it hits one of your patterns that it's marked as spam? If the latter, are you able to rank the patterns from most likely to least likely to be matched? And, if so, can you stop your process once a match is made? If all of those things are true, then you should be able to cut the time/CPU/energy required to do the filtering

Re:Short circuit (2)

CanadianMacFan (1900244) | about 8 months ago | (#44721625)

You might also want to look at how patterns are added to the file too. If they are added to the end then the latest spam of the day message will need to parse all of the patterns until it hits the latest pattern. Of course ideally you might want to set something up that looks at the hits each pattern gets so that you could parse the most likely patterns first followed by the latest patterns.

dspam (1)

Rob Bos (3399) | about 8 months ago | (#44721515)

Consider using a proper learning filter, like dspam. You can pipe it through procmail just as easily, and you can feed your corpus of spam into it. You won't get 100%, but it'll recognize spam you haven't seen. :0f
*
| /usr/bin/dspam --deliver=stdout

Re: dspam (1)

paulc (5348) | about 8 months ago | (#44722363)

I'll give a +1 for dspam. I run it on a couple of accounts via procmail under qmail and it works really well. When spam gets through to my inbox I just move it to a spam training folder and an hourly cron task passes that folder's contents back to dspam for training.

Re:dspam (1)

mishehu (712452) | about 8 months ago | (#44722521)

If I had mod points I'd have given you a +1 as well too. I've been using dspam for my own systems as well as clients' systems for years now, with MySQL as the backend (InnoDB tables though, not MyISAM). The only downside is that it can end up eating a fair amount of filesize, but it's extremely fast and highly accurate. Combine that with other methods like RBL, spf checks, dk, etc., and I get but a false-positive once every 3 months or more, and a false-negative once every 6-12 months.

SHR Spam Hit Rate 98.48%
HSR Ham Strike Rate: 0.23%
PPV Positive predictive value: 99.93%
OCA Overall Accuracy: 98.77%

And this is with the same database for the past 3-5 years or so now.

perl or python or whatever (1)

retchdog (1319261) | about 8 months ago | (#44721517)

I've heard, but never timed it myself, that perl is faster for regexp-type stuff than even the specialized tools, just from the massive amount of optimization it has accrued over the years; here [perlmonks.org] is a completely unbiased source. Use a perl or python script, and consider using Storable (perl) or pickle (python) to serialize the data structure, I guess, but just having the whole list in memory will help.

According to this [uidaho.edu] , perl regexps are (unsurprisingly) a superset of egrep's.

I don't see how introducing SQL could do much to help speed, or anything else, in this application.

Re:perl or python or whatever (1)

Jmc23 (2353706) | about 8 months ago | (#44721569)

regexps in cl-ppcre are faster than perl.

Re:perl or python or whatever (2)

retchdog (1319261) | about 8 months ago | (#44721579)

doing anything but repeated egreps is probably fast enough. he should do whatever is easiest, which probably isn't lisp.

Re:perl or python or whatever (1)

Jmc23 (2353706) | about 8 months ago | (#44721653)

Don't bring your prejudices into this!

It doesn't get much easier than someone not only handing you the code but also holding your hand and walking through every single function. Unless you want to use a magic black box and where's the fun in that?

Matching multiple simultaneous regular expressions (2)

careysb (566113) | about 8 months ago | (#44721575)

Many years ago I worked with a Unix development tool called LEX that could handle matching multiple patterns simultaneously. Perhaps there is an updated tool that would do the same thing. Java has a 3rd party library called ANTLR that might do the trick. It would involved re-compiling every time a new pattern is added but it should be extremely fast.

Sqlite will be awesome (2)

swillden (191260) | about 8 months ago | (#44721577)

Sqlite, or anything that uses an index, will be screaming fast.

Your statement of your current solution makes me wonder, though.. are you using "egrep -F -f pattern_file e_mail_message"? Or are you running egrep many times, once per line of the pattern file, or once per line of the message? I would think that given a pattern file egrep would be smart enough to do something better than repeatedly scanning the input, but based on the time it's taking, it sounds like that's happening.

Re:Sqlite will be awesome (1)

neonsignal (890658) | about 8 months ago | (#44722031)

I doubt that he is using "grep -F -f ...", because fgrep can search for a hundred thousand patterns in a megabyte of data in under a second even on a modest machine (and most of the time is building up the regex state machine). I suspect he is using "egrep -f", and lots of patterns with wildcards. Worse, he will be running it once on each email, which means rebuilding the regex state machine each time.

Re:Sqlite will be awesome (0)

Anonymous Coward | about 8 months ago | (#44722083)

"..using an external spam-patterns file, containing one pattern per line, and running an 'egrep -F' against it."

Sadly more likely, he is iterating the patterns file and grepping the email one pattern at a time.

Re:Sqlite will be awesome (1)

idunham (2852899) | about 8 months ago | (#44722499)

Agreed.
egrep -F is the same as fgrep, and it uses fixed strings.
If he says "patterns", it's obviously egrep or grep, probably egrep -f.
egrep -f patterns -lr maildir is likely to be faster, because of startup costs.

Whitelist first (0)

Anonymous Coward | about 8 months ago | (#44721637)

You could run whitelisting rules first to allow messages that are obviously non-spam through without them having to pass through all of the spam rules. This could be the standard address book whitelisting so all of your friends' and colleagues' messages pass immediately.

For a bit more complex solution you could run messages through something like SpamAssassin first -- for any messages that have a spam score above a certain threshold you run them through your custom rule set. Since you have a high degree of trust in your rule set you could make this threshold quite low -- again mainly so SpamAssassin will just act as a whitelist to let clearly good messages through immediately.
 

Easy way to handle spam... (1)

NotQuiteReal (608241) | about 8 months ago | (#44721663)

Just route everything from Facebook, LinkedIn, my dad, Apple and "i*" to the spam folder, and most of it is covered.

Oneword (0)

Anonymous Coward | about 8 months ago | (#44721719)

Junkemailfilter

Http://www.junkemailfilter.com

Outsource

Problem spotted. (4, Insightful)

girlintraining (1395911) | about 8 months ago | (#44721735)

The problem is that you're using egrep in the first place. Here's the thing -- the overwhelming majority of your cycles are getting sucked loading, initializing, executing, then unloading, that thread. It's not that using regular expressions is processor-intensive... it's that repeatedly launching the same executable is.

Use something that can load once, read in the patterns, check all the e-mails that are queued, sort them, then exit. Your execution time will go from 15 seconds to 150 milliseconds.

Re:Problem spotted. (3, Interesting)

complete loony (663508) | about 8 months ago | (#44721793)

If you have sufficient programming experience, I'd recommend basing this solution on redgrep [google.com] . It's an llvm based expression compiler that should be able to combine multiple expressions into a single machine code state machine, assuming it doesn't run out of memory in the process. With a bit of effort you could output all of your compiled expressions into a single executable so you'll only need to wait for the compilation time when you add more filters.

Re:Problem spotted. (1)

arth1 (260657) | about 8 months ago | (#44722095)

You mean like doing an egrep +F instead of multiple egreps? I sure hope he already does.

Re:Problem spotted. (0)

Anonymous Coward | about 8 months ago | (#44722581)

this is what I was thinking. egrep loading a pattern matching file once should be much much faster than 15 seconds. launching an egrep for every line in a pattern file, and then matching it against the email is a terrible way to do this.

Procmail is a fine tool -- but the wrong tool (5, Informative)

Arrogant-Bastard (141720) | about 8 months ago | (#44721801)

If spam has made it far enough that it's actually reached your personal instance of procmail, then there's been a problem earlier in the chain. Procmail rulesets should be a last resort, and they should only be asked to deal with minor issues that aren't dealt with via earlier rulesets.

The first line of defense are your perimeter routers. They should implement BCP 38, they should block bogons, and they should bidirectionally deny all traffic to/from the Spamhaus DROP list. In addition, they should block inbound port 25 traffic from everywhere on the planet that you don't need email from. In other words; the fact that someone in country X wants to email you is unimportant unless you actually wish to receive mail from them. Yes, this is a reversal of default-permit, for a simple reason: default-permit for SMTP stopped being reasonable around 2000. Use http://www.ipdeny.com/ [ipdeny.com] to pick up the ranges per-country and only permit what you need. (Obviously a major research university can't do this. But Joe's Furniture, which does not have customers in Peru or Pakistan or Greece, can.)

Then use blacklists, the best defense against spam we've ever developed. (Source: 30+ years of email experience) Spamhaus's Zen blacklist is a good one with a low FP rate and a tolerable FN rate. Augment these with local blacklists based on domains and network allocations. Augment those with as much blocking of generic hostnames and dynamic IP space as possible: real mail servers have real hostnames and are on static addresses.

Then enforce RFC requirements: sending host must have rDNS, that PTR must resolve, what it resolves to should be the sending host's IP. Sending host must HELO as FQDN or bracketed dotted-quad; if FQDN, must resolve. Sending host must not send traffic pre-greeting. And so on. Enforcing these DOES mean occasionally you block mail sent by non-spamming entities: but since they are incompetent non-spamming entities, why would you want mail from them?

Add greylisting. It'll handle a lot of annoying hosts that haven't learned to retry yet.

Rate-limit based on normative values for your site. For example: if analysis of a year's worth of mail logs shows that during that time you never received more than 10 messages a day from ANY host, then rate-limit at 30 or 40. You'll never hit in normal practice; but if you get hammered by a fast-sending host, you'll blunt the attack. Note that these don't have to be perfect to work: provided you send deferrals (SMTP response codes 4xx) instead of refusals (5xx) the worst that happens is that you will mistakenly impose a delay.

There's more -- it's possible to get quite crafty about this. But note that NONE of these measures pay any attention to content. There's a reason for that: spammers can defeat content-based measures at will. They won't have it so easy with these.

Deployed in production in various setups ranging from a dozen to eight million users, these steps yield a FP rate of about 10e-6 to 10e-7 and a FN rate around 10e-5 to 10e-6. Tuning helps, of course: initial rates can be higher but log analysis (which all sensible postmasters do) readily brings them down. If you have the luxury of running your own mail server just for yourself, then you can REALLY tune this setup: you should be able to get the FN rate down to 10e-7 after a few months.

Re:Procmail is a fine tool -- but the wrong tool (1)

thegarbz (1787294) | about 8 months ago | (#44722531)

That's a very informative post, but the first part is making a big assumption that someone has that level of control over the network. Much of what you say is exactly the type of filtering that is applied by spamassassin and other various tools at the end end of the chain. Many of us don't have the option to work higher up the chain.

Bayesian Mail Filter (1)

Trevin (570491) | about 8 months ago | (#44721831)

I've used bmf via procmail on my ISP shell account for years, and it was extremely reliable and accurate. As an added bonus, it automatically forwarded spam to uce@ftc.gov.

When my ISP discontinued the use of procmail filters, I moved it to my home computer and configured two filters in Evolution: the first one to auto-remove mail marked by my ISP as suspected spam, and the next to pipe the mail through bmf and remove it if it tested positive for spam. When I say "auto-remove", I mean it's moved to a spam folder where I can double-check it in case false positives get through.

http://sourceforge.net/projects/bmf/ [sourceforge.net]

Install CRM114 (0)

Anonymous Coward | about 8 months ago | (#44721851)

Install CRM114, set it up, and begin teaching it spam from non-spam.

Very quickly it will "learn" and you'll seldom ever see a spam message.

http://crm114.sourceforge.net/

Fail2Ban (0)

Anonymous Coward | about 8 months ago | (#44721859)

I've used Fail2Ban and some regular expressions to help filter out things. For example, when you email someone and you get the address wrong, you get an email kicked back with the 450 error code.

So, I use Fail2Ban to look for 450 error codes, and if it sees that 5x within 10 minutes, it blocks your IP address for 24 hours.

Couple that with blocking entire countries IP ranges (China, Russia, etc.), I see little to no spam at all.

regular expression optimiser (2)

lkcl (517947) | about 8 months ago | (#44721937)

i'd be interested to see what happens if you run those regex's through this:
        http://bisqwit.iki.fi/source/regexopt.html [bisqwit.iki.fi]

btw can we please get a copy of the patterns you're using? i think they might prove useful for other people. also i'd like to test them myself against regexopt.

oh - to the other person who suggested spamassassin? i tried that, i set it up to run at MTA-time. it often took THIRTY SECONDS to process a message. in fact it was so bad that i was forced to set a limit of 100k on incoming messages, as a lot of virus-ridden word documents (etc) were typically over 100k. that cut down the amount of CPU cycles but it was still far far too much memory and far too CPU intensive.

the one thing that did work well is greylisting, however the problem with greylisting i find is that if you happen not to be at the computer or have direct access to the server and people on the phone say "i'm sending you a message now, have you got it?" you *know* it's going to be at least an hour before it'll arrive. so, unless you can whitelist them in advance (which you can't always do) greylisting does actually interfere with legitimate business.

anyway: in the end i gave up and went to gmail, but with gmail fucking up how they're doing things i have to revisit this and set up a mail server again. thus we come full circle...

Re:regular expression optimiser (1)

thegarbz (1787294) | about 8 months ago | (#44722541)

oh - to the other person who suggested spamassassin? i tried that, i set it up to run at MTA-time. it often took THIRTY SECONDS to process a message. in fact it was so bad that i was forced to set a limit of 100k on incoming messages, as a lot of virus-ridden word documents (etc) were typically over 100k.

I'm sorry to say it but you must be doing something wrong. I have a very default installation of spamassassin and sendmail also running at MTA time and on my really crappy old spare parts server it never takes more than a second or two to process a mail item. This also does not appear to vary depending on email size, a 10MB email seems to take just as long as a plain text one. I don't think out of the box spam-assassin checks attachments or any type of external content.

Pay somebody (1)

DogDude (805747) | about 8 months ago | (#44721953)

Unless it's a fun hobby for you, it makes much more sense to just pay for email and let somebody else to it. Personal email can be gotten for about $2/month.

quit wasting your time (0)

Anonymous Coward | about 8 months ago | (#44722009)

there are million dollar companies that can detect it faster and even better than your OSS bullshit half assed script for free

quit pretending, its not 1994 anymore

Re:quit wasting your time (1)

gringer (252588) | about 8 months ago | (#44722063)

there are million dollar companies that can detect it faster and even better than your OSS bullshit half assed script for free

The NSA, for example. Use a US server as your email service provider, and you get filtering for free!

mutilthread? (1)

rtayek (606579) | about 8 months ago | (#44722293)

depending on where your time is going, consider splitting the file up into pieces and run each piece in a different thread.

Use perl (2)

Forever Wondering (2506940) | about 8 months ago | (#44722301)

A long time ago I benchmarked perl's regex engine against about 5 others. At the time, it was 10x faster than the nearest competitor for the same regex/data.

Also, you can use perl's "study". Or, split the regexes across threads.

Also, with perl you can do some hierarchical saviings. For example:
/Ffoo/ ...
/Fbar/ ...
/Fbaz/ ...

Could be redone as:
    if (/F/) {
... if (/Ffoo/)
... if (/Fbar/
... if (/Fbaz/)

    }

The above is trivial example, but you get the idea.

Also, how much time is spent compiling (vs. executing) the regexes in egrep? I imagine a lot and you have to do this for each incoming message.

Note that spamassassin (and hence perl) can be set up as a daemon where the regexes are compiled once. The messages are passed through a socket to the daemon. This means that the only CPU time spent is on executing the regexes--a considerable savings.

Additionally, perl regexes have [considerably] more functionality/utility than egrep ones. You might be able to recode/consolidate yours and get the same [or better] bang for less buck.

OpenBSD spamd (0)

Anonymous Coward | about 8 months ago | (#44722393)

Start with spamd and get your spam levels down immensely.

man page: http://www.openbsd.org/cgi-bin/man.cgi?query=spamd&sektion=8

bogofilter (0)

Anonymous Coward | about 8 months ago | (#44722447)

I have been using various procmail stuff but for years I am now relying on bogofilter.

I meanwhile have disabled autolearn as thats the stuff taking time.

I trained it with a couple megabytes of ham and spam and be done. From time to time when something gets classifies wrong ill push it for learning.

never had the whish to look for something else.

Buy a domain (1)

postglock (917809) | about 8 months ago | (#44722547)

I don't even use spam blockers. Instead I've purchased a domain, which is quite affordable nowadays. I have a catch-all redirect, so I any mail addressed to *@mydomain.com.

Then, I give a unique username to each organisation. e.g. slashdot@mydomain.com. If I receive spam at this address, I inform them, then kill the username. I can also just create slashdot2@mydomain.com if I want to keep dealing with their company.

Now, I receive only a few spam emails each year, so I need to do zero automated filtering. I also don't have to deal with the worry of false positives at all.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...