Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

What E-Mail Validation Tools Do You Use?

Cliff posted more than 7 years ago | from the return-to-sender dept.

87

morcego asks: "As we are all too much aware, spam is an increasing problem. Each of us has our own set of tools and methods to try and reduce the amount of spam we receive, each with different pros and cons. Also, on a more broad front, we have options like SPF (+ SRS), Microsoft's own Caller-ID, and Yahoo's DomainKeys that we can use. These days, it is incredibly easy to implement any (or all of these), using publicly available frameworks and libraries (libspf2, and milter, to name a few). I have been using SPF for quite some time now with some measurable results, although nothing earth shattering. Which of these are you using, if any? Why, or why not? Do you think any of them really contribute anything to fight spam?"

cancel ×

87 comments

Karaman (0, Redundant)

Karaman (873136) | more than 7 years ago | (#16670895)

Well, I use my brain!

Re:Karaman (1)

tsa (15680) | more than 7 years ago | (#16671279)

Me too. If I don't recognize the sender OR understand within one second what the subject header means, the mail goes in the bin. I don't get much mail from unknown persons at work, but if I do these persons better make sure their subject header is to the point and understandable otherwise it's their bad luck.

none (1)

gonk (20202) | more than 7 years ago | (#16670937)

I don't use anything other than dspam. It filters 99% of my spam for me. What more could I want?

Re:none (1)

Si (9816) | more than 7 years ago | (#16671405)

1% more?

Nice, but usefull? (1)

strredwolf (532) | more than 7 years ago | (#16670943)

The problem is that they can be spoofed, although not quite easily. That's because they're having folks self-setup the various systems.

Me, I would rather say "If your domain isn't in the same netblock as the ISP it represents, score heavily against."

Re:Nice, but usefull? (0)

Anonymous Coward | more than 7 years ago | (#16670971)

please explain in detail just how SPF can be "spoofed".

Re:Nice, but usefull? (0)

Anonymous Coward | more than 7 years ago | (#16674281)

please explain in detail just how SPF can be "spoofed".

I don't know about spoofed, but at the current cost of mass registered domains, its fairly easy for a organized spammer to register a domain, create valid SPF records, spam away for period X, then move on to the next domain before folks can get a domain based filter in place. It just hasn't panned out the way they hoped...

Re:Nice, but usefull? (1)

bogeskov (63797) | more than 7 years ago | (#16671053)

"If your domain isn't in the same netblock as the ISP it represents, score heavily against."

I (literally) don't get this "rule". Could you explain what you mean by "ISP it represents" in this sentense.

Re:Nice, but usefull? (1)

numbski (515011) | more than 7 years ago | (#16672151)

I'm just going out on a limb here. Let's say I send you an email. I come from hksilver.net. If you resolve hksilver.net, it would return (currently) 208.231.66.99. Now, right away you're going to come across a problem. I host my own domain, I have gotten my ISP to put a PTR record for my host. So reversing it will return mail.hksilver.net. If you were to go by netblock 208.231.66/24, and you're checking to make sure the email sources from that netblock, you'd be okay in my case about 75% of the time. The other 25% I'm sending from my laptop and relaying through my business partners' exchangers. Now, in an ideal world, I'd use SMTP Auth and always relay out of my own netblock, but at present I don't (simply haven't had time to set it up!), and I'm pretty savvy.

Now, go replace me with an Exchange admin. Uh huh. :\ That rule will "work", but false positives will be VERY high.

Re:Nice, but usefull? (1)

walt-sjc (145127) | more than 7 years ago | (#16672213)

I don't see spoofing as the problem. I see critical mass as the problem. Unless nearly ALL ISP's and email systems adopt a single "standard", the mechanism is useless. We don't have critical mass. I'm seeing less than 1% adoption rate for any of these systems.

Furthermore, these systems are not designed as anti-spam systems. Phishing and JoeJobs they may help with. Spam not at all. Since they don't help fight spam, there is no incentive to adopt them.

Re:Nice, but useful? (0)

Anonymous Coward | more than 7 years ago | (#16673301)

They indirectly help fight spam by fighting spoofing.

Re:Nice, but usefull? (1)

liquidpele (663430) | more than 7 years ago | (#16673935)

1) They eliminate sender spoofing in emails
2) Without sender spoofing, you can see what domain an email actually came from
3) Ban the bad domains in your emails rules.

wow

Re:Nice, but usefull? (1)

walt-sjc (145127) | more than 7 years ago | (#16677869)

1) Without critical mass, it doesn't.
2) No shit.
3) Ban all you want. Domains are cheap so spammers will create more...

End result - no change at all in spam volumes. If all the big ISP's got together and said that in January 1, 2008, they would no longer accept mail from anyone without an SPF record / SenderID, you MAY get 70% compliance. But I doubt it. In order to be truely effective, you need 90%+ compliance. Even at 100% compliance, you won't fix #3.

Re:Nice, but usefull? (1)

liquidpele (663430) | more than 7 years ago | (#16709887)

All I'm saying is that this is the only way it'll happen with current email tech. Any if you have spam filters that flag 100 spams coming from a domain, ban the domain automagically. I would think that would work *really* well, if like you said we ever got critical mass of SPF capable ISPs and began requiring it. The only reason they havn't, in my opinion, is that they make more money off the Spam themselves by selling anti-spam crap to their customers.

Hello (0)

Anonymous Coward | more than 7 years ago | (#16670955)

I am an anonymous coward.

Re:Hello (0, Offtopic)

aerthling (796790) | more than 7 years ago | (#16671013)

No, I am Anonymous Coward!

Re:Hello (0)

Anonymous Coward | more than 7 years ago | (#16671079)

No, I'm Spartacus!

Re:Hello (1)

tom17 (659054) | more than 7 years ago | (#16671225)

Hi, i'm Ed Winchester!

Re:Hello (1)

woozlewuzzle (532172) | more than 7 years ago | (#16671477)

I *am* Rock Quarry! I am!

Re:Hello (1)

brian1078 (230523) | more than 7 years ago | (#16679669)

No, My Name Is Inigo Montoya. You Killed My Father. Prepare To Die.

Mailvisa (2, Informative)

RAMMS+EIN (578166) | more than 7 years ago | (#16671011)

I wrote my own Bayesian filter, Mailvisa [inglorion.net] , to gain a better understanding of how Bayesian filtering works, and to be able to tweak the parameters. When I last measured it, it caught 93% of spam. Of all the filters I tried at the time (I think it was all filters in Debian sarge), only Bogofilter [sourceforge.net] scored better. This applies to both the amount of spam caught and the filtering speed. The closest thing to false positives I've gotten over the years were a few advertisement mails from my domain registrar.

I have only two problems with it: 1. I have to train it regularly, and 2. nowadays, lots of mail slips through, because it contains words related to programming languages.

Re:Mailvisa (1)

BrokenHalo (565198) | more than 7 years ago | (#16672311)

2. nowadays, lots of mail slips through, because it contains words related to programming languages.

I used to be very happy with the spam filtering that came with Thunderbird (after some preprocessing at the ISP's end). Now, however, most of the spam I'm seeing in my inbox is of text encapsulated in a single image which seems to fool the filter quite successfully. Not too sure how to get around this without having to sit down and spend some time working on it, which really means the spammers have won. :-(

Greylisting (1)

digitalchinky (650880) | more than 7 years ago | (#16671027)

Greylisting and DSPAM work for me. The odd spam still gets through, though the majority of those can be rejected with various postfix settings.

Re:Greylisting (1)

Athanasius (306480) | more than 7 years ago | (#16672471)

Seconded. Use of greylisting and the sbl-xbl from spamhaus easily drop the vast majority of attempted spam aimed at the mail server I admin. I back that up with spamassassin AND bogofilter because both of them still manage to catch enough spam that the other doesn't. For the month of October and only for my own email:
Both 334 (31%)
Bogo Only 256 (24%)
SA Only 140 (13%)
Neither 330 (31%)
Total 1060
And as you can see, due to the use of greylist+spamhaus RBL I actually end up receiving a high percentage of spam that neither spamassassin or bogofilter catch. Before I used greylisting and fixed the RBL usage that uncaught percentage was much lower, i.e. greylisting gets rid of a hell of a lot of spam. And, yes, I do train both SA and bogofilter on every spam that neither catch. The biggest culprits for getting through are those emails with random main body text plus an attached gif with the actual spam in it. Actually it's the spamhaus RBL catching most of it, 4843 items yesterday were permanently rejected via it. That's versus 1156 temporary greylisting rejections, some of which would have made it through subsequently. Note these stats are for the whole server, not just my own email. Yes, I'd have hated it if spamhaus.org had have been closed down :/.

Here cometh the plague of antispam-resistant spam (1)

orangesquid (79734) | more than 7 years ago | (#16672963)

The random main-body text spam is all over the place lately. It seems that as soon as spammers realize X won't pass the filters, they send much less X and more Y. The problem with the random text is that it's very hard to discern from legitimate e-mail (statistically speaking). Filters don't have a sense of context and conversation, even if they're so extravagant that they can perform cunnilingus on a hardwood floor. A simple validation system (SPF isn't a bad idea) would be a good step forward, if it was ubiquitous enough. Perhaps somebody can make it trendy to "get your SPF on" ?

SPF ? (1)

Athanasius (306480) | more than 7 years ago | (#16673771)

I've pondered over SPF myself, but I'm not really enamoured of it after reading all the pros and cons. I do publish a TXT record with SPF data for miggy.org, but only to say "these are the hosts/IPs that are DEFINITELY ok to receive email from claiming to be from miggy.org, but don't go dropping things on the floor just because I don't list another host here". i.e. people can use that record to whitelist (or upscore) the genuine miggy.org email, but won't use it to definitely blacklist miggy.org email from other hosts, although I guess they can downscore such if the like.

I don't use SPF at all at the MTA level, although I do allow Spamassassin's SPF rules to add to its scoring.

My main problem with SPF is the maillist one, and of course at least one solution to that, VERP, then interacts badly with greylisting. And of course that objection applies to the variations on SPF as well, to the best of my knowledge.

Actually the way I'm using SPF sums up my approach to spam counter-measures; try to use anything only as an advisory about the likelihood of the email being spam. My one exception to this is the use of Spamhaus' RBL as past experience has shown it to work near enough to 100% accurately to not be a problem (I've never had a user report a problem sending or receiving email with the culprit turning out to be an SBL-XBL false positive).

I use GMail (1)

Bob Cat - NYMPHS (313647) | more than 7 years ago | (#16671031)

Works pretty well.

I use GMail :) (3, Interesting)

brunes69 (86786) | more than 7 years ago | (#16671039)

After trying to tune SpamAssassin to work well for months, and being unimpressed by the hit/miss rate, I tok to forwarding all of my incoming email to GMail. I then forward all my email from GMail that is not spam back to my other account :0

I find this way I get 99.95% accuracy - things that GMail misses as spam, my local SpamAssassin catches. As a side bonus I have GMail's awesome interface to read my mail when on the road (much better than the Squirrel Mail I was using, and still better than RoundCube).

This brings up another point - I don't know why Google doesn't add IMAP connectivity to GMail, soyou could use it's interface to read email from other hosts. I don't see why their ad technology would not work with this scheme.

Re:I use GMail :) (1)

RAMMS+EIN (578166) | more than 7 years ago | (#16671277)

``This brings up another point - I don't know why Google doesn't add IMAP connectivity to GMail, soyou could use it's interface to read email from other hosts. I don't see why their ad technology would not work with this scheme.''

In fact, if they can forward your mail to another account (which they do) and they can offer POP3 (I think they do), they can offer IMAP, too.

Re:I use GMail :) (1)

wannabgeek (323414) | more than 7 years ago | (#16671699)

The problem is it becomes painful to view your mail in the other account. Unless you have an automatic filter somewhere to strip the gmail headers. Every mail would appear to have come from your gmail account. The sender of email is a useful thing for me to keep track of my mail.

Re:I use GMail :) (1)

Bios_Hakr (68586) | more than 7 years ago | (#16672161)

I have gmail auto-forward to my work account due to stupid webmail blocking policies. If I click "reply" in Outlook, the "To" address is not my gmail account; it is the person who sent the mail originally.

Oh, and the "From" field in my Outlook inbox shows the correct sender.

Re:I use GMail :) (1)

brunes69 (86786) | more than 7 years ago | (#16672621)

Not true, GMail preserves the original headers when it forwards.

Re:I use GMail :) (0)

Anonymous Coward | more than 7 years ago | (#16673227)

I was wondering the same thing. I was trying to migrate my email over to GMail and didn't want to have to check my old system for old email and gmail for new mail. I was planning on using IMAP to transfer the files up to GMail but I discovered they don't offer it.

The best reason I can figure is that they way they are tagging the emails for their thread views, etc. require the emails to be received or sent from their system? I'm guessing it's back end technical reasons as opposed to a policy/business reason.

Either way just get Google for your Domain released already damn it! It would solve so many of my problems.

Re:I use GMail :) (0)

Anonymous Coward | more than 7 years ago | (#16685847)

My guess is that IMAP's potential high rate of random access to small bits of information (metadata queries, etc) isn't an optimal use case on their clustered file system. POP3 access is much more linear.

Re:I use GMail :) (1)

PurifyYourMind (776223) | more than 7 years ago | (#16680033)

I'm no expert, but doesn't this mean you'd have to check in two places for false positives?

Spamassassin (1)

djmurdoch (306849) | more than 7 years ago | (#16671047)

I get about 99% success with Spamassassin. (I do train it on its errors, about every couple of weeks.) The most common leakage I was getting was bounces from domains when the spammer spoofed my domain name; I finally put an SPF record in place, and those seem to have stopped.

One thing I wish it would allow would be to train it on all rules, not just those that the Bayesian filters use. Some of the rules give me a lot of false positives, but they'd be fine for others: so why do we have to manually change the scores on them?

Greylisting (0)

Anonymous Coward | more than 7 years ago | (#16671055)

I use several methods, none of which look at the message apart from headers, but the biggest hit is from greylisting which knocks out more than 90% of spam at a stroke. Most of the rest is rejected by refusing email from servers with no reverse-lookup.

Greylisting and SBL+XBL (2, Interesting)

whitmer (142924) | more than 7 years ago | (#16671071)

While not necessary e-mail validation tools, greylisting and SBL+XBL blocking lists by Spamhaus have eradicated nearly all spam I used to get through all of the other filters.

Greylisting alone helped to lower e-mail traffic drastically and blocking lists take care of known spamming hosts. I'd recommend using both to anyone running a e-mail server.

Re:Greylisting and SBL+XBL (1)

cdwiegand (2267) | more than 7 years ago | (#16671939)

I was able to convince my boss to go back to having a linux mail gateway in front of our exchange server due to the good job (great job, really) that greylisting does! It was like night and day, and even I had a hard time believing it. Because we're a company, and the RBLs aren't always accurate, I can't use them, so some spam does get through, but very very little (on the order of 5 per day. 5!).

Re:Greylisting and SBL+XBL (1)

itwerx (165526) | more than 7 years ago | (#16675625)

...some spam does get through, but very very little (on the order of 5 per day. 5!).

Er, what's the context for that "5"? If your company only gets 100 emails a day that 5 is actually pretty lousy. (Now if they get 100K a day then it's great!)

Re:Greylisting and SBL+XBL (1)

walt-sjc (145127) | more than 7 years ago | (#16672425)

Don't expect Greylisting to reduce spam for long. Spamware is evolving and will start taking greylisting into account shortly, much like image spam gets around bayesian analysis. It's a matter of time before spammers start snagging email configuration info (such as SMTP Auth info) from pwned machines and sending spam via normal ISP gateways. Even rate limiting won't help as the number of pwned machines is massive, and growing every day.

BTW, even OCRing (which is very expensive computationally) of image spam is starting to fail as spammers start using noise and funky fonts (like a captcha) to foil anti-spam systems.

I really don't see long-term technological solutions to spam as long as it's so damned easy to pwn a machine.

Re:Greylisting and SBL+XBL (0)

Anonymous Coward | more than 7 years ago | (#16684911)

"Don't expect Greylisting to reduce spam for long. "

I don't see how it can really fail..

1. spam gets sent to me - it's not on a DNSBL - so I greylist it. Spam is also sent to a bunch of other people
2. the other people (some automatically due to spamtraps) add that IP to a DNSBL
2. greylist period is up - the server tries to send again - I 550 it due to it being on a DNSBL list that I use.

Greylisting buys you enough time for the blocking lists to catch up with spam runs. That's been my experience with greylisting, and I hated the thought years ok of having to use it.

What E-Mail Validation Tools Do You Use? (0)

Anonymous Coward | more than 7 years ago | (#16671113)

The telephone.

If you really, really want to know if a message is from someone it's the only way.

spf,CallerID,DKIM validates sender only not Spam ! (2, Interesting)

johnjones (14274) | more than 7 years ago | (#16671257)

all that SPF CallerID and DKIM does is validate the sender !

this cuts out about 70% of (stupid) spammers
you also need to blacklist people who send you spam (and you can be confident that you get them because of the above technologies)
if you Ever want to send lots of mail to hotmail users you need to have callerID setup yahoo and gmail both trust you more if you have domainKeys
so things are moving on and there is no reasson why people should not have at least one of SPF CallerID or DKIM setup on their domain !

you will note that people here also use filtering but the question is does the filtering feedback to the blacklists ?

regards

John Jones

p.s. I work in the mail vendor world...
 

SpamFilter (1)

marktoml (48712) | more than 7 years ago | (#16671275)

SpamBayes. After enough training it is spookily accurate at getting spam. I used to run SpamAssassin as a POP3 proxy and then filter the rest with SpamBayes, but recently (past year or so) SpamBayes has been enough.

This *might* be due to ISPs doing a better job of bulk filtering out the obvious junk before we even see it. Some of the domains I have that are on other than my main ISP do seem to end up with more spam, but after filtering via SpamBayes I see very little...

pf OS fingerprinting (3, Informative)

jnieuwen (524859) | more than 7 years ago | (#16671289)

I use the OS fingerprinting options from pf to block windows machines from delivering mail on the primary mx. This saves approximately between 300 and 1600 spams a day. Beside that, rejecting mail from hosts without an A record, blacklisting all hosts sending mail to spamtraps with spamikaze [spamikaze.org] , rejecting hosts which falsely claim to be a host in my domain and filtering with bogofilter.

Re:pf OS fingerprinting (1)

walt-sjc (145127) | more than 7 years ago | (#16672509)

blacklisting all hosts sending mail to spamtraps

So you blacklist all mail from yahoo, hotmail, gmail, msn, aol, verizon, earthlink, etc.? Because all of those servers send to spamtraps all the time.

Re:pf OS fingerprinting (1)

jnieuwen (524859) | more than 7 years ago | (#16672993)

Yes. But when a legitimate mail is send the mail is rejected by my server, the yahoo server (or hotmail etc.) bounces the mail to the sender, with normally the 554 reject message which contains a link to remove the host from the blacklist by the user. The idea is that spammers will not do this, but people who really want to send mail do. And if people really need me anyway, they can call me, or pay me a visit.

Re:pf OS fingerprinting (1)

walt-sjc (145127) | more than 7 years ago | (#16673473)

That makes very little sense. The big ISP's don't have one email server. They have hundreds. What will happen is that you will eventually blacklist all of them, and when a user gets a bounce, he can whitelist ONE of the servers, send his message again and get another bounce because he hit yet another bkacklisted server.

You are better off maintaining a per SENDER whitelist rather than per SERVER to be effective in this scenario (which is what we do for "evil" domains like yahoo and such that are heavily used by 419ers.) Evil domains get a large base spamassassin score "just because", with large negative offsets for whitelisted users (and other "secret" reasons.)

Re:pf OS fingerprinting (1)

jnieuwen (524859) | more than 7 years ago | (#16673675)

> What will happen is that you will eventually blacklist all of them

There is a timeout after which a host is removed from the blacklist.
But in fact I do not really see this as a problem, why should I want
email from a provider sending spam through their mailserver? The mailserver of an
ISP will only send spam created by their own users (on which they should act)
and will not act as an open relay for others.

Also note that I do not block on sender address, but on the IP of the delivering
mail server.

And the list is.... (1)

smkndrkn (3654) | more than 7 years ago | (#16671383)

This is the list of most of the stuff we run at the border:

Exim + greylisting +c lamav + Spamassassin.

Here are the plugins to spamassassin and custom rulesets:

Plugins:
---------
Razor2
SpamCop
AWL
MIMEHeader
ReplaceTags

Custom Rulesets
----------------
We use a selection of the SARE rulesets
70_sare_adult.cf
70_sare_bayes_poison_nxm.cf
99_FVGT_Tripwire.cf
bogus-virus-warnings.cf

This was stopping most of our spam...however we were still getting a lot of spam that contained images with the spammy message. So about 2 weeks ago I implemented the FuzzyOcr plugin on all of our border systems. I've received a single image spam since I launched it. It works great. I also tweak the rule scores based upon our situation and the type of spam that we get and I monitor the effectiveness of those changes on a fairly regular basis.

Re:And the list is.... (1)

martin (1336) | more than 7 years ago | (#16671613)

add the sare_stock and the FVGT rules, this'll stop the stock image with the huge overhead of fuzzyOCR.

also have a look at the other SARE and jennifer rules - I find these very useful.

Re:And the list is.... (1)

Dorkmunder (950796) | more than 7 years ago | (#16682609)

That is about the same list of what we use. I've got all the common Postifx restrictions first, followed by the Blacklists (Spamhaus XBL, etc) followed by Greylisting then onto amavis that manages SpamAssassin (with Razor2, FuzzyOCR) and ClamAV. Our two gateway servers drop about 70,000 emails a day (87% of all email coming through our gateway) and Spamassassin labels almost the rest of the junk mail(we have a high bar for discarding since we want to let the user have some control). We also have our gateways contact our Exchange servers for emails to upload for SpamAssassin to learn from (we have public folders that users can submit missed spam and wrongly labeled non-spam to). This has helped SA improve its filters tremendously.
We do use SPF a little (for faked bounc backs etc) but as folks have said, the critical mass isn't there yet to help.
And yes, spammers will find the next loophole but I'm sure we'll find the next fix quickly.

Re:And the list is.... (1)

smkndrkn (3654) | more than 7 years ago | (#16694705)

I've already seen a new technique to defeat the OCR software. Yesterday I got my first email with a spam message that contained a single image for each letter of the message. Of course FuzzyOCR didn't hit on this. Not sure how we'll get around that one.

Thunderbird Extension for Sender Verification (1)

taubz (322102) | more than 7 years ago | (#16671431)

I wrote a Thunderbird Extension for Sender Verification which implements SPF and DK on the client side, which may not be the best place to do it, but it's better than nothing at all. The extension is aimed at phishing, rather than spam. It also checks sender domains in several blacklists.

https://addons.mozilla.org/thunderbird/345/ [mozilla.org]
http://razor.occams.info/code/spf [occams.info]

Use ASSP (1)

jeremyclark13 (999183) | more than 7 years ago | (#16671709)

I personally use ASSP for my spam filtering. I use the SPF vailidation, RBL, Spam bucket address, multiple HELO checks, and of course Bayseian filtering. I've found that with all of this I've yet to see a spam mail in my inbox with 40+ days of uptime. Before I started using ASSP I would probably recieve two to three spams a day.

SPF is not antispam! (1)

FooAtWFU (699187) | more than 7 years ago | (#16671769)

SPF (and related technologies) are not designed to cut down on spam. They are designed to prevent Joe jobs [wikipedia.org] and address forgery. (It just so happens that most Joe Jobs are spam).

Re:SPF is not antispam! (1)

eric76 (679787) | more than 7 years ago | (#16675541)

SPF records can be useful to identify legitimate e-mail servers from selected domains.

Re:SPF is not antispam! (1)

Havokmon (89874) | more than 7 years ago | (#16679357)

SPF (and related technologies) are not designed to cut down on spam. They are designed to prevent Joe jobs and address forgery.

I just went through this with a security company for a Visa audit, so let me expand on this. They seemed to think that checking the Mail From: for a local user, when sender wasn't authenticated (I would assume - we never actually got that far), was a valid way of checking for forgeries. There are multiple problems with this.
  • Their testing was flat out wrong to begin with. They happen to use Exim, which when faced with an empty 'From:', replaces it with 'Mail From:'. So far this only seems to occur with Exim, Postfix, and Exchange. Definitely not qmail. Clients only view the 'From:' header - so they were confused with the results, not knowing what each header was for in the first place.
  • Any person who is trying to forge an address isn't actually going to leave the From: empty to be replaced with Mail From: - duh.
  • I'd really love to see any check that can verify a free-form field of a persons name and email. I think we're starting to get just a tad anal if someone can't use 'Rick L Romero', and they have to match their 'Real Name' exactly to use an external client.
  • Sure SPF can help with forgery, but SPF only checks MAIL FROM: So as you said - it's not an end-user forgery prevention method, it just prevents joe-jobs.

Supposedly this Mail From: 'forgery' testing information was found on a web site for doing PEN testing. Maybe they just stumbled on SPF verification and got confused. *shakes head*

So to follow up - SPF, and Mail From: checking are NOT for blocking spam, and NOT for preventing 'From' address forging.

DNS blacklists, SPF, Amavis, Spamassassin (1)

LinuxDon (925232) | more than 7 years ago | (#16671889)

The combination of 8 DNS blacklists, Amavis and Spamassassin works very well.
I used to get more than 300 spam mails per day (intercepted by Spamassassin), due to the use of DNS blacklists I now only receive about 15 spam mails per day wich are intercepted by Spamassassin.
Only about 3 spam e-mails per day actually make it into my mailbox, with zero false positives.

The good thing about DNS blacklists is that the spam e-mails are actually rejected in the mail protocol, therefore it will hit spammers directly and renders their spam bots useless.
The blacklists also reject dynamic ip addresses, which are all virus infected home computers.

The most effective blacklists I use:
spamcop.net
uceprotect.net (L1, L2, L3)
spamhaus.org (sbl-xbl)

Re:DNS blacklists, SPF, Amavis, Spamassassin (0)

Anonymous Coward | more than 7 years ago | (#16674961)

I bet you're losing a lot of legit email through spamcop, it has to be one of the worst and innacurate DNS blacklists around. It annoys me that people are still using them. You say you never get false positives, but can you really be bothered checking all the spam to see?

Re:DNS blacklists, SPF, Amavis, Spamassassin (1)

ACMENEWSLLC (940904) | more than 7 years ago | (#16676993)

I turned off Spamcop for a while. But it is a lot better now. With the proper exception entries, it works very well.

I use the Sorbs.net responses 5, 7, 9, and 11. I don't use the more common 6, nor 11. I did, but too many false positives.

I use spamcop, abuseat.org, antispam.or.id, dsbl.org, relays.ordb.org, and the spamhaus.org responses 2, 3, 4, and 6.

I also run my own lists. I reject around 12,000 and up e-mails a day with about 300 a day making it through to the anti-spam filters and maybe 100 getting past those.

What I don't get is why don't we combine a "Hot or Not" type program with a blacklist and make it open source?

You would need a user ID to create entries. You could rate an IP as spammy or not. After so many spammys ratings, it gets on one blacklist. After more, it gets added to another.

There would have to be math to remove entries after so long that have no complaints. If a userid is falsely stating certain IP's are not spam, that userid would get canceled and all his entries revoked and the lists updated, if needed.

Anyway, that's the simple summary of the idea.

Re:DNS blacklists, SPF, Amavis, Spamassassin (1)

b0s0z0ku (752509) | more than 7 years ago | (#16719109)

The blacklists also reject dynamic ip addresses, which are all virus infected home computers.

*All*? I run a mail, gaming, and web server off of a dynamic IP. Forwards out through a smarthost, so blacklisting isn't a problem, but it isn't infected with viruses nor am I using it for illegit purposes (ok, well it probably does violate my ISP's TOS, but fuck'em).

-b.

SPF and other Mail Filters... (1)

linuxg0d (913436) | more than 7 years ago | (#16671925)

I use the libspf2 however, using it is quite useless when you come to think of it. In reality the concept is amazing, however, if you think of it, it relies on 3rd parties envolvement. When you implement SPF, you check other users domain SPF records for validation purposes, however, what if the other users haven't specified their own records? Some reputable and large ISP's still do not have SPF setup. In reality, using SPF is great... as long as everyone else uses it. Having to rely on others when it comes to spam however, has proven to be futile on many occasions.

Re:SPF and other Mail Filters... (0)

Anonymous Coward | more than 7 years ago | (#16674837)

SPF is a bad idea, it's not that some large ISPs haven't yet deployed it, it's that some large ISPs already withdrawn support for it.

--
Google for more, I'm nobody's research assistant

MXLogic (1)

gothzilla (676407) | more than 7 years ago | (#16672309)

I'm the entire IT Dept at my work and I do not have the time to manage our own email server, let alone worry about keeping it secure. Most of our business comes in via email and most of those are crafted to look exactly like spam with huge lists of names in the TO: or CC: boxes and no subject line.
My problem was finding a way to filter spam without filtering even a single legit email. Lost email means a lot of lost revenue. The only solution I found in a year of searching was mxlogic.com. We still get spam, but not nearly as much and since you get a filter report daily of what email was filtered, our people can see if anything was caught that shouldn't have been. The result, much less spam and ZERO lost revenue.

Yeah I wish I had the time and expertise to run my own email server and keep it secure but the fact of the matter is that there are lots of shops that just can't. This is seriously the only solution I have found.

SPF. Postgrey and Spamassassin (1)

Greyfox (87712) | more than 7 years ago | (#16672627)

And that's been keeping the ones that get through down to two or three a week. Not enough for me to turn on hard SPF checking or demanding that email to me be encrypted with my personal PGP key. Configuring all that stuff certainly is a pain though -- it'd be nice if they could get it down to drop in components for the most common configurations.

RBL and SURBL on the server side (1)

Degrees (220395) | more than 7 years ago | (#16673497)

The postfix server uses RBLs to drop about 25,000 messages per day. If postfix accepts it, it gets handed off to a different server that does SURBL checks. (That is done by a commercial product called GWAVA [gwava.com] ). The SURBLs catch about another 2,000 messages per day.

I have published my SPF data - so at least other people have the option of identifying whether stuff that claims to have come from my domain is legitimate or not. But our mailers are not yet doing SPF lookups. When we have a little time, we will probably add it to the postfix server. If the site specifies in their SPF record to 'hard' drop email that comes from anywhere but x (and the connecting server is at y), we'll treat it like an RBL.

The down side to having GWAVA one hop in from the postfix server is that some spammers get paid if the receiving mail server accepts the whole message - a (250 OK) by postfix means the spammer gets paid even if GWAVA later throws away the message. The GWAVA for Linux product is in beta test; once it goes official, I might be able to move it onto the postfix server, and hang up on the bad messages earlier.

Re:RBL and SURBL on the server side (1)

b0s0z0ku (752509) | more than 7 years ago | (#16719155)

called GWAVA

You're running Groupwise? GWAVA is overrated and is mainly useful for integrating spam filtering into Groupwise's Internet Agent. Nothing that SpamAssassing + ClamAV + ProxSMTPd won't do for you. And that combination is available as part of a package for an IPCop firewall box called CopFilter. The only downside is that CopFilter isn't as configurable as it should be via the Web interface. But for a free product, it's pretty darn good.

-b.

Re:RBL and SURBL on the server side (1)

Degrees (220395) | more than 7 years ago | (#16735249)

That's cool. If we got into a monetary crunch, we would probably implement what you mention here. One of the nice things about GWAVA is that it we have it configured to send an HTML message to users daily, where they can pull a message out of the bit-bucket (so-to-speak). That is to say, they get a digest message of what was blocked, and if something was improperly blocked, they can have it sent to them anyway. Version 4 (due out any minute now) will take this to the next level, where users can manager their own white lists on the GWAVA server.

But it is nice to know that there are alternatives available that are also free. :-)

Re:RBL and SURBL on the server side (1)

b0s0z0ku (752509) | more than 7 years ago | (#16735457)

One of the nice things about GWAVA is that it we have it configured to send an HTML message to users daily, where they can pull a message out of the bit-bucket (so-to-speak). That is to say, they get a digest message of what was blocked, and if something was improperly blocked, they can have it sent to them anyway.

Copfilter has a digest option too. We're not using it ATM, since I have it set up to block only the most egregious examples of SPAM i.e. those with scores of 25 or above. The rest simply gets tagged and put in users' junk mail folders by the Groupwise clients. Users can do what they will with junk mail and are instructed to check their junk mail folders regularly. We haven't needed the feature so far, but Copfilter accepts whitelist mods via e-mail.

My only wishes for Copfilter are: that the SPAM filtering be somewhat more configurable from the Web interface and that it would be able to block e-mail not destined for local domains before it ever gets to the server. We had some asshat try to relay a lot of spam through us, and it basically clogged up the works (actually, it was more of a DDoS attack disguised as a spam relay attempt, but having non-local domains blocked at the firewall would have been nice).

-b.

Wetware (1)

aero6dof (415422) | more than 7 years ago | (#16673891)

I use a (usually) sophisticated biological neural network consisting of a multi-billion plus nodes with some primitive pre-determined wiring structures serving as a foundataion. Oh yes, and as preliminary step, I use dual-stage filters: spamassassin followed by crm114. Spamassassin seems to be fairly well behaved by not giving too many false positve spam indications, and CRM114 picks through the remainder false negatives to my satisfaction. I still end up picking through the spam folders, but its bulk and not too difficult to plow through many at a time once they initially sorted.

Defense in Depth (0)

Anonymous Coward | more than 7 years ago | (#16674073)

I've found there is rarely one single solution/software that provides *effective* anti-SPAM when you run your own mailserver.

Small shops or IT depts. without the technical know-how may be better served by farming their anti-SPAM out to a service. While running everything thru GMail sounds great, a quick read of the GMail TOS (or the TOS for any other "free" E-Mail service) should convince those who need reliability *and* accountability in their E-Mail service to either do it themselves or *pay* someone else to do it.

My personal anti-SPAM defenses use MILTERs, most-notably MIMEDefang. I invoke ClamAV and SpamAssassin thru MIMEDefang. I use sendmail, but its facilities for spotting the obviously fraudulent garbage are, like most stock MTAs, woefully inadequate. Beyond RBLs, the three really effective anti-SPAM features in sendmail are only available in v8.13.x, and are:

GreetPause - configurably delays presentation of the HELO banner, and rejects hosts that send commands prior to the HELO banner display

RateConn - configurably limits the number of new connnections a given IP/network/host/Domain may request at any one time

ClientConn - configurably limits the number of connections a given IP/network/host/Domain may have at any one time

Use of these features significantly (but not dramatically) reduces SPAM - best of all, the reduction is *very* early in the SMTP conversation, minimizing resource wastage.

MIMEDefang gives a lot of power to the mail system admin - of course, the power needs knowledge to be really useful. But some of the checks I do via MIMEDefang (exempting my own mail hosts, of course), by SMTP conversation step, are:

HELO - if it's an IP HELO, reject the connection if it lacks square or the IP does not match the actual IP of the connecting host; it it's a string, reject the connection if it is not an FQDN, or if it claims to be a host in any of my Domains, or it contains "localhost"

MAIL FROM - reject if the sender claims to be from any of my Domains

RCPT TO - reject if the recipient address is not valid or not in any of my Domains

The nice thing about MIMEDefang is that I can do those checks long before DATA - the spammer's ability to waste my resources is minimized.

After DATA, I run the E-Mail thru ClamAV, look for suspicious characters in the headers, malformed MIME, etc. If the E-Mail passes all that, I run it thru SpamAssassin and tag it if needed. I allow my users to set individual SPAM score limits, so the final act is to delete any recipients for whom the SPAM score exceeds their personal limit (and discard the E-Mail entirely if no one is left to receive it), re-build it so it conforms to standards, and then deliver it.

For those users who take advantage of the SPAM score limits, very little SPAM leaks thru. Even for those who don't, the SPAM is usually tagged as such.

My personal anti-SPAM philosophy is "Reject early, reject often". The sooner you spot the obviously fraudulent connection and drop it, the less of your resources the spammer gets to waste. A lot of SPAM can be stopped long before the DATA step - save "expensive" (in terms of bandwidth, CPU and disk) tools like ClamAV and SpamAssassin for the more-clever SPAM.

My ISP contracts with Postini for Spam filtering. (1)

Richard Steiner (1585) | more than 7 years ago | (#16674161)

They've done so for the past few years, and it seems to work *very* well.

See their web site here [postini.com] ...

Re:My ISP contracts with Postini for Spam filterin (1)

John Hasler (414242) | more than 7 years ago | (#16681671)

> They've done so for the past few years, and it seems to work *very* well.

My previous ISP imposed Postini on me with no notice (they sent me an email bragging about it three days after they started using it). It passed 50% of the spam and stopped 20% of the ham. I turned it off.

Re:My ISP contracts with Postini for Spam filterin (1)

Richard Steiner (1585) | more than 7 years ago | (#16691191)

Interesting. My ISP introduced it as an opt-in service (just like they introduced SpamAssassin and various other tools to the user base), and while it did require some fine tuning, I've had very few problems with it (I get a handful of Spams a day which it doesn't catch, and I see one or two false positives a month).

I don't blame you for dropping it given how it was introduced at that ISP, but I think you also lost a chance to use a fairly effective anti-spam tool.

My mail server says, get the hell of its lawn! (1)

Akardam (186995) | more than 7 years ago | (#16675275)

In other words, it's crochety as hell. I have all the "speak the RFC's exactly or thy shall not pass" options turned on. I publish a SPF record, for what good it will do. I also 5xx reject anything from overseas.

Even though this is my own personal mail server, I haven't had too many false positives as far as rejects go... certianly nothing that a tweak here or there in the allow/deny hosts file wouldn't take care of.

All in all, I've recieved less than a dozen pieces of spam in the last year and a half. Not too shabby, I think.

Greylisting (2, Informative)

eric76 (679787) | more than 7 years ago | (#16675459)

I use spamd on OpenBSD to do greylisting. That cuts an enormous amount of spam out.

For those who aren't familiar with greylisting, when an smtp server attempts to deliver an e-mail the from address, to address, and IP address of the sender are put in a database and the mail is refused with a non-permanent error code.

Assuming the smtp server sending the e-mail follows the RFC, it will try again later. When it tries again after at least 20 minutes from the original attempt, it accepts the e-mail and adds the IP address of the source to a whitelist. For the next 30 days, any e-mails from it are white-listed. After that, the server is verified again.

I also keep a seperate white-list for non-RFC compliant servers and for frequent senders. Some servers only try one to three times and quit. Another problem is e-mail from some large e-mail farms may make each attempt to deliver the e-mail from a different server with different IP addresses, so I'll add their e-mail addresses to the white-lists as well.

One method I use for adding IP addresses of selected senders that send a lot of legitimate e-mail to the whitelist is to look up their SPF records and use that to identify the usual e-mail servers for the domain.

A few ISPs appear to put their entire address space in the SPF record. For example, panix.com's SPF record is

panix.com text = "v=spf1 ip4:166.84.0.0/16 ip4:198.7.7.0/24 ?all

Needless to say they don't get whitelisted since I only want to whitelist e-mail servers, not their users spam-zombie computers.

In other words, I use the SPF records to identify legitimate e-mail servers from selected domains only.

I use CanIt (1)

dheltzel (558802) | more than 7 years ago | (#16678679)

I used to "roll my own" with SpamAssassin and MimeDefang. Then I started using CanIt [roaringpenguin.com] at work (I liked them initially because the author is the author of MimeDefang). They have a free version that works well for me at home now. We have been using it for about 4 years at works and it does a great job incorporating grey listing, SA, MimeDefang, ClamAV, etc. into an easy to install and maintain system with a nice web interface and a database backend. It can scale well when we need it to and the support is great (a MAJOR factor for my company).

Did I mention it is cheaper than the other commercial offering as well. OSS, great support, low cost!

Dennis
(I know this sounds like a commercial, but I am not affiliated with Roaring Penguin in any way other than being a very satisfied customer)

Re:I use CanIt (1)

anon mouse-cow-aard (443646) | more than 7 years ago | (#16716471)


We were using MimeDefang + SA for a while, but it wasnt enough. Second the vote for Canit... just (as in Wednesday) rolled out Canit/PRO to serve mailboxes for 5000 full-time employees. Works well, cost is very reasonable. It has the benefit of the centralized solution for reduced maintenance, but we can use the web interface to customize mail flows for people with particular needs.

Spampal (1)

Paracelcus (151056) | more than 7 years ago | (#16680707)

www.spampal.org

I use a client side filter (0)

Anonymous Coward | more than 7 years ago | (#16680755)

The last three years I used a free client side filter on Unix, just a shell-awk script, a baysian-like filter. Also a white list of the addresses I have accepted, not a challange-response type of white list, but a passive one of the addresses I accept.

I train it every couple weeks to generate the white list and the spam probability tokens.

I pay poor children... (1)

GWBasic (900357) | more than 7 years ago | (#16680831)

I pay poor children in [???] $0.01 / hour to filter my mail for me. It's cheaper then buying SPAM filtering software.

TMDA Catches All My Spam (1)

DickHodgman (265853) | more than 7 years ago | (#16683849)

TMDA [tmda.net] catches all my spam. I does not examine content. It sends a request for response to all unknown senders. Since the vast majority of spam has forged return addresses, no responses are sent back and the mail stays in the TMDA pending queue until it expires. Humans, on the other hand, reply, and their mail is removed from the pending queue and gets through. When I set up TMDA, I populated the whitelist with all the email addresses of my correspondents and lists.

Around 75% (150/200 daily) of my email is spam. After a month my false positive rate was around 0.5% (1/200) and most of those were mass mailed offers I would not miss. My false negative rate is around 0.02% (1/6000); every month or so a spam message is validated; I just move the address from validated to blocked so I'll never see it again.

I never have to see the 150 spam messages that come each day. I check the pending queue only when a business that sends email with a non-responding return address is sending me a message, like an online order confirmation.

The user can generate a keyword address when signing up for a list; messages from this address are allowed through without whitelisting. If later compromised; that address can be put on the revoked list.

I use tmda.cgi for configuration.

Re:TMDA Catches All My Spam (1)

kitterma (757172) | more than 7 years ago | (#16691665)

"Since the vast majority of spam has forged return addresses, no responses are sent back" That's right and all the innocent owners of the forged addresses get stuck dealing with your spew. It works for you, but it's a pretty selfish way to go about it. Personally, I have not and will not ever respond to challenge/response schemes until I see one that manages to not spam innocent bystanders. If important enough, I call them. Otherwise, oh well.

ASSP (1)

Sedennial (182739) | more than 7 years ago | (#16684825)

We use ASSP at work (a government entity) and it is effective enough that when we DO have a spam slip through, users usally call to complain about it. It happens rarely enough that they forget to forward it to spam@<ourdomain.org>.

I also use it at home and have nearly the same effectiveness.

As far as various technologies, I don't believe any solution which relies sole upon one or two technologies will be that effective. ASSP seems to be the best so far at combining SPF/Greylisting/bayesian/various others. I implemented several versions of anti-spam systems for filtering an average 300k+ messages per day at an ISP and NOC peaking around 500-650k during holidays, so I do have SOME prior experience with this issue. :) We looked at SpamAssassin, DSPAM, plain bayesian filtering (libmilter), ip blacklisting, RBL, forced validation schemes, .... ad nauseum. Unfortunately I hadn't gotten around to testing ASSP yet.

SpamAssassin's success. (1)

elvey (86546) | more than 7 years ago | (#16700537)

There are lots of short-term solutions, to which spammers adapt as they get widely adopted.

For example, content filtering in general is largely a short-term solution. Spammers invent and use obfuscation tricks; tools detect them, spammers invent new ones. Rinse, repeat.

Longer term solutions have to address root causes. These increase the consequences of spamming. IP blacklists, URI blacklists, domain blacklists, for example, result in negative consequences for bad actors and their associates. (Including folks who claim that they're not associates, where that association consists of sending money to the same folks for network connectivity, i.e. being customers of an ISP or webhost or ESP that harbors spamming customers.)

The way things are going, I see a continuing trend toward reputation services, where the reputation is that of an identity confirmed using one (or more) of the Email Authentication technologies - CSV (my favorite), SPF or DKIM/DomainKeys. (I've been building one, so I'm biased.) Only senders with positive (not just neutral) reputations will get through. Greylisting will, as another poster mentioned, be key in preventing spammers from getting one step ahead.

Complementary long term solutions include HashCash (e.g RubberStamp) -type solutions and better security.

SpamAssassin is a victim of its own success - it's so widely used that the first thing spammers do is send their mail through a server running it, and tweaking the message until it gets through that portion of its filters that are content-based. Of course SpamAssassin's Bayesian filter component helps in that regard, as do RulesDuJour, and other features that are not on by default. It works very well when tuned.

The unfortunate fact is that most ISPs and end users refuse to step up and shoulder the costs to keep their systems secure enough not to be sources of spam. They take on spamming customers and allow infected computers to remain on their networks. Until antispam measures impose costs that force these costs to be born (i.e. internalize the externalities), there will be more false positives and negatives.
Check for New Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...