Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×

What E-Mail Validation Tools Do You Use? 87

Posted by Cliff from the return-to-sender dept.
morcego asks: "As we are all too much aware, spam is an increasing problem. Each of us has our own set of tools and methods to try and reduce the amount of spam we receive, each with different pros and cons. Also, on a more broad front, we have options like SPF (+ SRS), Microsoft's own Caller-ID, and Yahoo's DomainKeys that we can use. These days, it is incredibly easy to implement any (or all of these), using publicly available frameworks and libraries (libspf2, and milter, to name a few). I have been using SPF for quite some time now with some measurable results, although nothing earth shattering. Which of these are you using, if any? Why, or why not? Do you think any of them really contribute anything to fight spam?"
This discussion has been archived. No new comments can be posted.

What E-Mail Validation Tools Do You Use? More

What E-Mail Validation Tools Do You Use?

Comments Filter:

  • none (Score:1)

    by gonk ( 20202 )
    I don't use anything other than dspam. It filters 99% of my spam for me. What more could I want?
  • The problem is that they can be spoofed, although not quite easily. That's because they're having folks self-setup the various systems.

    Me, I would rather say "If your domain isn't in the same netblock as the ISP it represents, score heavily against."

    • Re: (Score:1)

      by bogeskov ( 63797 )
      "If your domain isn't in the same netblock as the ISP it represents, score heavily against."

      I (literally) don't get this "rule". Could you explain what you mean by "ISP it represents" in this sentense.

      • Re: (Score:2)

        by numbski ( 515011 ) *
        I'm just going out on a limb here. Let's say I send you an email. I come from hksilver.net. If you resolve hksilver.net, it would return (currently) 208.231.66.99. Now, right away you're going to come across a problem. I host my own domain, I have gotten my ISP to put a PTR record for my host. So reversing it will return mail.hksilver.net. If you were to go by netblock 208.231.66/24, and you're checking to make sure the email sources from that netblock, you'd be okay in my case about 75% of the time.
    • I don't see spoofing as the problem. I see critical mass as the problem. Unless nearly ALL ISP's and email systems adopt a single "standard", the mechanism is useless. We don't have critical mass. I'm seeing less than 1% adoption rate for any of these systems.

      Furthermore, these systems are not designed as anti-spam systems. Phishing and JoeJobs they may help with. Spam not at all. Since they don't help fight spam, there is no incentive to adopt them.
      • Comment removed based on user account deletion
        • 1) Without critical mass, it doesn't.
          2) No shit.
          3) Ban all you want. Domains are cheap so spammers will create more...

          End result - no change at all in spam volumes. If all the big ISP's got together and said that in January 1, 2008, they would no longer accept mail from anyone without an SPF record / SenderID, you MAY get 70% compliance. But I doubt it. In order to be truely effective, you need 90%+ compliance. Even at 100% compliance, you won't fix #3.

  • Mailvisa (Score:3, Informative)

    by RAMMS+EIN ( 578166 ) on Wednesday November 01, 2006 @07:43AM (#16671011) Homepage Journal
    I wrote my own Bayesian filter, Mailvisa [inglorion.net], to gain a better understanding of how Bayesian filtering works, and to be able to tweak the parameters. When I last measured it, it caught 93% of spam. Of all the filters I tried at the time (I think it was all filters in Debian sarge), only Bogofilter [sourceforge.net] scored better. This applies to both the amount of spam caught and the filtering speed. The closest thing to false positives I've gotten over the years were a few advertisement mails from my domain registrar.

    I have only two problems with it: 1. I have to train it regularly, and 2. nowadays, lots of mail slips through, because it contains words related to programming languages.
    • 2. nowadays, lots of mail slips through, because it contains words related to programming languages.

      I used to be very happy with the spam filtering that came with Thunderbird (after some preprocessing at the ISP's end). Now, however, most of the spam I'm seeing in my inbox is of text encapsulated in a single image which seems to fool the filter quite successfully. Not too sure how to get around this without having to sit down and spend some time working on it, which really means the spammers have won. :-(
  • Comment removed based on user account deletion
    • Seconded. Use of greylisting and the sbl-xbl from spamhaus easily drop the vast majority of attempted spam aimed at the mail server I admin. I back that up with spamassassin AND bogofilter because both of them still manage to catch enough spam that the other doesn't. For the month of October and only for my own email:

      Both 334 (31%)
      Bogo Only 256 (24%)
      SA Only 140 (13%)
      Neither 330 (31%)
      Total 1060

      And as you can see, due to the use of greylist+spamhaus RBL I actually end up receiving a

      • The random main-body text spam is all over the place lately. It seems that as soon as spammers realize X won't pass the filters, they send much less X and more Y. The problem with the random text is that it's very hard to discern from legitimate e-mail (statistically speaking). Filters don't have a sense of context and conversation, even if they're so extravagant that they can perform cunnilingus on a hardwood floor. A simple validation system (SPF isn't a bad idea) would be a good step forward, if it w
        • I've pondered over SPF myself, but I'm not really enamoured of it after reading all the pros and cons. I do publish a TXT record with SPF data for miggy.org, but only to say "these are the hosts/IPs that are DEFINITELY ok to receive email from claiming to be from miggy.org, but don't go dropping things on the floor just because I don't list another host here". i.e. people can use that record to whitelist (or upscore) the genuine miggy.org email, but won't use it to definitely blacklist miggy.org email fro
  • Works pretty well.

  • I use GMail :) (Score:4, Interesting)

    by brunes69 ( 86786 ) <`gro.daetsriek' `ta' `todhsals'> on Wednesday November 01, 2006 @07:48AM (#16671039)
    After trying to tune SpamAssassin to work well for months, and being unimpressed by the hit/miss rate, I tok to forwarding all of my incoming email to GMail. I then forward all my email from GMail that is not spam back to my other account :0

    I find this way I get 99.95% accuracy - things that GMail misses as spam, my local SpamAssassin catches. As a side bonus I have GMail's awesome interface to read my mail when on the road (much better than the Squirrel Mail I was using, and still better than RoundCube).

    This brings up another point - I don't know why Google doesn't add IMAP connectivity to GMail, soyou could use it's interface to read email from other hosts. I don't see why their ad technology would not work with this scheme.

    • ``This brings up another point - I don't know why Google doesn't add IMAP connectivity to GMail, soyou could use it's interface to read email from other hosts. I don't see why their ad technology would not work with this scheme.''

      In fact, if they can forward your mail to another account (which they do) and they can offer POP3 (I think they do), they can offer IMAP, too.
    • The problem is it becomes painful to view your mail in the other account. Unless you have an automatic filter somewhere to strip the gmail headers. Every mail would appear to have come from your gmail account. The sender of email is a useful thing for me to keep track of my mail.
      • I have gmail auto-forward to my work account due to stupid webmail blocking policies. If I click "reply" in Outlook, the "To" address is not my gmail account; it is the person who sent the mail originally.

        Oh, and the "From" field in my Outlook inbox shows the correct sender.

      • Re: (Score:2)

        by brunes69 ( 86786 )
        Not true, GMail preserves the original headers when it forwards.
    • I'm no expert, but doesn't this mean you'd have to check in two places for false positives?
  • I get about 99% success with Spamassassin. (I do train it on its errors, about every couple of weeks.) The most common leakage I was getting was bounces from domains when the spammer spoofed my domain name; I finally put an SPF record in place, and those seem to have stopped.

    One thing I wish it would allow would be to train it on all rules, not just those that the Bayesian filters use. Some of the rules give me a lot of false positives, but they'd be fine for others: so why do we have to manually change
  • While not necessary e-mail validation tools, greylisting and SBL+XBL blocking lists by Spamhaus have eradicated nearly all spam I used to get through all of the other filters.

    Greylisting alone helped to lower e-mail traffic drastically and blocking lists take care of known spamming hosts. I'd recommend using both to anyone running a e-mail server.

    • Re: (Score:1)

      by cdwiegand ( 2267 )
      I was able to convince my boss to go back to having a linux mail gateway in front of our exchange server due to the good job (great job, really) that greylisting does! It was like night and day, and even I had a hard time believing it. Because we're a company, and the RBLs aren't always accurate, I can't use them, so some spam does get through, but very very little (on the order of 5 per day. 5!).

      • Re: (Score:2)

        by itwerx ( 165526 )
        ...some spam does get through, but very very little (on the order of 5 per day. 5!).

        Er, what's the context for that "5"? If your company only gets 100 emails a day that 5 is actually pretty lousy. (Now if they get 100K a day then it's great!)
    • Don't expect Greylisting to reduce spam for long. Spamware is evolving and will start taking greylisting into account shortly, much like image spam gets around bayesian analysis. It's a matter of time before spammers start snagging email configuration info (such as SMTP Auth info) from pwned machines and sending spam via normal ISP gateways. Even rate limiting won't help as the number of pwned machines is massive, and growing every day.

      BTW, even OCRing (which is very expensive computationally) of image spam

  • spf,CallerID,DKIM validates sender only not Spam ! (Score:3, Interesting)

    by johnjones ( 14274 ) on Wednesday November 01, 2006 @08:27AM (#16671257) Homepage Journal
    all that SPF CallerID and DKIM does is validate the sender !

    this cuts out about 70% of (stupid) spammers
    you also need to blacklist people who send you spam (and you can be confident that you get them because of the above technologies)
    if you Ever want to send lots of mail to hotmail users you need to have callerID setup yahoo and gmail both trust you more if you have domainKeys
    so things are moving on and there is no reasson why people should not have at least one of SPF CallerID or DKIM setup on their domain !

    you will note that people here also use filtering but the question is does the filtering feedback to the blacklists ?

    regards

    John Jones

    p.s. I work in the mail vendor world...
     
  • SpamBayes. After enough training it is spookily accurate at getting spam. I used to run SpamAssassin as a POP3 proxy and then filter the rest with SpamBayes, but recently (past year or so) SpamBayes has been enough.

    This *might* be due to ISPs doing a better job of bulk filtering out the obvious junk before we even see it. Some of the domains I have that are on other than my main ISP do seem to end up with more spam, but after filtering via SpamBayes I see very little...

  • pf OS fingerprinting (Score:3, Informative)

    by jnieuwen ( 524859 ) on Wednesday November 01, 2006 @08:31AM (#16671289) Homepage
    I use the OS fingerprinting options from pf to block windows machines from delivering mail on the primary mx. This saves approximately between 300 and 1600 spams a day. Beside that, rejecting mail from hosts without an A record, blacklisting all hosts sending mail to spamtraps with spamikaze [spamikaze.org], rejecting hosts which falsely claim to be a host in my domain and filtering with bogofilter.
    • blacklisting all hosts sending mail to spamtraps

      So you blacklist all mail from yahoo, hotmail, gmail, msn, aol, verizon, earthlink, etc.? Because all of those servers send to spamtraps all the time.
      • Yes. But when a legitimate mail is send the mail is rejected by my server, the yahoo server (or hotmail etc.) bounces the mail to the sender, with normally the 554 reject message which contains a link to remove the host from the blacklist by the user. The idea is that spammers will not do this, but people who really want to send mail do. And if people really need me anyway, they can call me, or pay me a visit.
        • That makes very little sense. The big ISP's don't have one email server. They have hundreds. What will happen is that you will eventually blacklist all of them, and when a user gets a bounce, he can whitelist ONE of the servers, send his message again and get another bounce because he hit yet another bkacklisted server.

          You are better off maintaining a per SENDER whitelist rather than per SERVER to be effective in this scenario (which is what we do for "evil" domains like yahoo and such that are heavily used
          • > What will happen is that you will eventually blacklist all of them

            There is a timeout after which a host is removed from the blacklist.
            But in fact I do not really see this as a problem, why should I want
            email from a provider sending spam through their mailserver? The mailserver of an
            ISP will only send spam created by their own users (on which they should act)
            and will not act as an open relay for others.

            Also note that I do not block on sender address, but on the IP of the delivering
            mail server.
  • This is the list of most of the stuff we run at the border:

    Exim + greylisting +c lamav + Spamassassin.

    Here are the plugins to spamassassin and custom rulesets:

    Plugins:
    ---------
    Razor2
    SpamCop
    AWL
    MIMEHeader
    ReplaceTags

    Custom Rulesets
    ----------------
    We use a selection of the SARE rulesets
    70_sare_adult.cf
    70_sare_bayes_poison_nxm.cf
    99_FVGT_Tripwire.cf
    bogus-virus-warnings.cf

    This was stopping most of our spam...however we were still getting a lot of spam that contained images with the spammy message. So about 2 weeks

    • Re: (Score:2)

      by martin ( 1336 )
      add the sare_stock and the FVGT rules, this'll stop the stock image with the huge overhead of fuzzyOCR.

      also have a look at the other SARE and jennifer rules - I find these very useful.
    • That is about the same list of what we use. I've got all the common Postifx restrictions first, followed by the Blacklists (Spamhaus XBL, etc) followed by Greylisting then onto amavis that manages SpamAssassin (with Razor2, FuzzyOCR) and ClamAV. Our two gateway servers drop about 70,000 emails a day (87% of all email coming through our gateway) and Spamassassin labels almost the rest of the junk mail(we have a high bar for discarding since we want to let the user have some control). We also have our gateway

      • Re: (Score:2)

        by smkndrkn ( 3654 )
        I've already seen a new technique to defeat the OCR software. Yesterday I got my first email with a spam message that contained a single image for each letter of the message. Of course FuzzyOCR didn't hit on this. Not sure how we'll get around that one.
  • I wrote a Thunderbird Extension for Sender Verification which implements SPF and DK on the client side, which may not be the best place to do it, but it's better than nothing at all. The extension is aimed at phishing, rather than spam. It also checks sender domains in several blacklists.

    https://addons.mozilla.org/thunderbird/345/ [mozilla.org]
    http://razor.occams.info/code/spf [occams.info]
  • I personally use ASSP for my spam filtering. I use the SPF vailidation, RBL, Spam bucket address, multiple HELO checks, and of course Bayseian filtering. I've found that with all of this I've yet to see a spam mail in my inbox with 40+ days of uptime. Before I started using ASSP I would probably recieve two to three spams a day.
  • SPF (and related technologies) are not designed to cut down on spam. They are designed to prevent Joe jobs [wikipedia.org] and address forgery. (It just so happens that most Joe Jobs are spam).

    • Re: (Score:2)

      by eric76 ( 679787 )
      SPF records can be useful to identify legitimate e-mail servers from selected domains.

    • Re: (Score:2)

      by Havokmon ( 89874 )

      SPF (and related technologies) are not designed to cut down on spam. They are designed to prevent Joe jobs and address forgery.

      I just went through this with a security company for a Visa audit, so let me expand on this. They seemed to think that checking the Mail From: for a local user, when sender wasn't authenticated (I would assume - we never actually got that far), was a valid way of checking for forgeries. There are multiple problems with this.

      • Their testing was flat out wrong to begin with. Th
  • The combination of 8 DNS blacklists, Amavis and Spamassassin works very well.
    I used to get more than 300 spam mails per day (intercepted by Spamassassin), due to the use of DNS blacklists I now only receive about 15 spam mails per day wich are intercepted by Spamassassin.
    Only about 3 spam e-mails per day actually make it into my mailbox, with zero false positives.

    The good thing about DNS blacklists is that the spam e-mails are actually rejected in the mail protocol, therefore it will hit spammers directly a
    • The blacklists also reject dynamic ip addresses, which are all virus infected home computers.

      *All*? I run a mail, gaming, and web server off of a dynamic IP. Forwards out through a smarthost, so blacklisting isn't a problem, but it isn't infected with viruses nor am I using it for illegit purposes (ok, well it probably does violate my ISP's TOS, but fuck'em).

      -b.

  • I use the libspf2 however, using it is quite useless when you come to think of it. In reality the concept is amazing, however, if you think of it, it relies on 3rd parties envolvement. When you implement SPF, you check other users domain SPF records for validation purposes, however, what if the other users haven't specified their own records? Some reputable and large ISP's still do not have SPF setup. In reality, using SPF is great... as long as everyone else uses it. Having to rely on others when it c
  • I'm the entire IT Dept at my work and I do not have the time to manage our own email server, let alone worry about keeping it secure. Most of our business comes in via email and most of those are crafted to look exactly like spam with huge lists of names in the TO: or CC: boxes and no subject line.
    My problem was finding a way to filter spam without filtering even a single legit email. Lost email means a lot of lost revenue. The only solution I found in a year of searching was mxlogic.com. We still get spam,
  • And that's been keeping the ones that get through down to two or three a week. Not enough for me to turn on hard SPF checking or demanding that email to me be encrypted with my personal PGP key. Configuring all that stuff certainly is a pain though -- it'd be nice if they could get it down to drop in components for the most common configurations.
  • The postfix server uses RBLs to drop about 25,000 messages per day. If postfix accepts it, it gets handed off to a different server that does SURBL checks. (That is done by a commercial product called GWAVA [gwava.com]). The SURBLs catch about another 2,000 messages per day.

    I have published my SPF data - so at least other people have the option of identifying whether stuff that claims to have come from my domain is legitimate or not. But our mailers are not yet doing SPF lookups. When we have a little time, we will pr

    • called GWAVA

      You're running Groupwise? GWAVA is overrated and is mainly useful for integrating spam filtering into Groupwise's Internet Agent. Nothing that SpamAssassing + ClamAV + ProxSMTPd won't do for you. And that combination is available as part of a package for an IPCop firewall box called CopFilter. The only downside is that CopFilter isn't as configurable as it should be via the Web interface. But for a free product, it's pretty darn good.

      -b.

      • Re: (Score:2)

        by Degrees ( 220395 )
        That's cool. If we got into a monetary crunch, we would probably implement what you mention here. One of the nice things about GWAVA is that it we have it configured to send an HTML message to users daily, where they can pull a message out of the bit-bucket (so-to-speak). That is to say, they get a digest message of what was blocked, and if something was improperly blocked, they can have it sent to them anyway. Version 4 (due out any minute now) will take this to the next level, where users can manager thei
        • One of the nice things about GWAVA is that it we have it configured to send an HTML message to users daily, where they can pull a message out of the bit-bucket (so-to-speak). That is to say, they get a digest message of what was blocked, and if something was improperly blocked, they can have it sent to them anyway.

          Copfilter has a digest option too. We're not using it ATM, since I have it set up to block only the most egregious examples of SPAM i.e. those with scores of 25 or above. The rest simply gets

  • I use a (usually) sophisticated biological neural network consisting of a multi-billion plus nodes with some primitive pre-determined wiring structures serving as a foundataion. Oh yes, and as preliminary step, I use dual-stage filters: spamassassin followed by crm114. Spamassassin seems to be fairly well behaved by not giving too many false positve spam indications, and CRM114 picks through the remainder false negatives to my satisfaction. I still end up picking through the spam folders, but its bulk and n
  • They've done so for the past few years, and it seems to work *very* well.

    See their web site here [postini.com]...

    • > They've done so for the past few years, and it seems to work *very* well.

      My previous ISP imposed Postini on me with no notice (they sent me an email bragging about it three days after they started using it). It passed 50% of the spam and stopped 20% of the ham. I turned it off.
      • Interesting. My ISP introduced it as an opt-in service (just like they introduced SpamAssassin and various other tools to the user base), and while it did require some fine tuning, I've had very few problems with it (I get a handful of Spams a day which it doesn't catch, and I see one or two false positives a month).

        I don't blame you for dropping it given how it was introduced at that ISP, but I think you also lost a chance to use a fairly effective anti-spam tool.
  • In other words, it's crochety as hell. I have all the "speak the RFC's exactly or thy shall not pass" options turned on. I publish a SPF record, for what good it will do. I also 5xx reject anything from overseas.

    Even though this is my own personal mail server, I haven't had too many false positives as far as rejects go... certianly nothing that a tweak here or there in the allow/deny hosts file wouldn't take care of.

    All in all, I've recieved less than a dozen pieces of spam in the last year and a half. Not

  • Greylisting (Score:3, Informative)

    by eric76 ( 679787 ) on Wednesday November 01, 2006 @01:45PM (#16675459)

    I use spamd on OpenBSD to do greylisting. That cuts an enormous amount of spam out.

    For those who aren't familiar with greylisting, when an smtp server attempts to deliver an e-mail the from address, to address, and IP address of the sender are put in a database and the mail is refused with a non-permanent error code.

    Assuming the smtp server sending the e-mail follows the RFC, it will try again later. When it tries again after at least 20 minutes from the original attempt, it accepts the e-mail and adds the IP address of the source to a whitelist. For the next 30 days, any e-mails from it are white-listed. After that, the server is verified again.

    I also keep a seperate white-list for non-RFC compliant servers and for frequent senders. Some servers only try one to three times and quit. Another problem is e-mail from some large e-mail farms may make each attempt to deliver the e-mail from a different server with different IP addresses, so I'll add their e-mail addresses to the white-lists as well.

    One method I use for adding IP addresses of selected senders that send a lot of legitimate e-mail to the whitelist is to look up their SPF records and use that to identify the usual e-mail servers for the domain.

    A few ISPs appear to put their entire address space in the SPF record. For example, panix.com's SPF record is

    panix.com text = "v=spf1 ip4:166.84.0.0/16 ip4:198.7.7.0/24 ?all

    Needless to say they don't get whitelisted since I only want to whitelist e-mail servers, not their users spam-zombie computers.

    In other words, I use the SPF records to identify legitimate e-mail servers from selected domains only.

  • I used to "roll my own" with SpamAssassin and MimeDefang. Then I started using CanIt [roaringpenguin.com] at work (I liked them initially because the author is the author of MimeDefang). They have a free version that works well for me at home now. We have been using it for about 4 years at works and it does a great job incorporating grey listing, SA, MimeDefang, ClamAV, etc. into an easy to install and maintain system with a nice web interface and a database backend. It can scale well when we need it to and the support is great


    • We were using MimeDefang + SA for a while, but it wasnt enough. Second the vote for Canit... just (as in Wednesday) rolled out Canit/PRO to serve mailboxes for 5000 full-time employees. Works well, cost is very reasonable. It has the benefit of the centralized solution for reduced maintenance, but we can use the web interface to customize mail flows for people with particular needs.

  • www.spampal.org
  • I pay poor children in [???] $0.01 / hour to filter my mail for me. It's cheaper then buying SPAM filtering software.

  • TMDA [tmda.net] catches all my spam. I does not examine content. It sends a request for response to all unknown senders. Since the vast majority of spam has forged return addresses, no responses are sent back and the mail stays in the TMDA pending queue until it expires. Humans, on the other hand, reply, and their mail is removed from the pending queue and gets through. When I set up TMDA, I populated the whitelist with all the email addresses of my correspondents and lists.

    Around 75% (150/200 daily) of my ema

    • "Since the vast majority of spam has forged return addresses, no responses are sent back" That's right and all the innocent owners of the forged addresses get stuck dealing with your spew. It works for you, but it's a pretty selfish way to go about it. Personally, I have not and will not ever respond to challenge/response schemes until I see one that manages to not spam innocent bystanders. If important enough, I call them. Otherwise, oh well.
  • We use ASSP at work (a government entity) and it is effective enough that when we DO have a spam slip through, users usally call to complain about it. It happens rarely enough that they forget to forward it to spam@<ourdomain.org>.

    I also use it at home and have nearly the same effectiveness.

    As far as various technologies, I don't believe any solution which relies sole upon one or two technologies will be that effective. ASSP seems to be the best so far at combining SPF/Greylisting/bayesian/various oth
  • There are lots of short-term solutions, to which spammers adapt as they get widely adopted.

    For example, content filtering in general is largely a short-term solution. Spammers invent and use obfuscation tricks; tools detect them, spammers invent new ones. Rinse, repeat.

    Longer term solutions have to address root causes. These increase the consequences of spamming. IP blacklists, URI blacklists, domain blacklists, for example, result in negative consequences for bad actors and their associates. (Includin

Slashdot Top Deals

I have hardly ever known a mathematician who was capable of reasoning. -- Plato

Close