Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Extending SpamAssassin and Amavis

Zonk posted more than 5 years ago | from the they-look-totally-natural dept.

Spam 25

An anonymous reader writes "Spam filtering solutions are a necessary evil in today's e-mail climate. There are many different tools and systems available for the filtering and removal of spam e-mail. Tools like SpamAssassin and more detailed agents, such as Amavis use a variety of different methods to identify and capture spam. An IBM article shows how you can extend SpamAssassin and Amavis, providing additional filtering facilities to lower the amount of spam hitting e-mail boxes."

cancel ×

25 comments

Sorry! There are no comments related to the filter you selected.

bwah (0)

Anonymous Coward | more than 5 years ago | (#22277444)

bwah. dan, jucam o dota?

I've comletely stopped spam (0, Funny)

Anonymous Coward | more than 6 years ago | (#22277524)

I recently developed a new technique to completely stop spam from reaching my inbox. I reject all incoming mail. Yes, there are obviously some false positives with this technique, but here's the great part, the more spam they send me, the lower the percentage of false positive I get!

Our just use GMail... (2, Interesting)

daveisfera (832409) | more than 6 years ago | (#22277770)

Or you could just use an email provider like GMail that has good spam filtering and let them worry about it.

Re:Our just use GMail... (1)

lattyware (934246) | more than 6 years ago | (#22280226)

Yeah, unless you happen to like IMAP like me.

Re:Our just use GMail... (1, Informative)

Anonymous Coward | more than 6 years ago | (#22280394)

GMail supports IMAP.

Re:Our just use GMail... (1)

Tacvek (948259) | more than 6 years ago | (#22284890)

GMail supports IMAP.

Indeed. But when I last looked, it used IMAP in a very strange way, that no mail client had good support for.

Here is a list that a client needs to be able to support for the best Gmail/IMAP experience:

  • Regular IMAP features
  • Support for flagging an unflagging a message. Ideal if the message flagging is shown as a star, as it translates to starring and un-starring a GMail message)
  • The client should have propper message threading support
  • The client should be able to deal well with a message being in more than one IMAP folder at the same time. Specifically, this should not crash it, and it should be able to store the message efficiently. (Not keeping multiple copies of the message on disk as part of the offline cache, but keep only one copy with references)
  • The client should not not save deleted messages locally, or in a special folder on the server
  • Any built-in spam blocking of the client should be disabled
  • The client should send messages through the Gmail SMTP server (this will ensure sent messages are added to that folder automatically).
  • The client should not store sent messages anywhere (as the smtp server will do that for it
  • The client should syncronize its "drafts" folder with [Gmail]/Drafts
  • The clients "mark as spam" feature (if it has one) should move messages to [Gmail]/Spam.
  • When in [Gmail]/Spam the "unmark as spam" feature (if any) should simply move the message to the "Inbox folder". It should be enabled for all messages in this folder.
  • When working in most folders (all except [Gmail]/Spam, [Gmail]/Trash, and possibly "[Gmail]/All Mail"), the button that sends the IMAP delete command should be labeled "remove tag"
  • In [Gmail]/Spam and [Gmail]/Trash folders, the IMAP delete command button should be labeled "Purge".
  • It is entirely unclear from the Gmail documentation what the IMAP delete command does to messages in "[Gmail]/All Mail". Whatever that does, the IMAP delete command button should be labeled appropriately.
  • There should be annother button available that moves a message to [GMail]/Trash that is labeled "Delete". (should be disabled folders where the IMAP delete button is labled "Purge")
  • Finally, there really should be a toolbar button labeled "Tag", when clicked drops down a list of the Imap folders (a cascading list because of the possibility of subfolders). Selecting a folder from this list should result in the IMAP commands to copy the selected message(s) to the selected folder. (It would be preferable to hide the "[Gmail]" metafolder and subfolders in this drop-down list).

At the moment, I am not aware of any IMAP client that can offer all that. Thunderbird didn't the last time I checked, and would require not only several completely new extensions, but some changes to the underlying source too. I would not be shocked if mutt could do all this, (obviously the scripting features would be needed), but even if it does, it would still require a fair amount of work. So at the moment no client is even close to ideal. I'm guessing Thunderbird will slowly gain many of the listed features, but it likely will never have them all.

Re:Our just use GMail... (1)

Doug Neal (195160) | more than 6 years ago | (#22294090)

Yep, Gmail's IMAP implementation is basic, to say the least.

* IMAP folders work, but for some reason on the web interface it calls them "labels" and lists them all in a tiny little side panel in a flat list, not as a hierarchy of folders.
* Occasionally the folders fail to synchronise for an hour or so at a time then mysteriously start working again.
* Only this morning I had a message that I just couldn't mark as read - kept popping up as an unread message on the next sync (argh!)
* The whole IMAP service is slow as hell

Gmail IMAP is pretty sucky and I think Google know it. After all, you don't see any ads when you're using your IMAP client...

I said it once, and I'll say it again... (2, Informative)

siyavash (677724) | more than 6 years ago | (#22277806)

I know this solution isn't fit for "average user" but I use white list aliases, when I write my email in some website or whereever, I do @domain.com , as soon as I start to get SPAM, two things happen :

1. I know WHERE the SPAM is comming from and who rat my email out.
2. I can close that specific alias and no more SPAM.

Using the above technique, I see ZERO SPAM... like good old days of emailing.

Nice (1)

Gazzonyx (982402) | more than 6 years ago | (#22278484)

So what about public boards where a spider can pick up your email address? That being said, this is a really good idea; do you have it automated at all? For instance, do you have a catch all that is white listed so long as the incoming email address isn't black listed? Or do you manually add a white list entry for each new email?

Re:Nice (1)

siyavash (677724) | more than 6 years ago | (#22280218)

I open up groups of aliases, for example forum.*@domain.com and when needing to use one, I just give away forum.@domain.com since forum.* is already open I don't have to open them up every time I need an "email", in the start spammers guess a couple of them, like sales@ info@ billing@ but those are easily closed.

So all open until I get spam from one of them and then I close it. As long as you have a fairly unique grouping forum.*@ news.*@ etc, no problem at all.

You see, in my opinion having ONE email and giving it to everyone is like having your door open in middle of the city. But with my way, you "create" a new door for every person out there and you can shut that door off quickly. I really like the fact that this way I see who rat my email out. You will be surprised how many sites sell your email.

And if I need to have my email on my website, I just use a mailform.

Try this 30 days and you will realize why email got so big in the start, because it's a great great communication tool when there is no SPAM. :)

Re:Nice (1)

siyavash (677724) | more than 6 years ago | (#22280230)

I also forgot, I use Outlook so when sending out emails I use a dead pop3 which can send but not receive like noreply@ or something, and then you just set the "Send replies to" option of the email to correct alias for that person.

Another thing, to automate the above, I found this tool called "Outlook bells & whistles" ( http://www.emailaddressmanager.com/outlook-bells.html [emailaddressmanager.com] ) pretty cheap too, it's an add-on you can install on top of Outlook and you can set rules like "If emailing to a specific person, then use this email as reply to" this way you won't have to manually do all the work.

Re:Nice (1)

Bert64 (520050) | more than 6 years ago | (#22282606)

"pretty cheap" ? You pay extra for such simple features as that?

Re:Nice (1)

siyavash (677724) | more than 6 years ago | (#22288236)

Yes ofcourse. If you do not have enough money to buy stuff at $30 and consider that "expensive", perhaps it is time to change your life or line of work. I eat double that amount every day ;)

Re:Nice (1)

Bert64 (520050) | more than 6 years ago | (#22307966)

It's not about being able to afford it...
It's about spending money on something that should be a standard default feature.
Paying an extra $20 for a car door isn't gonna break the bank, but you wouldn't expect a car to arrive without it's doors.

And it soon adds up, seems stupid to keep wasting small amounts of money.

Re:Nice (1)

karmatic (776420) | more than 6 years ago | (#22280332)

TMDA [tmda.net] does everything one needs. It can do whitelisting, blacklisting, time-based email addresses, challenge-response, and even includes a proxy you can use to automatically tag outgoing email for you.

Re:I said it once, and I'll say it again... (1)

Bert64 (520050) | more than 6 years ago | (#22282588)

I do something similar, but by creating a subdomain too, so for example:
mail@slashdot.org.mydomain.com
That way, not only do i know who ratted me out, but i can also change the MX records to point back at their own servers.

As for public boards, they should not really be posting plain email addresses. Tho it's fairly easy to identify when this has happened rather than someone ratting out my address.

On the other hand, slashdot's mail obfuscation has some interesting results, i quite often get mails to things like nospam.slashdot.org.mydomain.com etc.

What a useless article. (1)

Vellmont (569020) | more than 6 years ago | (#22278390)

Anyone that's done much exploration of spam filtering already knows the basic architecture of self-learning filters. This article has nothing new.

Recently I've implemented greylisting [wikipedia.org] on my mail server. The drop in spam has been enormous, though there have been a couple cases where email didn't go through on the first try.

Essentially it's a step inserted into the SMTP transfer. The first time a given email+ip address attempt to send you mail, the server responds "try again later". If it tries again after 5 minutes, the mail is accepted. If it tries again before 5 minutes, it gets another "try again later".

The only problem is a small number of SMTP servers won't try again after a "try again later" message. The number of poor SMTP servers is fairly small, I'd estimate 1%-2% of all users.

Re:What a useless article. (2, Insightful)

falzbro (468756) | more than 6 years ago | (#22279320)

Anyone that's done much exploration of spam filtering already knows the basic architecture of self-learning filters. This article has nothing new.
Well, since Amavis/Spam Assassin don't do what the article addresses out of the box, much of what's listed in it is useful. However, anyone interested in this article should probably check out Maia Mailguard [maiamailguard.com] , which does most of what the article talks about and much more, with a web interface.

Recently I've implemented greylisting on my mail server. The drop in spam has been enormous, though there have been a couple cases where email didn't go through on the first try.
Yes, greylisting is nice. However, this has nothing to do with Amavis/SA. Greylisting is another technique that one would use along side SA.

Re:What a useless article. (1)

Vellmont (569020) | more than 6 years ago | (#22284390)


Yes, greylisting is nice. However, this has nothing to do with Amavis/SA.

Nothing except that it's another spam fighting technique, which is the ultimate goal here.

Re:What a useless article. (1)

g_adams27 (581237) | more than 6 years ago | (#22279372)

Anyone that's done much exploration of spam filtering already knows the basic architecture of self-learning filters. This article has nothing new.

Which might be why the article is completely different than your one-sentence summary of it.

Bayesian filtering is briefly mentioned as a solution in the introduction of the article, before the author gets into the real meat of his suggestions on improving that baseline - spam-reporting mailboxes that are automatically processed and passed to SpamAssassin and Razor, improving the implementation of whitelists and blacklists, mail filtering for easier manual processing, and generating reports to determine which techniques are working and which aren't.

Seriously, go read the article. I'm glad the author wrote it; that Perl script for processing an IMAP mailbox looks quite handy.

Re:What a useless article. (1)

Degrees (220395) | more than 6 years ago | (#22279862)

To extend that a little further, something that is very effective is to set up a spamtrap email address. If you have a web site, you put a hidden email address on it that is solely used as a source of spam. If people cannot read it, but web-crawlers can, then you have your bad guys.

Using that Perl script to dredge the spamtrap mailbox via IMAP makes an easy source to train against.

Re:What a useless article. (1)

Vellmont (569020) | more than 6 years ago | (#22284372)


Seriously, go read the article.

I did read the article. It had nothing useful in it for me.

Flexible smtp daemon? (2, Informative)

r0.ini (631472) | more than 6 years ago | (#22279286)

If you need a real flexible smtp daemon, and can program in perl, I would recommend qpsmtd [develooper.com] . Give it a try, you can create your own plugins (to handle spam or whatever you need) so easily you won't believe it.

SpamAssassin is a huge memory waster (0)

Anonymous Coward | more than 6 years ago | (#22280502)

SA is not a bad application but in my own experience it is a huge memory eater. I personally find DSPAM and/or CRM114 and/or OSBF-Lua better suited to catch spam/ham.

I use Postfix 2.5.0 and have on top of it running DSPAM, DSPAM RABL, SQLGrey, policyd-weight, postfwd, DKIM-Milter, SID-Milter, DCCM (DCC as a milter), SPF, AMaViS-New, a bunch of Postfix own anti-UCE techniques and other small things. All nice installed and glued together. It's working like a Swiss watch. And I am able to filter more mails per second then with SpamAssassin on the same system. I have less false positive/negative and I give full control to the end user (thanks to DSPAM) to influence his/her filtering.

For me SA is a nice program and I know it well but it takes way way to much resources from my system for simple filtering. When ever I do benchmarking I realize that in SA I need to count the SECONDS PER MESSAGE and in other tools like DSPAM/CRM114/OSBF-Lua I count the MESSAGES PER SECOND. That is a huge difference.

Speed is not everything but the memory usage is a problem. SA just eats and eats and eats memory. Why can't the developers make SA to stay below 10MB or 20MB memory usage? My DSPAM uses less then 4MB memory. From time to time when it needs to process a huge mail, then memory usage jumps up but gets back down again. On SA I can watch the memory being taken over by SA. Every hour this monstrum uses more and more. I hate that!

Since I implemented filtering. (1)

Z00L00K (682162) | more than 6 years ago | (#22284570)

Since I implemented filtering using several different services I haven't seen any junk mails.

I have the following config in my sendmail.mc:

FEATURE(`require_rdns')dnl
FEATURE(`block_bad_helo')dnl
FEATURE(`enhdnsbl', `zen.spamhaus.org', `"Message from $&{client_addr} rejected - see http://www.spamhaus.org/query/bl?ip="$&{client_addr}', `t')dnl
FEATURE(`enhdnsbl', `bl.spamcop.net', `"Message from $&{client_addr} rejected - see http://spamcop.net/bl.shtml?"$&{client_addr}', `t')dnl
FEATURE(`dnsbl',`combined.njabl.org',`Message from $&{client_addr} rejected - see http://njabl.org/lookup?$&{client_addr}')dnl
FEATURE(`dnsbl',`list.dsbl.org',`Message from $&{client_addr} rejected - see http://www.dsbl.orgdnl/
FEATURE(`dnsbl',`dnsbl.sorbs.net',`"Message from $&{client_addr} rejected - see http://www.sorbs.net/"')dnl
FEATURE(`dnsbl',`dnsbl-1.uceprotect.net',`"Message from $&{client_addr} rejected - see http://www.uceprotect.net/"')dnl
FEATURE(`dnsbl',`dnsbl-2.uceprotect.net',`"Message from $&{client_addr} rejected - see http://www.uceprotect.net/"')dnl
FEATURE(`dnsbl',`dnsbl-3.uceprotect.net',`"Message from $&{client_addr} rejected - see http://www.uceprotect.net/"')dnl
And I haven't had any persistent problems with legitimate emails coming through, which means that this setup works relatively well. I can't claim that this list is the ultimate or that it's perfect, but it works for me. The disadvantage is that it requires Sendmail [sendmail.org] , but for any *NIX hacker this shouldn't be a problem.

There isn't even any problem doing a secure setup for persons roaming, in which case it's possible to set up a SMTP AUTH on a different port. I have at the same time elected to use SMTPS (SMTP over SSL), which means that any password and information sent over the net is encrypted.

Below is the code I use for listening on a secondary port (465/smtps) with AUTH and certificate handling for encryption.

DAEMON_OPTIONS(`Port=25')dnl
DAEMON_OPTIONS(`Port=465, Modifiers=as')dnl
define(`confPRIVACY_FLAGS', `noexpn novrfy authwarnings')dnl
define(`confFALLBACK_MX', `smtp.bredband.net')dnl
define(`confCACERT_PATH', `...ssl')dnl
define(`confCACERT', `...ssl/cacert.pem')dnl
define(`confSERVER_CERT', `...ssl/certs/smtp.pem')dnl
define(`confSERVER_KEY', `...ssl/certs/smtp.pem')dnl
define(`confAUTH_OPTIONS', `p,y')dnl
TRUST_AUTH_MECH(`GSSAPI DIGEST-MD5 PLAIN LOGIN')dnl
define(`confAUTH_MECHANISMS', `GSSAPI DIGEST-MD5 PLAIN LOGIN')dnl
You will have to hack the path "...ssl" into something real if you are going to use the above. And be familiar with OpenSSL [openssl.org] .

At least the actions I have taken discourages the spammers good enough and makes me feel reasonable safe. (there is always another leak, but you have to find it first).

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?