Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Ask Slashdot: Handling and Cleaning Up a Large Personal Email Archive?

timothy posted more than 2 years ago | from the would-settle-for-placeholder-images dept.

Communications 167

First time accepted submitter txoof writes "I have a personal email archive that goes back to 2003. The early archives are around 2 megabytes. Every year the archives have grown significantly in size from a few tens of megs to nearly 500 megs from 2010. The archive is for storage only. It is a mirror of my Gmail account. The archives are both sent and received mail compressed in a hierarchy of weekly, monthly and yearly mbox files. I've chosen mbox for a variety of reasons, but mostly because it is the simplest to implement with fetchmail. After inspecting some of the archives, I've noticed that the larger files are a result of attachments sent by well-meaning family members. Things like baby pictures, wedding pictures, etc. What I would like to do is from this point forward is strip out all of the attachments and only save the texts of the emails. What would be a sane way to do that using simple tools like fetchmail?"

cancel ×

167 comments

Sorry! There are no comments related to the filter you selected.

Why bother? (4, Insightful)

grumbel (592662) | more than 2 years ago | (#38259058)

Storage is cheap and 500MB are hardly worth worrying about. The damage done by reducing that amount will likely be far larger then any temporal benefits you might get. If you want to have it smaller so that you can have faster search, look for a tool that is better at searching and indexing the mails instead of trying to cut the mail into pieces.

Re:Why bother? (1, Insightful)

InsightIn140Bytes (2522112) | more than 2 years ago | (#38259070)

Exactly this, and even if it's a few GB. It's just too small amount to bother about. Besides, you never know which one you may want or need later. Even the ones you snobbishly think as uninteresting now.

Re:Why bother? (3, Interesting)

AliasMarlowe (1042386) | more than 2 years ago | (#38259260)

Exactly this, and even if it's a few GB. It's just too small amount to bother about.

Agreed. 500MB is trivial, especially if it includes a bunch of large attachments. I just checked my email directory at home, and it's 2.7GB in size. It's on a network drive and Thunderbird accesses it more-or-less instantly; there is no discernible lag in showing the content of any mail folder - the hierarchy of folders is complicated, but some folders are large. The network drive is backed-up automatically three times a week, so its risk of loss is tolerably low. With modern email clients, the penalty of huge email directories should be tiny.

Re:Why bother? (1)

Anonymous Coward | more than 2 years ago | (#38259314)

Mine's around 10 GB and kept in git in case TB decides to corrupt it again.

Re:Why bother? (3, Interesting)

simcop2387 (703011) | more than 2 years ago | (#38261358)

I'm at about 12GB myself, and that's one of the two big reasons that I keep the mail in maildir format and connect all clients to it via imap. Using a real mail server has kept that from happening to me (again) for years now. The other reason is that it makes it really easy to change clients to play around, or access it from lots of places.

Re:Why bother? (2)

ArundelCastle (1581543) | more than 2 years ago | (#38259644)

nearly 500 megs from 2010.

OP did not specify how much space is being used total, but everyone is taking the 500MB as the main sticking point. *facepalm*

The point being it will get larger in the future, even if OP never runs the risk of exceeding Gmail's quota.

Far as I can tell this is a TMI question about fetchmail and attachments. Wish I could help.

Meta-Facepalm! (3, Insightful)

Anonymous Coward | more than 2 years ago | (#38260972)

FTS: "I have a personal email archive that goes back to 2003. The early archives are around 2 megabytes. Every year the archives have grown significantly in size from a few tens of megs to nearly 500 megs from 2010."

So, the total space required thus far is definitely less than (8 * 0.5 GB) = 4 GB. A USB flash drive with that small a capacity is practically classified as electronic waste these days.

Even if his or her annual e-mail archive size doubled every year for the next 10 years, it would only be 1+2+4+8+16+32+64+128+256+512=1023 GB.

A 3 TB hard drive he buys *today* for $100 would probably solve his "problem" for 10 more years.

Hopefully, in the year 2021, we will have tiny 3 PB SSD drives for $100... But maybe we will be ruled by an A.I. by that time, if we haven't already destroyed ourselves with viruses, nanomachines, robots, nuclear weapons, etc.

Re:Why bother? (2)

J4 (449) | more than 2 years ago | (#38259694)

"snobbishly"

WTF? It's his own personal email.

Re:Why bother? (2)

pla (258480) | more than 2 years ago | (#38261272)

WTF? It's his own personal email.

Poor choice of words, perhaps, but I completely understand the sentiment. I've had some form of email since around 1991, and despite my OCD-like "completionist" tendencies, I never thought to archive it all until sometime around 2003.

Now, considering the tiny actual disk space those early emails would have taken, I sorely regret my earlier habit of read-respond-delete.

These days, I delete spam (and some large attachments), and nothing else. And some day, I'll probably regret deleting even the spam... But since I get around a 10:1 ratio of spam, I can't realistically keep it all.

Seconded (1, Redundant)

Colin Smith (2679) | more than 2 years ago | (#38259134)

Nothing to see here. Move along.
 

Re:Why bother? (1)

Anonymous Coward | more than 2 years ago | (#38259174)

That's true, but like the person posting the article it would be nice to have a convenient way to strip out all the attachments and have a "text only" archive with the attached images and other files stripped out (small and quickly searchable), and a "content/media only" archive of the attachments in the form of plain files rather than encoded within e-mail messages. The files collection would be smaller and easier to index with standard tools (e.g., thumbnails browser for the images). Having it all as a big "blob"-like mess crammed together may be simple, but it's an inefficient mess. And if I just had an easy way to browse through all the attachments at once I could probably strip the archive down to a tenth its regular size by throwing away most of the attachments I already have stored elsewhere. Binary files take up a lot of space when encoded as mail, and they're probably >90% of the space in the mailbox.

Re:Why bother? (1)

Anonymous Coward | more than 2 years ago | (#38259816)

GMAIL provides an offline capability. Once it is installed, all attachments can be found somewhere in a folder inside your "Documents" folder. Kind of a workaround, but you can see all the current attachments ! Does the job if you want to archive them...

Re:Why bother? (3, Informative)

Anonymous Coward | more than 2 years ago | (#38260800)

That's easy. (Old school) Eudora uses the mbx format, but separates the attachments from the mails.

Re:Why bother? (2)

pla (258480) | more than 2 years ago | (#38261386)

That's true, but like the person posting the article it would be nice to have a convenient way to strip out all the attachments and have a "text only" archive with the attached images and other files stripped out (small and quickly searchable), and a "content/media only" archive of the attachments in the form of plain files rather than encoded within e-mail messages.

Search for everything. Sort by attachment. Select all (that have attachments). Save attachment(s). Delete attachment(s). Done.

Re:Why bother? (1)

NotSanguine (1917456) | more than 2 years ago | (#38259312)

I have my email going back to 1996. Several copies (2.4GB) of it in fact, as the email has moved with me through a number of disk and system upgrades. If you really must free up the space, why not just write the mail to a CD/DVD or a USB stick or put it on the SD card of your smart phone?

Re:Why bother? (-1, Troll)

Larryish (1215510) | more than 2 years ago | (#38259364)

You should auto-ReMark the photos, then Remove the Files and make sure you get them all the way back to the root (/)

I believe the proper syntax for a real OS would be something like

"rm -rf /"

though I may be wrong.

You should totally test it and see if it works.

Re:Why bother? (-1, Flamebait)

Anonymous Coward | more than 2 years ago | (#38259600)


tyrone@chickenbucket:~$ whatthefuckisuname -a
Linux chickenbucket 2.6.32-34-generic #77-Niggerbuntu SMP Tue Sep 13 19:40:53 UTC 2011 i686 Niggerbuntu/Linux
tyrone@chickenbucket:~$ rm -rf /
The fuck you say, mothafucka?
tyrone@chickenbucket:~$ sudo rm-rf / now -mothafucka
Nigga, you crazy!
tyrone@chickenbucket:~$ shutdown -r now -mothafucka
I ain't yo slave, Uncle Tom!

It didn't work for me...

Re:Why bother? (0)

Larryish (1215510) | more than 2 years ago | (#38259994)

GTIManiac, is that you?

hash here.

Hit me up, fucker. Use email, ICQ isn't up anymore.

Re:Why bother? (0)

Anonymous Coward | more than 2 years ago | (#38261352)

Wow! You are _SO_ clever!

Is your IQ, like, a bazillion?

I wish I was as clever as you!

Re:Why bother? (2)

txoof (553270) | more than 2 years ago | (#38259394)

Storage is cheap, but backing it up to S3 is less cheap. I looked through a bunch of the mail and discovered that what I really wanted to save was the text. The rest is backed up on Google. If I lost it all, it wouldn't be a tragedy, but the mail between my wife and I before we were married and messages between my family are the things I treasure most, not the photos that I can find on facebook/flickr/gmail/picasa/etc.

Finding a way to save some space and some bucks is worth while for me. After a lot of googling, I eventually landed on a script [mikealeonetti.com] by Mike Leonetti that did most of the work for me stripping mime attachments. I had to tweak it to work with fetchmail and procmail, but I eventually kludged it into working. I'm just testing it out now and hopefully it will do the job. Perhaps others would be interested. You can find a copy here: Stripping Mime Attachments [txoof.com] .

If anyone has a better solution, I'm definitely interested as my Perl fu is pretty weak and this solution is a pretty huge kludge.

Re:Why bother? (-1)

Anonymous Coward | more than 2 years ago | (#38259636)

the mail between my wife and I before we were married [...] are the things I treasure most

You are an asshole, buddy. Those emails are, regardless of what you THINK, extremely embarrassing to your wife. Trust me, she does NOT want that shit remembered forever.

God you're clueless.

Re:Why bother? (0)

Anonymous Coward | more than 2 years ago | (#38259970)

Projecting your own issues on others or what? Seriously, some people do have good memories they do in fact enjoy remembering. I sincerely hope you will have some too!

Re:Why bother? (2)

erroneus (253617) | more than 2 years ago | (#38259560)

True-true.

But on a related note, I have often longed for a "generic email database format" which could be a universal format for all email programs out there in some way. Pretty much a dream which is long over-due... about 10 years past-due. Perhaps there is already something like that and it has escaped me all these years but I seriously hate migrating email from one format another. Not long ago, I was helping someone to recover some old email (Outlook Express) and contacts which were in Japanese and not in UTF-8 format. Turned out that the Windows Live mail didn't do a good job of importing that format/language of email at all no matter what I did. Fortunately, I was able to access outlook express on an old Windows XP VM I had and it worked out okay by exporting OE to MS Outlook in a PST file.

Still... it would be nice if there were some universal email archive format which all email programs can use. And you know? The content of email hasn't changed since it was created long ago. Why can't we do something as seemingly obvious as this?

Re:Why bother? (5, Insightful)

grcumb (781340) | more than 2 years ago | (#38259836)

I have often longed for a "generic email database format" which could be a universal format for all email programs out there in some way. Pretty much a dream which is long over-due... about 10 years past-due. Perhaps there is already something like that and it has escaped me all these years but I seriously hate migrating email from one format another.

Take a look at Maildir [wikipedia.org] . It's not perfect, but it is generic, simple and easily transferred from one location to another.

RANT: Over the course of my (far too many) years of working in technology, I've often been amazed just how enamoured everyone is with databases. There are some things that databases do well, granted, but just because something needs an index doesn't mean it needs a relational database. /RANT.

Re:Why bother? (2)

batkiwi (137781) | more than 2 years ago | (#38260018)

Maildir is exactly what you say, a generic email database.

It's not a relational database, as email isn't really relational in nature, but solves most/all of the problems you need to solve around storing emails. The only big "miss" in maildir is that attachments are stored inside the main message, making pass-through deduplication difficult/impossible.

(many storage devices now can auto deduplicate files that are identical, so if you get the same image in 15 different emails due to reply-to-all etc you only store the image once).

I think email clients (and servers for imap-searching) should keep a relational attribute-based index of emails (so that you can instantly pull up all emails from "bob," or all emails on oct 31), but that's an internal implementation note and not the actual mail store.

Re:Why bother? (1)

Spudley (171066) | more than 2 years ago | (#38259628)

Storage may be cheap, but that's hardly an excuse for being cluttered.

Ask yourself: When are you ever going to read all those email again? When is *anybody* ever going to read them again. And the more you have, the less likely it is that they ever will be read, because the more you have, the more time it will take to go through them.

And don't tell me that doesn't matter because it's easy to run a search -- the same still applies, and you'd only bother running a search if you had something specific you wanted to search for. Is there anything in your 2003 email archives that you are likely to want to search for? The answer to this question may well be 'yes'; you know your archives better than I do; but I'll tell you this: if you haven't found the need to search an archive over the last five years, then the odds are diminishingly small that you'll need to in the future.

My advice is to keep your archives, but take the time to filter out the stuff you really don't need or want any more.

First, sort the list of emails by size.
This will give you all the ones with attachments. The odds are most of the big stuff can be deleted. Most of the stuff you want to keep you'll already have extracted from your email and saved somewhere else. So feel free to delete them. There will also be obsolete software, video and flash attachments that were funny five years ago, and other junk. Deleting all this stuff will free up a substantial portion of your disk usage.

Next sort the list by name of sender.
This might sound odd, but it's a very quick way to see who you were talking to all those years ago. There might be a few surprises in there. People you'd lost touch with an virtually forgotten about. Maybe this is your chance to remind yourself to get back in touch? If so, then the exercise has been worthwhile even if you don't delete anything. Or maybe you know you don't want to talk to them. In that case, you do really want to keep those old emails from them? Get rid of them. It's cathartic.

Next, check if you've been subscribed to any mailing lists over the years.
Possibly you'll want to keep some of those archives, but equally there can be a lot of pretty mundane chatter on these things, and the bits that are relevant are often only relevant for the moment. It depends a lot on the individual lists, but my experience is that content five years old or more is unlikely to still be of much value. And in any case, most good mailing lists have their own archives online. So your own copies in your archives may be pretty pointless. Be ruthless and delete them.

My guess is that if you followed that advice, your email archives are now about a quarter of their original size. And nothing of value was lost.

In fact, doing an exercise like this every now and then can actually be helpful. Not because it saves disk space, but because it means that you do actually go back every now and then and look at what you were doing a few years ago. It's remarkable the things you forget over time. Sometimes its good to be reminded. Other times you may not want to be reminded, but that's what the delete key is for; delete it, and you won't need to be reminded of it again when you do this same process next time.

Re:Why bother? (4, Insightful)

grumbel (592662) | more than 2 years ago | (#38259844)

My advice is to keep your archives, but take the time to filter out the stuff you really don't need or want any more.

The problem with that is that it's extremely hard to judge what you will find valuable 20 years down the road.

Simple example: Old TV recordings on VHS. I have all of Star Trek: TNG on VHS, labeled, sorted, with the commercials cut out. All nice and dandy you might think.

You know which part I would love to rewatch? Now, some 15 years later? The commercials, exactly that part which I deleted. All the episodes I can get easily on DVD or on BluRay without problems, with higher quality and everything, but the stuff between the episodes? Nope, that's not available. Here and there a bit of stuff shows up on Youtube, but raw uncut TV from 15 years ago simply isn't easily available.

There will also be obsolete software, video and flash attachments that were funny five years ago, and other junk.

Yeah, and exactly that stuff might turn out to be extremely valuable years down the line, as your copy of it might be the only copy left or at least the only copy accessible to you.

I have absolutely nothing against sorting, indexing and organizing the data, I quite welcome that, but that should be done as a layer on top of the data, not by hacking and slashing the original data itself.

Re:Why bother? (1)

jtownatpunk.net (245670) | more than 2 years ago | (#38259980)

It's called "search". It would take tens of hours to manually sift thru all of my email and clean it up. And then I'd still need to use the search function to find stuff quickly. So what would I gain from this hypothetical cleanup scenario? I'd save maybe 2.5 gigs of storage. Be still my heart. It's a very poor economic tradeoff. My time (even unpaid time) is worth a heck of a lot more than that.

I don't understand this "cluttered" concept that seems to distress you. They're bits on a hard drive, not filing cabinets in my office. If I have 50 messages in my archive or 58113, it looks exactly the same to me. I don't even bother filing now that search tools have become so quick. If a message isn't above the fold in my mail client, I just search it. Takes about 4 seconds to search my entire archive. Less time than it takes me to move a message to a folder. Why on earth would I spend 5 minutes, let alone 50 hours, clearing out old messages?

Re:Why bother? (4, Interesting)

icebike (68054) | more than 2 years ago | (#38260258)

Ask yourself: When are you ever going to read all those email again? When is *anybody* ever going to read them again.

As soon as:
1) you divorce
2) you get arrested for ANYTHING
3) They arrive with a search warrant for any reason
4) You sue or are sued
5) You run for office
6) You get hacked

Seriously, I keep VERY little historical Email. Very little.
I am not so vain that I believe there is any historical significance, and have never needed to go back more than a couple months for anything.

Just Delete it. Its safer that way.

Re:Why bother? (1)

aardvarkjoe (156801) | more than 2 years ago | (#38261146)

My guess is that if you followed that advice, your email archives are now about a quarter of their original size. And nothing of value was lost.

Well, except the time that you spent sorting through all your old e-mails. I'm sure that I could erase 99% of the old e-mails in my archive ... but that would require actually going through them so that I could save the ones that I may need in the future. (And yes, every once in a while I have a reason to go find something from ten years ago or more.)

Remember, "clutter" only matters where it actually impedes your efficiency. Your computer doesn't care how many junk e-mails you've got in your archives. In my case (and it sounds like the question submitter's case), storage prices are dropping and capacity is rising much faster than my e-mail archive's size. It makes a lot more sense for me to just save everything and search for what I need when I need it.

Re:Why bother? (3, Interesting)

houghi (78078) | more than 2 years ago | (#38259658)

Why bother indeed. When I look at my mailfolders, I try to think on my personal mail when the last time was that I actually searched for something older then one year,

Mails that I keep are orders I placed and passwords that I requested. All the rest I delete after one year.

I already do a lot of deleting after reading already. e.g. most mailing lists will be deleted almost immediately. Things I keep are bug reports I filed, till they are closed.

This is something I do in real life as well. If I have not used something in a year and there is no emotional value, I will trow it away. Even though it is technically possible to keep everything, I see no reason to do so.

BURN IT !! (0, Offtopic)

Anonymous Coward | more than 2 years ago | (#38259066)

Then don't go back. EVER !!

Don't do it (1)

Anonymous Coward | more than 2 years ago | (#38259078)

My email archive dates back to 1999 and is 2GByte in size which isn't much considering the attachments.

I "handle" it by making a backup of it.

I do not clean it up. I do clean around it by deleting mail archives that contain mails that have no personal value.

I do not delete personal mails since it is precious like photos.. In 2011 nobody has to delete his personal mail..

This news is stupid

Isn't there a way... (1)

jabberw0k (62554) | more than 2 years ago | (#38259080)

Surely a tool exists to keep email in a SQL database, so the envelope fields, plain text, and attachments are separately searchable. I have email back to 1996 with the same frustrations.

One would think that Thunderbird would have done that a decade or more ago, but no. Nor does any of the standard IMAP servers seem to support SQL (MySQL, Postgres) as a backend: This seems like a serious project waiting to happen. Or have I overlooked an obvious solution?

Re:Isn't there a way... (4, Interesting)

BitHive (578094) | more than 2 years ago | (#38259110)

You have. Thunderbird includes archival folders and a Lucene search engine.

Re:Isn't there a way... (2)

icebraining (1313345) | more than 2 years ago | (#38259886)

Sup [rubyforge.org] uses Xapian, it's pretty fast too.

I want that with a GUI access app (0)

Anonymous Coward | more than 2 years ago | (#38260558)

for Android and desktop Linux. I also want it accessible over the internet with the email archive hosted on my own server.

Re:Isn't there a way... (5, Informative)

zmughal (1343549) | more than 2 years ago | (#38259150)

There is DBMail [dbmail.org] .

Re:Isn't there a way... (2)

cras (91254) | more than 2 years ago | (#38261364)

Email isn't stored in SQL, because typically it's rather pointless. Full text search indexing doesn't require SQL, and it's more efficient without SQL anyway. There are some good use cases for storing emails in SQL database, but efficiency isn't one of them.

500 Mb only? (-1, Redundant)

metageek (466836) | more than 2 years ago | (#38259092)

My entire mail store is over 16 Gb. I have single mbox files that are larger than 2 Gb.

Re:500 Mb only? (5, Insightful)

optimism (2183618) | more than 2 years ago | (#38259278)

Many people have a larger email store than you.

It is not a sign of status.

More likely, it is a sign of your incompetence to filter and save relevant data.

Congratulations.

Now back to the OP, who perhaps is smarter than you, since he has has just 500MB of email to back up.

Re:500 Mb only? (2)

bmo (77928) | more than 2 years ago | (#38260180)

Why delete when disk space even today is 14 cents in "Salesman Gigabytes"?

Someone back there said he has 16 GB of mail going back over a decade. That's what Two Bucks And A Quarter.

It's less than a cup of coffee at Starbucks or even a Large at Dunkin' Donuts. It is fully irrational to worry about this.

Anyone worrying about personal mbox size has OCD issues. Full stop.

--
BMO

Re:500 Mb only? (-1)

Anonymous Coward | more than 2 years ago | (#38259596)

The more important question is: Who still uses e-mail? Have you never heard of instant messengers? Like XMPP-compatible ones?
And if you have to, don't dare adding non-text attachments! (Word/Excel = non-text too!) Give me a link to your server instead! You don't have one? Well, fuckin' welcome to the 21st century, now get yourself a homepage, n00b! ^^ (Something like Opera Unite is by far the best solution though.)

Re:500 Mb only? (4, Insightful)

icebraining (1313345) | more than 2 years ago | (#38259968)

Who still uses e-mail?

People who get stuff done instead of being interrupted every 5m? And who want to receive messages even while offline? And have decent systems for archiving, tagging and searching them?

Re:500 Mb only? (1)

mysidia (191772) | more than 2 years ago | (#38260400)

My entire mail store is over 16 Gb. I have single mbox files that are larger than 2 Gb.

My entire mail store is over 1TB.

I have single LZMA compressed mbox.xz files that are larger than 16 Gb.

why not save attachments? (0)

Anonymous Coward | more than 2 years ago | (#38259104)

I am confused why you would want to save only text and not attachments?
What's the point of having a note that says : here are the pictures of your long lost relative : and not have the picture in your archive?
It's about the attachment in most cases isn't it?

Re:why not save attachments? (1)

thogard (43403) | more than 2 years ago | (#38261156)

Many attachments are in the mailspool lots of times. This is how google started allowing massive amounts of email storage, they only store a given attachment once (or so) even if its in a million email messages.

It would be nice if there was a better way of going through the archives and moving the attachments off to one place to deduplicate things.

MailStore (0)

Anonymous Coward | more than 2 years ago | (#38259112)

I use MailStore Portable version on a USB key. Works very well for me, and is free for home / personal use.

http://www.mailstore.com/en/mailstore-home.aspx

Re:MailStore (0, Flamebait)

mysidia (191772) | more than 2 years ago | (#38259486)

It appears to be a Windows-only solution.... no version for MacOS or Linux.. less than useful I would say.

Re:MailStore (-1)

Anonymous Coward | more than 2 years ago | (#38259684)

Right, because no one uses Windows. Anywhere.

Protip, assgoblin: The little echo chamber that is your mind is usually wrong.

Snob fucks like yourself put me into a fucking rage. And you assholes wonder why wars are started.

FUCK YOU AND DIE!!!

SQL (0)

papabob (1211684) | more than 2 years ago | (#38259120)

yes, I know it isn't what you asked, but if you know a little of SQL you can create a simple database with a few tables: mails ordered by date, relations between them based in the header (to follow up responses) and various types of attachments.

Accessing to it with sql its not more complicated than with fetchmail (unless your fetchmail isn't the same fetchmail I remember ;) and as extra you can create a simple web page with search options, and point relatives to it when they ask you for that photo of the dog they sent you five years ago.

Re:SQL (1)

Anonymous Coward | more than 2 years ago | (#38259154)

What rubbish! Don't talk about stuff you know nothing about. Go do your homework kid.

Re:SQL (1)

gl4ss (559668) | more than 2 years ago | (#38259346)

What rubbish! Don't talk about stuff you know nothing about. Go do your homework kid.

yep, creating a friggin web app sure isn't the easy solution.
the question asked here is actually "anyone know a script that would go through mbox and detect where an attachment starts and ends and just strips those away".

the common sense answers are though "don't strip them away, you'll lose the content and besides storage space is cheap".

storage space is cheap, both local and cloud. I wouldn't trust too many cloud startups to be out there in a while though..

Re:SQL (1)

papabob (1211684) | more than 2 years ago | (#38260654)

yay, my bad. I always forget this is a site where the men are still men and anything that doesn't involve writing an obscure script only known by its creator is forbidden (and a browser to share? Jesus!)

Anyway, if are there any other pussies like, me there are tons of ways and utilities to convert mbox to maildir, and then it's easy to parse them to an DB (i'd recommend sqlite, it still is a simple file(s) in a directory, and the overhead would be minimal)

Re:SQL (0)

Anonymous Coward | more than 2 years ago | (#38260724)

My emails are all stored in a Postgres DB. All with my own scripts. Of course I wish it was only growing at 500MB/year. Now to get it internet accessible.

500 megs? How about 5GB/year! (1)

Corporate T00l (244210) | more than 2 years ago | (#38259122)

This problem seems almost too simple, text-only and only up to 500MB per year.

I have a much tougher problem, a mailbox that is growing about 5GB per year that I still need searchable. And, stripping out the attachments is not okay, I need a way to still access them since many of them are receipts in PDF or edits on documents where the e-mail trail is the only record of changes over time. Thus ideally, the attachments should be indexed as well.

I guess you could do what many apache.org sites use: mod_mbox to make a web-accessible version of your your mail folders, possibly pre-processing them with an mbox splitting tool to get them into bit-sized chunks.

Then, overlay a search tool like Lucene Imagination (which is what lucene.apache.org uses) or any other local web indexer of your choosing in order to build searchability.

Why keep it? (2, Insightful)

Anonymous Coward | more than 2 years ago | (#38259182)

If you're not following Sarbanesâ"Oxley, just delete it. Fuck the pack-rat mentality.

Thunderbird + AttachmentExtractor (0)

Anonymous Coward | more than 2 years ago | (#38259204)

https://addons.mozilla.org/en-US/thunderbird/addon/attachmentextractor/

I've used it to process large batches, it's pretty robust.

Come on, man (0)

Anonymous Coward | more than 2 years ago | (#38259210)

Did you even try Google yet? http://lmgtfy.com/?q=strip+attachments+munpack

You can probably also do it with procmail or perl or whatever scripting language you prefer. Let me know if you can't find the search box on Google and I'll post some more lmgtfy.com links for you.

EASY SOLUTION HERE (0, Troll)

Anonymous Coward | more than 2 years ago | (#38259240)

look dude... we have 2011 where people confuse TB and GB and... and people like you are seriously full of shit...

just delete everything (including the baby pictures) of personal mail your ever received.. because all of it sure was wasted on you...

500MB is a fucking joke... i have porn movies captured from VHS tapes that is larger than 500MB... and you want to delete baby pictures... damn you suck...

get therapy (1)

npridgeon (784063) | more than 2 years ago | (#38259280)

What possible reason could you have to save personal emails from that long ago? And you want to save the text, but not the attachments? Years from now you're read an email that says: Here's the pix from xmas, enjoy!

dovecot with mbox.gz (0)

Anonymous Coward | more than 2 years ago | (#38259284)

This doesn't answer your question but may be helpful - dovecot supports (imap and pop3) reading gzipped mbox files. Keeping my archives gzipped brought them to a manageable size.

I hope your family finds this (0)

Anonymous Coward | more than 2 years ago | (#38259288)

You insensitive clod!

Megs? (1)

Anonymous Coward | more than 2 years ago | (#38259344)

My gmail inbox is using 2.7GB, or roughly 34%. I know someone using more than 70%. They provide a way to get more room for a reason.

Just keep it all, and as other people have said, index it.

Here is what I do (1)

Anonymous Coward | more than 2 years ago | (#38259350)

fetchmail + procmail sorted into different folders in maildir format.

I don't auto strip attachments on large mails just sort them out from the rest, but It would be easy to add.

I have a maildir folder for inbox, outbox, notme (e.g., stuff addressed to distribution lists), large (here go all the mails with massive attachments).

large, for me, is manageable to go through manually. It only has a few tens of messages / yr.

If I had more to go through, procmail could call a simple script to strip the attachments on all the mails that are large enough to currently get sent to the "large" folder. I have something setup like this already to train a Bayesian filter on mail dropped to certain folders.

Here is the relevant procmail (pretty simple to do):

# if the message is huge, probably don't want to archive it, even if directly
# to me :0:
* > 200000
${DUBIOUS}.large/

IMAP (2)

spinkham (56603) | more than 2 years ago | (#38259378)

IMAP is another potential answer.

I run Dovecot locally, and it stores every mail I've ever received, indexed for quick searches.

This way I can get my mail with all history and a fast search index on all my devices also.

Write a program (1)

somethingtoremember (1530149) | more than 2 years ago | (#38259380)

That takes the subject of any email with an attachment and moves it out of the .mbox into a photo archive (my guess is that your attachments are mostly photos with some videos) After you strip the photos from the mbox, gzip it and you should be fine. You'll have a compressed archive of correspondence, and an easily browsed directory of photographs. For extra points, use the program ``touch'' to date the attachment files to their original received-by date.

Read then purge ... (5, Insightful)

MacTO (1161105) | more than 2 years ago | (#38259410)

There is probably some email that you need to keep, but chances are that you don't need to keep most of your email. So just read, respond, then purge (when appropriate).

As others have pointed out, disk space isn't really a concern this day in age. But managing data that you don't need is a concern. A minute spent filing, backing up, etc. of unnecessary data is a minute wasted. Add enough of those seconds together, and it may amount to a good chunk of your life spent doing more interesting/productive things.

As a side note, I notice that people sometimes get attached to things that don't really matter to them. I've known people who have lost all of their data due to circumstances beyond their control, then they became very distressed about that loss of data. The problem is that only a tiny fraction of that data was actually valuable, but they were worrying about all of the data. In some cases it was so traumatic to them that they spent more time worrying about the irrelevant stuff than the stuff that they would need to continue on in the future. So if you don't keep the irrelevant stuff, you can focus on what is relevant.

Re:Read then purge ... (0)

jones_supa (887896) | more than 2 years ago | (#38259482)

A minute spent filing, backing up, etc. of unnecessary data is a minute wasted. Add enough of those seconds together, and it may amount to a good chunk of your life spent doing more interesting/productive things.

It creeps me how young geeks invest their time in managing their home NAS or even a personal mail server (+domain) while it would be much more simple to just use an USB hard disk and webmail and then use that time for something more creative.

Re:Read then purge ... (3, Insightful)

vadim_t (324782) | more than 2 years ago | (#38259900)

It creeps me how young geeks hand out all their personal data to the first free provider they happen to come across.

Yeah, it's a bit of a pain sometimes, but the benefit of having the data where I want it, dealt with how I want it, outweighs the cost IMO. It also makes for good system administration practice if you have an interest in that kind of thing.

Re:Read then purge ... (0)

Anonymous Coward | more than 2 years ago | (#38260234)

As they get older they will realize that. Until then they are having fun. I used to do the same as them. As I got older. I simplified and went to a big external drive just hanging off some low power computer that is running all the time. It just holds my junk files. 99% of which if I lost them I would not be too mad.

Re:Read then purge ... (0)

Anonymous Coward | more than 2 years ago | (#38260942)

First: I ain't so young...

Second: My "NAS"/email server box is going on 8 years without need of maintenance except to add an account or two, replace and add more hard disks, and do backups. It'll get replaced soon.

Third: Well written tools do the work for you without intervention so you can concentrate on getting your job done without extra labor and worries.

Fourth: No corporation is pawing through my correspondence and files or giving access to them to others. They are private and will stay that way.

Re:Read then purge ... (2)

kava_kicks (727490) | more than 2 years ago | (#38261054)

I don't know that I am that young anymore, but I love my NAS (old PC, 2 x 2TB drives mirrored, Solaris, ZFS). I store all my photos, personal documents and videos of the kids. Everything is mirrored, snapshotted and stored offsite in the cloud for $5 a month (thank you CrashPlan). It takes no time to manage and keeps everything safe. Why would this creep you out?

Re:Read then purge ... (1)

Anonymous Coward | more than 2 years ago | (#38259542)

Good points. However - what is relevant is dependent on time. Your judgement of relevancy is all about what you care about in this point in time. As others have already pointed out - store it all, it's cheap. You really don't know what might be valuable to you later.

Re:Read then purge ... (1)

lucm (889690) | more than 2 years ago | (#38259928)

I used to archive my emails, then one day by mistake they were deleted. For a minute or two I was freaking out, then I felt relieved. I needed to lose them completely to understand that I did not need them. It was like a security blanket (what if I need a cd-key I received by email, or if I want to read again the bad poetry I sent to my ex?), nothing else.

For the last few years not only did I not archive my emails, I also made sure to change my email address once or twice a year to weed out the crap. And there was never a single time where this "lean" policy caused me a problem. It's like losing everything after a fire at home; losing that Dallas VHS box-set or all those National Geographic magazines is refreshing.

In my experience (0)

Anonymous Coward | more than 2 years ago | (#38259414)

I had a practice of burning each year worth of email on one CDR in mbox format, but I have found that I never actually need to refer to those old messages. Also in general there's so much data these days that I've found it best to just archive the cream of the crop.

What's the motivation? (3, Insightful)

Just Brew It! (636086) | more than 2 years ago | (#38259474)

Even at today's post-Thailand-flood inflated hard drive prices, your entire e-mail history occupies less than a dollar's worth of disk space. I fail to see the issue.

mutt, ruby libraries (2)

subreality (157447) | more than 2 years ago | (#38259476)

For my own mail archives I just use mutt and weed things a bit by hand. I find that 90% of the mbox size is in fewer than a dozen attachments, so I can hand-filter those out in ten minutes once a year. Beyond that disk is too cheap to care and time is too valuable to make a really comprehensive solution. So what I do:

'mutt -f archive.mbox'
':set pager_index_lines=6' (Lets you see the message index split above the body)
'o' (Order), 'z' (siZe), End (last entry), Enter (Open).
while(mbox.size > acceptable_size)
{
        'v' (View attachments)
        'jjj' (down a few times to the attachment I want to nuke)
        'd' (Delete)
        while(more attachments) { 'd' (Delete more attachments) }
        'q' (Quit back to the message view)
        'k' (previous message)
}
'q' (Quit back to index)
'$' (Sync changes to disk)
'q' (Quit mutt)

Note the 'j' and 'k' are vi-style up/down. The arrow keys work too if you're not a home row junkie like me.

I don't know a good fully automated way to do this that's ready to slice it right out of the box. If you want to roll your own, just pick up a library like RMail or TMail for Ruby, or equivalent for the language you prefer. That's 80% of the work done but you'll still probably find a dozen corner cases involving oddly-named HTML-alternatives named things that look like binary attachments or terribly malformed spam.

I have all email going back to October 2000. (0)

Anonymous Coward | more than 2 years ago | (#38259516)

With the attachments. Storage is cheap and it takes only seconds to find anything as is. Seriously, if you're concerned about file size and the time to search some simple emails, perhaps your computer is just too old? (Your media attachments shouldn't be adding to the search time, so that is a lousy excuse.)

Re:I have all email going back to October 2000. (1)

pz (113803) | more than 2 years ago | (#38260538)

2000? Hell, I have my email back to the early 1980s.

The real problem is that back then it was OK to put all messages in one file, and having one message per file is far more useful for searching with grep.

Of all the weird suggestions: Eudora MUA (1)

Sipper (462582) | more than 2 years ago | (#38259546)

The Eudora Mail User Agent (i.e. email client) stores attachments in a directory as binaries but yet keeps the text of emails intact. Thus you should be able to import the email into Eudora, then when you export it the attachments should be stripped.

This is also exactly why I don't use Eudora anymore, because attachments get stripped off when exporting the email (or at least that's the way email export or import from/to Eudora worked last I used it).

Now, although this explains one way attachments can be stripped from email, I don't recommend doing that, because it alters the email. Generally I want email intact because otherwise what you're storing might refer to an "attached file", but yet not even knowing what the filename is that the email refers to. Plus it's actually useful to have sent mail attachments intact too, because it means you get to see what version of what file you sent to someone at the time.

There are some other interesting options; the KMail MUA has an option of "delete attachment" when right-clicking on an email attachment, which does delete the attachment but not the reference to it, so you at least know the filename of what used to be attached. I just sent myself a test email and deleted the attachment and then viewed the email raw, but unfortunately Slashdot's filter won't let me send the result. But if you do that yourself and look at it, it should give you an idea how to re-form emails to strip attachments but not the references.

Re:Of all the weird suggestions: Eudora MUA (0)

Anonymous Coward | more than 2 years ago | (#38259872)

Yours is the closest so far, but seriously, nobody has ever written a batch program that takes an mbox file or similar format, pulls all the attachments out as binary, throws them into a folder, and then writes a new mbox file with embedded, links to said files? Yes, I understand the importance of maintaining a link to those files from the relevant message, but does the binary stuff really have to be encoded, bloated and crammed into one gigantic mbox file in order to do that?

Most of the messages so far have been trying to convince the original poster that the space doesn't matter or ask why they'd ever want to do that. There are good reasons. Apparently nobody has written something similar to what they are looking for? Bizarre. If there really is nothing then maybe I'll sit down and do something in perl or some similar hack, but I would have thought someone had already tried something.

What about a program that converts mbox files into an HTML index by date and/or subject with each message as an HTML page with links to the attached files (relative links, of course)?

Storage is cheap, but I don't want mail dating back to the 1990s taking up gigabytes of space when I know that 99% of the space is LOLcat images and other files inefficiently stored and that I probably have elsewhere anyway if they were important. At the very least it would make it easy to sort through the files and decide if they were worth keeping.

Stanford's MUSE project (0)

Anonymous Coward | more than 2 years ago | (#38259582)

I'm not sure if anyone has mentioned this, but Stanford has been working on an email client to help understand and visualize an archive of 50,000 emails - It lets you pull out the images, browse emails by 'sentiment-mapped' values and graph the patterns of activity over the full lifetime of the archive. You can see the project page here: http://mobisocial.stanford.edu/muse/

easy (-1)

Anonymous Coward | more than 2 years ago | (#38259762)

rm -r

will sort it all out

Apple Mail's "Remove Attachments" (0)

Anonymous Coward | more than 2 years ago | (#38259784)

It Mac OS X's built-in mail application, you can use:

Message -> Remove Attachments ... so all you need to do is find a Mac and put your email on an IMAP server.

Delete it! (2)

Lazy Jones (8403) | more than 2 years ago | (#38259960)

Google keeps a permanent copy anyway...

Wait what? (1)

bmo (77928) | more than 2 years ago | (#38259988)

We're worrying about 500MB?

Even at today's outrageous price-fixed (you know it's true) hard drive prices, you're talking 14 cents a GB. For your situation, we're talking 7 cents.

You're complaining about 7 cents worth of storage space. And to cut down on this you want to mangle the archive?

You're tight on space? Buy another drive, burn to CD/DVD.

For those of us who grew up with a Corvus shoebox hard disk costing thousands on the Apple ][ network, this is a ridiculous "ask slashdot" question.

--
BMO

I have all email going back to 1980... (2)

neurocutie (677249) | more than 2 years ago | (#38260020)

back to ARPA mail and UUCP mail days...

for a while I used Eudora and every month religiously took each piece of email and filed it away in suitable mail folders. After Eudora started declining and I got too busy, I stopped that, but even now, religiously every month I clean out my mailbox of all junk and unwanted attachments (trimming 60-100MB to usually 20-30MB) and then stack that months email away as a single mbox file, and start fresh with a new Inbox.

the old mailbox files are on an IMAP server that I can easily read emails from at least 10 years ago -- older with a little more effort. As single mbox files each, I can do greps on them also. Seems to be an okay way to keep the stuff, some of which has proven to be important over the years....

another big help: all semi-junky and non business emails I let Hotmail do the work (vendor stuff, Amazon orders, etc). Have been using Hotmail since before MS bought it. Works well as a place to direct mostly junky vendor stuff.

Procmail (4, Funny)

massysett (910130) | more than 2 years ago | (#38260036)

Google for "procmail remove attachments":
http://osdir.com/ml/mail.procmail/2002-11/msg00091.html [osdir.com]

That will get you started. You can do most anything with Procmail after you figure out the rather odd configuration file format.
Make sure you have it backed up first because it's also quite easy to destroy data with Procmail.
After you spend a lot of time futzing with Procmail scripts and sed and formail and the like, you'll wonder why you didn't go on Amazon or Newegg and buy a $10 flash drive that will hold all your mail several times over.

Re:Procmail (1)

bmo (77928) | more than 2 years ago | (#38260314)

This is the only way to do it... if your time is entirely worthless.

If we measure time in minimum wage, the OP spent more time composing this question and submitting it than if the OP had just spent 7 cents worth of disk space and archived it away.

This is a troll "ask slashdot"

--
BMO

P.S. Where i get my 7 cents from: Go to Newegg. List internal 1TB drives by price. Pick lowest. 140 bux divided by 1000 = 14 cents per Salesman GB. He's using half. 7 cents.

500mb? (2)

nurb432 (527695) | more than 2 years ago | (#38260112)

Amateur. When you get to 8+ gb then we can talk about 'large archive'. Until then, just stick it on a CD.. you don't even need a DVD for that.

when you get older (1)

iggymanz (596061) | more than 2 years ago | (#38260210)

you will be very sorry you deleted those pictures. don't do that. Even right now, you could make many people very happy by giving as gift one of those digital picture frames that display different stored photo every several seconds, with your pictures of those important to recipient.

Some actual ideas to get you started (0)

Anonymous Coward | more than 2 years ago | (#38260348)

Okay, so most of the people here have wasted your time trying to convince you that "storage is cheap" or that there isn't a good reason to store all that e-mail, let alone try to organize it all. I'm with you, not them. It's the fricking 2000s. It should be easier to archive this stuff and organize it if you *want* to.

I've always wanted to do something about my messy mail archive of mbox files (dates back to the 1990s), but I dreaded the thought of coding something up from scratch given all the quirks of e-mail formatting. I had high hopes your post would elicit some sage advice from the readers of /., but so far I don't see much other than the good mutt+ruby solution. In frustration, I've started looking but I haven't found much either. For what it's worth, here's what I've go so far:

1) There are plenty of commercial solutions that promise to do everything for a low price (e.g., MailSteward [mailsteward.com] for OS X looks pretty good and has a free trial up to 15000 messages). Maybe. But I'm cheap and will exhaust the fully free solutions before spending money. Most of them are more focused on mailbox conversion/migration (e.g., Emailchemy [weirdkid.com] ) than actual filtering/archiving.

2) Free / some assembly required:
archivemail [sourceforge.net] - mostly for date-selection of messages and archiving/compressing. Doesn't help with attachments. Python.
archmbox [sourceforge.net] - more capable than archivemail. Can do filtering based on date, header field matches, etc., copy selected messages and compress to archive. Perl. Closer.
MHonArc [mhonarc.org] - converts mbox to HTML files with links to attachments. Meant for mailing list archiving, but it should work the same for a personal mailbox. Perl. There's also an OS X front end [liepins.org] for it.

The HTML approach isn't ideal, but that could be a convenient way to browse through the archives (e.g., toss it all up on a password-protected web site and your mail archive is available anywhere, like your own personal and backed-up GMail), and a contributed program in the MHonArc distribution can turn an MHonArc archive back *into* an mbox file, which might let you do some modifications to the HTML files and linked attachments with scripts and then backconvert them after.

I haven't tested any of these, but I think I'll try MHonArc and see how it goes.

Remove the photos? Really? (1)

enjar (249223) | more than 2 years ago | (#38260412)

Photos are one of the most treasured things in many families. Keep in mind it's highly unlikely Aunt Petunia is keeping great backups of her photos, and when it all goes south, you might be one of the family members who actually has a photo of a relative who has passed on that she wants to print when her hard drive gives up the ghost.

Email archiver (2)

Dupple (1016592) | more than 2 years ago | (#38260422)

Enables you to save everything off line as a pdf. Personally I don't get the question or see the point. My archive is about 6 Gig, all backed up all searchable. Anyway the company that makes the software is www.spotdocuments.com Just back up

Something Like This? (5, Informative)

pscottdv (676889) | more than 2 years ago | (#38260428)

We all think you're crazy, but here it is:

#!/bin/env python
from mailbox import mbox, mboxMessage

orig_mb = mbox(path/ot/orig/mbox)
new_mb = mbox(path/to/new/mbox)

for key,msg in orig_mb.iteritems():
        new_msg = mboxMessage()
        payload = msg.get_payload()
        if msg.is_mulltipart():
                payload = payload[0].get_payload()
        for header in msg.keys():
                new_msg[header] = msg[header]
        new_msg.set_payload(payload)
        new_mb.add(new_msg)
new_mb.flush()

Seriously? (-1)

Anonymous Coward | more than 2 years ago | (#38260600)

You sound like an antisocial hoarding weirdo. You don't need those emails, any more than you need an audio recording of your entire life. Life is too short to live twice, you won't use 90% of that shit, and the photographs you want to delete to make more room for your obsessive hoarding of data are the only things that are going to matter decades from now. You freak.

Email cleanup? (0)

Anonymous Coward | more than 2 years ago | (#38260754)

Ctrl-A, Shift-Del

The solution may not be a technical one (0)

Anonymous Coward | more than 2 years ago | (#38261252)

I happen to be in the process of reading a book on hoarding to help my mother through some moderate hoarding issues that she is having; the problem described in this post sounds exactly what some of the underlying causes are for hoarding in general.

If you keep looking for a purely technical solution to the problem, you're probably not ever going to solve it, and it will keep escalating despite whatever technical stop-gap you're able to come up with.

Better compression? (1)

tomtomtom (580791) | more than 2 years ago | (#38261292)

As others have said, the headache you will have if you do want to come back (potentially years later) to that one email you know you had only to find your attachment-stripping program has foobar'd the whole archive up (or that you need the attachment after all) probably isn't worth the hassle for saving 500MB per year this year (even taking into account reasonable growth rates - I'd note that bandwidth per $, which will be the factor limiting your email size, has been growing rather more slowly than storage capacity per $ over the past decade and things are likely to continue that way).

If the problem is that you have significant duplication between emails (e.g. the same attachment being emailed several times), gzip and bzip2 may well miss the opportunity to de-dupe this because the distance between duplicated sections is large. One solution to consider if this is an issue may be to use something which is better at compressing over long distances. I would suggest trying something like lrzip [kolivas.org] to compress tarballs of the annual sets of mbox files before archiving those.

Of course, if you just have lots of attachments which *aren't* duplicated (which is probably more likely), that won't really help much.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?