Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Ma.gnolia User Data Is Gone For Good

kdawson posted more than 4 years ago | from the lost-in-the-clouds dept.

Social Networks 450

miller60 writes "The social bookmarking service Ma.gnolia reports that all its user data was irretrievably lost in the Jan. 30 database crash that knocked the service offline. Ma.gnolia founder Larry Halff recently discussed the crash and the lessons to be learned from Ma.gnolia's experience. A lesson for users: don't assume online services have lots of staff and servers, and always keep backup copies of your data. Ma.gnolia was a one-man operation running on two Mac OS X servers and four Mac minis."

cancel ×

450 comments

Sorry! There are no comments related to the filter you selected.

Mac reliability (5, Funny)

Anonymous Coward | more than 4 years ago | (#26930981)

Crashing Macs? That's unpossible!

Re:Mac reliability (0)

Anonymous Coward | more than 4 years ago | (#26931091)

Crashing Macs? That's unpossible!

Shoulda used Micro$haft.

Re:Mac reliability (4, Insightful)

jetsci (1470207) | more than 4 years ago | (#26931137)

So umm...I have a confession...

I had no idea anyone actually used Mac's as servers. Sure, I bet you can get apache running or something but I didn't realize anyone had. Therefore, this is my first bit of exposure to this idea of Macs as servers and its all negative!

Woe is me.

Re:Mac reliability (-1, Troll)

Anonymous Coward | more than 4 years ago | (#26931351)

Using Macs is bad enough...but four Mac minis?! The money spent on cute little toys could have instead bought much beefier x86 hardware running a *nix.

Yet another example of MacFag "Oooh, shiny" idiocy. He probably own a Zune and plays Pokemon, too.

Re:Mac reliability (1, Troll)

cspaz (799390) | more than 4 years ago | (#26931375)

Idiot. Yeah he's a "MacFag" and owns a ZUNE. Did you even think before you hit submit? Of course you didn't...

Re:Mac reliability (0)

Anonymous Coward | more than 4 years ago | (#26931441)

iPods are no longer cool, everyone has them.

Re:Mac reliability (3, Funny)

Anonymous Coward | more than 4 years ago | (#26931531)

annd zunes were never cool. and no one has them...

Re:Mac reliability (-1, Offtopic)

icannotthinkofaname (1480543) | more than 4 years ago | (#26931439)

Hey! I play Pokemon, you insensitive clod!

No, really, I do. *looks up Pokedex on Diamond card* On my Pokemon Diamond card, which I've had since release, I have 387 obtained and 407 owned. The save file is going on 150 hours of gameplay time.

However, I do not own a Zune or any Apple products, so we're all good on those two points.

Re:Mac reliability (5, Insightful)

beelsebob (529313) | more than 4 years ago | (#26931477)

Yes, because they don't come with apache and php pre-installed, only a ticky box away from running.

Seriously, do people still not realise that OS X is just UNIX with a pretty UI?

Re:Mac reliability (5, Funny)

Idiomatick (976696) | more than 4 years ago | (#26931563)

In the same sense that horses and monkeys are JUST mammals. Doesn't mean that they share THAT much in common...

Re:Mac reliability (2, Informative)

the_humeister (922869) | more than 4 years ago | (#26931683)

No, but Mac OS 10.5.x can properly be called Unix [opengroup.org] , but only the Intel version, not the PPC version.

Re:Mac reliability (-1, Troll)

Anonymous Coward | more than 4 years ago | (#26931655)

It's just UNIX for morons. Again, why are we surprised that they got fucked up?

Re:Mac reliability (1, Insightful)

Anonymous Coward | more than 4 years ago | (#26931691)

And here I thought OS X was just BSD with the Mach kernel and some fancy API.

Re:Mac reliability (1)

SSCGWLB (956147) | more than 4 years ago | (#26931711)

Well, kinda.

Unix that you pay for, with limitations imposed by apple, running on mac hardware.

Thanks for trying apple.

Re:Mac reliability (4, Interesting)

FishWithAHammer (957772) | more than 4 years ago | (#26931925)

Are any of the free BSDs or Linux variants certified Unixes?

(Honest question, I don't know.)

Re:Mac reliability (0)

Anonymous Coward | more than 4 years ago | (#26931825)

Seriously, do people still not realise that OS X is just UNIX with a sluggish UI?

there....and such.

Re:Mac reliability (2, Interesting)

Tibor the Hun (143056) | more than 4 years ago | (#26931721)

It's hard to know whether you're trolling or not.
There are OS X servers out there and they perform rather well. I know because I admin 50 of them, and have met hundreds of others who administer them in school systems across the state.

You may also be familiar with iTunes, or Apple's movie trailer website. I'm sure a large part of those are Xserves and Raids.
I'm not saying they are maintenance free, but they are out there.

Furthermore, a few years back there was a rather large beowulf cluster of mac towers that scored quite well on top500.org.

OS X is unix, and claiming ignorance about unix on a site like this is well...

Re:Mac reliability (4, Insightful)

SatanicPuppy (611928) | more than 4 years ago | (#26931817)

Mac servers are pretty. They do okay, they have nice swanky data enclosures, and the form factor is roughly the same as anyone elses.

It's just whether or not you want to use OS X. I disagree that OS X is "just unix," however. It's not even "just linux" or "just bsd". OS X has it's own warts, and while it may be stable and friendly, I'd rather have a real *nix running on less pretty hardware.

The best use I've ever had for the big Mac servers is running as a file server in a windows/mac environment. If you still have any pre-OS X machines around, that's about the only way to get them all on the same machine (If you say windows mac volume, I'm mailing a dead fish to your house).

Otherwise, you know, you can install apache, whatever, but it's not any different from using a regular linux server in terms of increased functionality, and there are some significant OS update issues that can cause problems. Mac updates are of the all or nothing school, and they WILL break stuff, so you need to be careful.

Re:Mac reliability (1)

briggsl (1475399) | more than 4 years ago | (#26931659)

the first thing I did was chuckle, mac OS must be the most unreliable OS ever

Re:Mac reliability (4, Funny)

neokushan (932374) | more than 4 years ago | (#26931719)

I want to say something here, but I get the feeling that no matter what I say, I just wont be herd.

In other news (5, Funny)

jetsci (1470207) | more than 4 years ago | (#26930987)

Facebook was recently brought down when their hamster keeled over and ceased powering their Amiga.

Re:In other news (5, Funny)

elrous0 (869638) | more than 4 years ago | (#26931203)

In that case, the death of Mr. Furry was a much greater loss than the data.

Re:In other news (2, Funny)

jetsci (1470207) | more than 4 years ago | (#26931325)

Flowers have been sent to Mrs. Furry.

Re:In other news (0)

Anonymous Coward | more than 4 years ago | (#26931677)

And in a neat bit of coincidence, Senora Furry esta mi amiga!

Re:In other news (5, Funny)

palegray.net (1195047) | more than 4 years ago | (#26931753)

I think it's a tragedy that the first thing that came to mind reading your post involved a blood-spattered "furry" (as in a dude wearing some fluffy costume). The Internet has done terrible things to my mind.

Re:In other news (0)

Anonymous Coward | more than 4 years ago | (#26931405)

And nothing of value was lost.

Re:In other news (1)

Spazztastic (814296) | more than 4 years ago | (#26931491)

And nothing of value was lost.

But now who will know of my status updates, you insensitive clod!

Re:In other news (1)

jetsci (1470207) | more than 4 years ago | (#26931517)

Whats wrong with what I've been doing? The GPS chip behind your ear does a great job. And only $14.99!

Re:In other news (0)

Anonymous Coward | more than 4 years ago | (#26931739)

That was an unfounded and baseless rumor spread by Atari ST losers. I've proven that my stock A500 is still far superior to Vista.

Your bookmarks belong.... (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#26931003)

ba ha ha ha....all your bookmarks are belong to us...until we put them into the bit bucket and then they are belong to no one!!!!

Food for Stallman (2, Insightful)

Rinisari (521266) | more than 4 years ago | (#26931051)

This bad news is delicious food for Stallman's argument against "cloud" services.

Re:Food for Stallman (5, Insightful)

ZeroPly (881915) | more than 4 years ago | (#26931127)

Stallman's argument is more that cloud services are almost always non-open. He does not have a per se objection to cloud services - and if you were to reveal all your source code and protocols, I doubt it would be objectionable to him.

Of course it's impossible to free cloud services in the sense of modification and distribution, but if the source is open you have the chance to make your own.

Re:Food for Stallman (1, Insightful)

Anonymous Coward | more than 4 years ago | (#26931367)

In response to your sig... Never let an officer see you swallow something that he may reasonably believe to be evidence. Really. You will regret it. Swallowed != lost.

Re:Food for Stallman (1)

orasio (188021) | more than 4 years ago | (#26931561)

In fact Stallman is against software as a service, he said so when he was in my city.
Even if he didn't say it, it goes against the FSF phylosophy. The whole point is giving freedom to the user. If someone else has your data, you lose some freedom to do what you want with your data.

Re:Food for Fault-Tolerance (2, Insightful)

RobBebop (947356) | more than 4 years ago | (#26931227)

It's food for any argument against any web service that doesn't publish it's reliability information or publicize the data for what types of mechanisms it has in place in case of disasters like a corrupt database, fried motherboard, or busted hard drive.

There's a design methodology that's used by NASA for manned missions: Any individual component should be able to fail without compromising the mission. Of course, in the last few decades we've seen 2 out of 5 Shuttles go ka-boom! so obviously this NASA guideline isn't enough and it's *REALLY* hard to prevent failure when a perfect storm of multiple systems experience failure at the same time.

So if anything, I'd say this is an argument that supports robust, reliable, fault-tolerant design rather than just kludging a half dozen systems together and calling it a "web service".

Re:Food for Stallman (0)

Keruo (771880) | more than 4 years ago | (#26931259)

Just exactly what is his argument here?
"by GNU/Linux, he could have built redundant system and even saved some money"?

Cloud services are broken by concept.
They offer virtually redundant high availability system but the reliability is also virtual.
Cloud computing model works only for data which allows certain percent of data to be lost at anytime.
 

Re:Food for Stallman (1)

CannonballHead (842625) | more than 4 years ago | (#26931469)

Just wanted to let you know that "unleash the fyoorie" makes me laugh every time. I'm not sure why.

Re:Food for Stallman (1)

interglossa (1110251) | more than 4 years ago | (#26931887)

I think Richard's argument is more in line with his discussion of ownership of information rather than the fragility of the cloud per se. Cloud angst has been rather building broadly lately for general reasons with such tremors as google abandoning services like notebook and yahoo pulling the plug on Briefcase. There are also some interesting talks on TED about this topic. We are not very far in coming to grips with it, perhaps economic collapse will help with that.

Needless loss (5, Insightful)

hattig (47930) | more than 4 years ago | (#26931073)

Argh, why not just add a backup or replication database on one of the spare Mac Minis?

That way you would have needed a complete server farm disaster to mess things up irretrievably.

Re:Needless loss (1)

macbuzz01 (1074795) | more than 4 years ago | (#26931171)

Replication != Backup
A backup is good, but worthless if not verified.

Re:Needless loss (1)

palegray.net (1195047) | more than 4 years ago | (#26931763)

Replication in place of backup is exactly what got this guy screwed in the first place.

Re:Needless loss (4, Insightful)

qoncept (599709) | more than 4 years ago | (#26931209)

Or back it up, like, once a day, or week, or ever, to a flash drive or something. That's a lesson that's already been learned, and it's common sense. I'm terrible about backing up my own data (anything I've lost and recovered is usually something that just happened to be on a remote web server somewhere, coincidentally, because it was always intended to be on the web). But all of my websites, with other users' data, are backed up. It doesn't take a very complex scheme or much thought. A cron job to dump your database and tar your web structure and then copy it to a different location.

I definitely have my doubts that someone who could make this mistake is all that capable "lessons learned."

Re:Needless loss (2, Insightful)

metamatic (202216) | more than 4 years ago | (#26931765)

Except cron+tar isn't sufficient. You need versioning. Otherwise if your database is corrupted and you don't notice immediately, your backup gets corrupted automatically.

I back up my web sites using cron + rsync + rdiff-backup.

Re:Needless loss (1)

kliklik (322798) | more than 4 years ago | (#26931923)

I've recently discovered automysqlbackup.sh [sourceforge.net] . It took 4 minutes to set up and now it happily backs up all my databases to the local disk and emails them to me. It's brilliant!

Re:Needless loss (1)

NeverVotedBush (1041088) | more than 4 years ago | (#26931221)

I would bet the admin realizes this all too well now.

I can't believe anyone would run a commercial system without backing things up. Hell, even home users, if they have anything of value, need to do backups.

It's just not that hard these days especially with cheap NAS boxes, low-cost hard drives, etc.

Re:Needless loss (1)

0100010001010011 (652467) | more than 4 years ago | (#26931289)

Why is it that I feel the stuff I have setup at home is more robust than some "professional" shops? Is the world more like The Daily WTF [thedailywtf.com] than I've been lead to believe?

I'm not saying my system is perfect, but it's redundant in at least two locations.

Laptop<->Server<->Server HD 2<->Dreamhost.

My MySQL databases which just keep stuff like weather and temp (from my 1-Wire system) is dumped nightly and sent to my Gmail account. (It's also not a few TB server...) but seriously. How hard is it to toss in a few extra hard drives and do a rolling backup?

Re:Needless loss (5, Insightful)

eln (21727) | more than 4 years ago | (#26931347)

A simple periodic dump to an external hard drive would have at least been something. I know that small-time operations shouldn't be expected to have robust backup schemes, but if your primary purpose is to store other people's data, the FIRST thing on your mind should be how to back it up. Once you lose someone's data, they'll never use anything you put out again, and they'll tell all their friends not to either.

Re:Needless loss (2, Insightful)

Anonymous Coward | more than 4 years ago | (#26931539)

Small time operations should be expected to back up just the same as large ones. I wrote simple routines to backup my db nightly, compress it, upload it and on the receiving end decompress it and restore it. If any step fails it emails me. I check it manually every week and save backups for 2 years (quite a bit of data but for legal reasons it's necessary).

The whole setup took me maybe a day to get working. There is NO excuse for not having backups.

I lost one of my primary servers on a sunday at around 5pm. Had everything working by the next morning on my backup servers. Never was I so happy that I had good backup and implementation/restoration plans.

Re:Needless loss (5, Insightful)

Ephemeriis (315124) | more than 4 years ago | (#26931437)

Argh, why not just add a backup or replication database on one of the spare Mac Minis?

That way you would have needed a complete server farm disaster to mess things up irretrievably.

Replication gives you redundancy, much like RAID does. It lets you survive a hardware failure or two. It is not a backup. If the building burns down, or a tree falls on your server room, or lightning fries everything you are still screwed.

What they needed was a backup. A tape, or removable HDD, or a flash drive, or a CD, or something that can be taken out of the building on a regular basis. Once a day, once a week, once a month... Whatever.

Then, no matter what happens to your live hardware, you've got a backup you can restore from. Buy some new hardware, throw your backup at it, done!

And nothing of value was lost. (1, Funny)

Anonymous Coward | more than 4 years ago | (#26931107)

Couldn't resist.

Who the hell is Ma.gnolia (5, Insightful)

Jailbrekr (73837) | more than 4 years ago | (#26931111)

And how can they be slashdot worthy when they are a social networking site with ONLY a half a terabyte of data? In short, who cares?

Re:Who the hell is Ma.gnolia (-1, Troll)

dotancohen (1015143) | more than 4 years ago | (#26931749)

And how can they be slashdot worthy when they are a social networking site with ONLY a half a terabyte of data? In short, who cares?

Because they have an Alexa rating higher than Slashdot's?

Actually, they don't, but I would not be surprised if /. has no more than a terabyte of data. It's all text, no images. That's what lets it stand up to the /. effect on itself, and why it's the only site that didn't crash like a hijacked 767 on 11-9-2001. Size is not all that matters, even in social networking pissing contests.

Re:Who the hell is Ma.gnolia (0)

Anonymous Coward | more than 4 years ago | (#26931801)

and why it's the only site that didn't crash like a hijacked 767 on 11-9-2001.

You're not from around here, are ya? And that thar beard looks a mite suspicious...

lesson #1 (4, Interesting)

petes_PoV (912422) | more than 4 years ago | (#26931131)

on the 'net you can't tell the major corporation from the kid in a garage

lesson #2, trust no-one with your data

lesson #3 disaster recovery capability only exists after it's been tested

lesson #4 backups are useless unless you can prove you can recover from them

Re:lesson #1 (2, Funny)

keytohwy (975131) | more than 4 years ago | (#26931521)

I thought lesson #1 was "don't get high on your on supply"

Re:lesson #1 (1)

Ephemeriis (315124) | more than 4 years ago | (#26931543)

lesson #1 on the 'net you can't tell the major corporation from the kid in a garage

lesson #2, trust no-one with your data

1 and 2 don't really matter if you've got a backup. Who cares if it's some kid in a garage if you've got a backup? If it's more convenient to have your data on some kind of web service, go for it! But make sure you've got a backup.

lesson #3 disaster recovery capability only exists after it's been tested

lesson #4 backups are useless unless you can prove you can recover from them

This is really where things fall apart over and over again. I've seen tons of clients with no backup at all... Or a backup that they've never tested and they just assume it's working correctly.

It isn't a backup unless you can take it off-site, and it isn't a backup unless you know you can get your business back up and running with it.

Not the platform's fault... (2, Interesting)

wandazulu (265281) | more than 4 years ago | (#26931139)

Good backup strategies are critical to any operation, regardless of platform. I've seen similar things happen with MSSQL server databases as well as Oracle running on the most powerful Sun box you can get (circa 2001).

One database backup strategy I've seen used rather successfully is doing a straight SQL dump every night and then copying the sql file over to somewhere else; even if the database became hopelessly corrupted there's still a way to re-import everything.

Of course, this is in *addition* to mirroring, tape backups, etc.

Re:Not the platform's fault... (1)

BigJClark (1226554) | more than 4 years ago | (#26931371)


Agreed. We have a mirror that we do weekly EXPorts from, as not to slow production environment. On prod have a second safety net of RMAN, but I've never trusted it. I've taken all the bloody courses, it just seems too failure prone. Heaven forbid you open your database with reset logs. It mucks the SCN up, or something irrevocably small. I'm still not confident about changing Oracle versions and have backwards compatibility.

In short, yeah, exp is tried and tested for recovery.

Re:Not the platform's fault... (1)

grassy_knoll (412409) | more than 4 years ago | (#26931545)

Opening oracle with resetlogs resets the online redo logs and sets the log sequence number to 1; also called creating a new incarnation of the database.

That prevents applying archive logs from before the reset (i.e. previous incarnations ) but which may contain more recent data than what's in the datafiles.

I use EXP/IMP myself, but for larger databases it can be impractical. One of my systems takes around 120 hours of processing time to read in an export and write it to a blank schema ( which we tested when building mirrored servers for development ).

Still, having an export is much better than a blank stare when someone tells you the backup tape is unreadable.

Re:Not the platform's fault... (1)

BigJClark (1226554) | more than 4 years ago | (#26931787)


This I know. But, heaven forbid you open your database like that, and you are screwed. I believe this oversight has been correct in Oracle 11g, but I'm not entirely sure, as I'm trained in 10g. We tend to keep our OLTP databases relatively clean via purge processes, and offload required data to our OLAP. As we have to maintain 7 years for litigation purposes on tape backup, having the EXP is basically yeah, the only thing I trust. It has never taken over 24hrs to perform an EXP, and this meets our daily backup requirement. It hasn't happened yet, but if we ever had to recall data from 7 years ago, I would re-install the old oracle version, IMP the data and present. I'm not entirely sure I can do this with RMAN.

Re:Not the platform's fault... (1)

BSAtHome (455370) | more than 4 years ago | (#26931725)

Yes, but straight dumps take time, a lot of time. They need to be consistent to be useful and then you have to hold table/db locks which can interfere with the operation. Even if you can dump it without locks, 500GByte of data over a GBit link takes al least 1.2 hours. And that assumes that you can get the data that fast, let alone transport it. Mysql is slow at doing dumps on innodb (myisam can be copied rather easily).

When databases and tables get large, things start to suck big time if you want real backups. Probably the most effective is doing snapshots of the filesystem while you hold a lock in the database and flush it hard. Then you can copy the snapshot over all the time you need.

Hilarious time-line... (0)

Anonymous Coward | more than 4 years ago | (#26931141)

Friday, February 20, 2009, 10:01 AM PST:

WE'RE FUCKED!! WE'RE FUCKED!! GAME OVER MAN, GAME OVER!!!

Tuesday, February 17, 2009, 10:50 AM PST:

Oh. Shit. Unfortunately, database file recovery has been unsuccessful and I won't be able to recover members' bookmarks from the Ma.gnolia database. This means that the public bookmark recovery tools are the only source for recovering your bookmark collections.

If you are interested in hearing more about what happened, the history of Ma.gnolia in general, and future prospects, you can watch the latest Citizen Garden podcast below, which was recorded last week. As I mention in this podcast, I am working on relaunching Ma.gnolia as a private service on a more robust infrastructure in the coming months. I'll update this page and the twitter account with those and any other developments.

Friday, February 13, 2009, 7:00 PM PST:

The data recovery folks let me know that they're still working, but I should hear more from them by Tuesday. Everything's cool, no worries, man!

Monday, February 9, 2009, 3:10 PM PST:

Just posted a new bookmark recovery tool for our members who used the scheduled blog posting system. Relax...you'll get your stinkin' data back. Whiners.

Sunday, February 8, 2009, 9:50 PM PST:

There are now more than twice the number of bookmarks in the Web Cache recovery tool than there were on Friday. I can't guarantee results, but members may want to try again.

Friday, February 6, 2009, 6:00 PM PST:

I've been able to extract a set of public bookmarks from a cached copy of the Ma.gnolia web site, and these are now available via the Web Cache recovery tool. I'm working on expanding the size of this cache and the number of bookmarks I'll be able to provide and will update this page, the bookmark recovery tips thread at Get Satisfaction, and Twitter when I do so.

Additionally, I'm working with data recovery specialists to recover the Ma.gnolia data store; but, unfortunately I won't hear from them until next week. As with the other data recovery efforts, I'll keep you posted as to their progress I as get any updates.

Dear Ma.gnolia Members and Visitors,

So far, my efforts to recover Ma.gnolia's data store have been unsuccessful. While I'm continuing to work at it, both from the data store and other sources on the web, I don't want to raise expectations about our prospects. While certainly unanticipated, I do take responsibility and apologize for this widespread loss of data.

In this past year, many of us have seen much loss around us. While bookmarks seem small on the national or global scale, I know that many of you had built intellectual and social capital through the bookmarks, groups, and connections you made here. For those who had shown their support for Ma.gnolia by buying one or more premium feature subscriptions, that's one thing you won't be losing: refunds will be issued for those purchases within two weeks from today.

Ma.gnolia was approaching the third anniversary of its public launch; for me, it was the project and people to which I'd devoted most of my time, energy, and love for nearly four years. It's still a little too soon to give word about the return of Ma.gnolia the service and the future of the M2 project, but I will keep this site and our Twitter account updated as those decisions are made.

In the meantime, I can provide a few pointers to some resources that can help:

If you've been publishing your bookmarks through any RSS feeds or aggregation services like FriendFeed, you can re-capture some of them before those feeds expire.

We've set up a recovery tools page with several options. We're still adding more.

If you're looking for a place to start a new collection, I think Diigo is a good option to check out for its groups, cross-service posting features and attentive staff.

Further tips for recovering bookmarks can be found or posted in a thread at Ma.gnolia's page on Get Satisfaction.
Sincerely, Larry

Re:Hilarious time-line... (0)

Anonymous Coward | more than 4 years ago | (#26931647)

Would it have killed you to put that in chronological order?

How not to do it. (1)

Seth Kriticos (1227934) | more than 4 years ago | (#26931145)

Well, at least they now know how not to do it properly. The probability that it will happen again is quite small, especially because no one will trust them anymore in the first place and it will be really hard to start anew. Their best bet is probably to start a new service under another name and another look.

Lesson? (4, Insightful)

The Moof (859402) | more than 4 years ago | (#26931153)

discussed the crash and the lessons to be learned

Lessons such as "Regularly monitor and maintain backups like and business should?"

Re:Lesson? (0)

Splab (574204) | more than 4 years ago | (#26931261)

Thats all fine and dandy, except, where I work someone has to prioritize our assignments, one of those is getting the backup scripts up and working for nightly backups, but so far that job has been prioritized way below adding new fizzbang to the interface.

Currently I'm just grabbing a new snapshot of the database once in a while.

(On the positive side we know we can recover our backups, that at least is verified)

Re:Lesson? (1)

Ephemeriis (315124) | more than 4 years ago | (#26931581)

Thats all fine and dandy, except, where I work someone has to prioritize our assignments, one of those is getting the backup scripts up and working for nightly backups, but so far that job has been prioritized way below adding new fizzbang to the interface.

Currently I'm just grabbing a new snapshot of the database once in a while.

(On the positive side we know we can recover our backups, that at least is verified)

I see this all too often with our clients... Backups are considered a low priority because everything is up and running right now. Nobody wants to waste time/money on something that isn't necessary right now.

Then they have some kind of horrible software crash that eats all their data... Or the hardware goes up in a puff of smoke... Or the building floods... And all of a sudden they're scrambling to get things back up and running.

Usually after a disaster like that they'll take backups more seriously... If they're still in business.

Re:Lesson? (0)

Anonymous Coward | more than 4 years ago | (#26931393)

RAID much? Bah. mac server.

Re:Lesson? (1)

sakdoctor (1087155) | more than 4 years ago | (#26931415)

Some other web 2 site died a month or two ago.

The story was on /. but I don't remember the services name. Turned out the guy had a single copy on a RAID array which got wiped, game over.

Lesson still not learned apparently.

Re:Lesson? (1)

rudy_wayne (414635) | more than 4 years ago | (#26931645)

"Ma.gnolia was a one-man operation running on two Mac OS X servers and four Mac minis"

So what? Hard drives are cheap. Buy a couple and make backups.

Re:Lesson? (5, Interesting)

Knowbuddy (21314) | more than 4 years ago | (#26931895)

Lessons such as "Regularly monitor and maintain backups like [any] business should?"

I love it when people say things like this. It shows me that they've never actually had to set up an enterprise backup strategy. I'm certainly not defending the Ma.gnolia guys, but I also can't stand it when people are on a shakier soapbox than they realize.

I'm sorry, but when you are used to the whiz-bang-pretty of Web2.0, the state of enterprise-level backups is horrifically archaic and dismal. And, btw, given the size of today's hard drives and databases, for pretty much all intents and purposes "Enterprise" == "More than one computer with more than just a few files on a drive".

Compare and contrast: a 1 TB hard drive will run you roughly $100. Do you know how much it then costs to backup that TB?

  • LTO-4 tapes, 800GB each, $50-$150 each tape plus roughly $2500 for the drive. Figure 2 tapes/day * 10 days backups = 20 tapes * $100 = $2000 in tapes alone. Congrats, that 1 TB just cost you $4500 in enterprise backups ... not to mention the time involved each day in doing a backup. You might save yourself some time and money by doing incrementals ... but then you have to balance that risk with complexity of backups and difficulty in restores.

  • NAS is trickier. The cheap NAS solutions, sub $1000 such as Buffalo and LaCie, aren't going to get you much more than a TB or two. And at that point, are you really any better off than the RAID solution? Maybe, maybe not. As you start to scale into IBM or Dell solutions, you are almost immediately beyond a $2500 price point before you even get to hard drives. Oh, and don't forget the cost of a gigabit switch so that it doesn't take you days to do a single backup.

  • iSCSI? Seriously? Not an option for SOHO businesses.

Then there's backup software to contend with. It's not just as simple as "go buy a copy of BackupExec" -- there's different licensing for databases, and network backups, and whatever arcane rules they want today. I'm a PC guy so I can't talk much about Enterprise-level Mac backup solutions, but I can without a doubt say that Time Machine is not one of them.

It's even more dismal when it comes to Open Source solutions. Have you ever actually tried to setup Bacula? It may be the 600lb gorilla of OS backup solutions, but it's still a royal pain. And to the "just set up a cron job for rsync" guys, c'mon, really? Good luck with that.

So, please, let's dispense with the thought that backups are easy. Backups really suck. Hard. That's why so many people want to think of RAID as a backup solution -- because the step from one hard drive to two or three is easy, but the next step is much farther away than you think.

Two Mac OS X servers and four Mac minis (0)

Anonymous Coward | more than 4 years ago | (#26931177)

Apparently they were waiting on the port of the drive imaging software.

Lessons Learned?? (4, Insightful)

ACK!! (10229) | more than 4 years ago | (#26931183)

Like frickin' having a backup? Isn't that one of the first things you ever learn if your business relies on computers + userdata?

Slashdot is backed up, right? (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#26931219)

kdawson and cmdrtaco-

This is the part where you remind us that Slashdot is robust, redundant, and backed up.

Macs (2, Funny)

Anonymous Coward | more than 4 years ago | (#26931225)

You shouldn't use shiny plastic ornaments for serious business.

Re:Macs (1)

dotancohen (1015143) | more than 4 years ago | (#26931707)

You shouldn't use shiny plastic ornaments for serious business.

Or fruits.

Re:Macs (0)

Anonymous Coward | more than 4 years ago | (#26931883)

Cause:
~It's business.
It's business time.
You know when I'm down to just my socks it's time for business that's why they call it business socks.
It's business.
It's business time.~

Business Time by Flight of the Conchords

Ma-who? (1, Insightful)

Anonymous Coward | more than 4 years ago | (#26931321)

One man operation which doesn't make backups, sounds fairly typical to me, remind me again why this is Slashdot worthy?

Time Machine Anyone? (2, Interesting)

thittesd0375 (1111917) | more than 4 years ago | (#26931385)

My Mac servers run snapshots to external drives every hour. When something goes badly, it's back up in a few minutes. Not sure why that wouldn't have been done here.

Re:Time Machine Anyone? (1)

jrothwell97 (968062) | more than 4 years ago | (#26931755)

Presumably because the database was stored in a single .sql file, mirrored by each server, Time Machine wouldn't be particularly effective, because it would copy across a (new) copy of the massive database every time.

Time Machine is excellent for backing up lots of little files (on a home PC, say, or a web server's /httpd) but for backing up big files, it's very inefficient. Additionally, Time Machine wasn't included with OS X 10.4 (both distributions), so if it wasn't running Leopard or Snow Leopard, you'd be stuck with configuring cron to run rsync manually (although why this wasn't done I don't know.

Re:Time Machine Anyone? (1)

drinkypoo (153816) | more than 4 years ago | (#26931901)

flyback [flyback-project.org] - I bet you could get it to work on OSX :D

What's with the OS X users losing data? (3, Insightful)

PrimeWaveZ (513534) | more than 4 years ago | (#26931403)

I mean, just because a few medium-profile sites running on Macs have experienced a failure causing data loss doesn't make them unique. Every OS and every type of hardware will, at some point, experience a failure. It's the PEOPLE that make the failure a problem, and it sure looks like this tard was a problem.

Who the hell doesn't back up their data? Seriously? This is "Slashdot worthy" because some hapless Mac user lost their data. BOO FUCKING FAIL. Move on.

Boo hoo (0)

Anonymous Coward | more than 4 years ago | (#26931417)

Honestly, how sad can I get that some people lost their social bookmarks. Start over. All us geeks are perfectly happy to reformat our hard drives and reinstall our OS every 9 months or so... so quit crying over your beloved social bookmarks! They're for girly-men anyhow.

"Private relaunch?" (4, Interesting)

mr_mischief (456295) | more than 4 years ago | (#26931433)

"Gee, Bob, we have the proof that this thing works. Why don't we sell it already?"

"Well, Bill, nobody wants to buy it and grandfather in all the whining freeloaders and their data."

"It's too bad we can't just drop all the data and start fresh."

"Well, why not, Bill? All we have to do is say it's been lost and can't be recovered. We can tell the buyer what's actually happening so they don't think we're total IT rejects who couldn't figure out a data retention policy."

"That's why I like working with you, Bob. You always have a way around the problem."

Have fun with it. The names have been changed (one changed anyway and one added), well, because it probably has nothing to do with reality. It sure is fun to ponder, though.

Finally! A privacy solution! (5, Funny)

Captain Spam (66120) | more than 4 years ago | (#26931465)

All right, let me get this straight: First you people bitch and moan when Facebook says they'll save user data forever. NOW you people bitch and moan when this site loses user data forever! You're never happy, are you?!?

Hardware RAID isn't magic, mirrors aren't archives (3, Insightful)

argent (18001) | more than 4 years ago | (#26931573)

LH: "The server was RAID. Its disk was RAID, so that's one of the things we're looking at. But it was a software RAID, so if it's a filesystem problem then... that's not gonna do any good because the the errors were RAIDed as well."

Since the file system and database were corrupted, it wouldn't matter if it was hardware RAID or software RAID. That's not the problem at all, the problem is there was no archival backup, and their only backup was a file sync... that replicated the database errors on the backup.

To backup a database, you dump it in a serialized form, or maintain a serialized form of the data in parallel with the database.

Free service (2, Insightful)

tsstahl (812393) | more than 4 years ago | (#26931585)

And the users got what they paid for.

Simple as that.

The flip side is that this guy's service will probably be the MOST reliable going forward.

Of course he should have had reliable backups; now he is the poster child for backups. Remember, nobody pays you for backups, only for restores.

Lesso learned (1)

Todd Knarr (15451) | more than 4 years ago | (#26931587)

Users: if you're trusting your data to someone else, you need to insure one of two things. Either you need a signed, iron-clad written contract guaranteeing service with nasty penalty clauses requiring the service to compensate you fully for all the costs of data loss (and sufficient insurance and/or confidence that the service has the wherewithall to pay those penalties and not just flee into bankruptcy leaving you holding the bag anyway), or you need a backup of your data under your own control and in a form where you can upload it to another service if you need to change services. If you don't have one or the other, you will end up being caught in something like this. The only question is when it'll happen. And I can pretty much tell you that no service you can afford will give you the first option. So you'd better evaluate the cost of backing up your data yourself and the costs of losing it and decide which one you're going to pay.

Couldn't they just do a restore with Time Machine? (1)

Dopeskills (636230) | more than 4 years ago | (#26931591)

Time for ZFS

0,5 TB = 500 GB (1)

Max_W (812974) | more than 4 years ago | (#26931619)

He kept all on one hard disk? Even I know that it is wrong. I presented my spouse a PC on her birthday with the hard disk of 500 GB, I mean it s not that hard to back up 500 GB nowadays.

Backup Testing? (2, Insightful)

skyriser2 (179031) | more than 4 years ago | (#26931621)

Ouch... Isn't part of a backup strategy to sometimes attempt a recovery from a backup, on a test system?

Who needs to back up? (1, Funny)

Anonymous Coward | more than 4 years ago | (#26931663)

I just assume that the FBI or NSA have a back door into my server and are making copies of everything for me.

Social Bookmarking? (1)

whisper_jeff (680366) | more than 4 years ago | (#26931693)

What the hell is a "social bookmarking service"? Since the site is dead, going to their webpage didn't help clear that up at all. Is it seriously a social networking site where people share _bookmarks_?

Re:Social Bookmarking? (0)

Anonymous Coward | more than 4 years ago | (#26931821)

To my understanding, it was sort of like del.ico.us

(So, to a limited degree, yes.)

Most sites where you can put bookmarks (and associated notes) online and allow others to access them call themselves "social bookmarking sites".

Again? (1)

Murpster (1274988) | more than 4 years ago | (#26931695)

Didn't almost the exact same thing happen to some blog hosting service within the past month? Wow.

don't trust "the cloud" (1)

HAMgeek (908543) | more than 4 years ago | (#26931829)

This is why I'm reluctant to trust "the cloud" with my data. ALWAYS keep a local copy and only use "the cloud" for backup.

User Error (1)

Idiomatick (976696) | more than 4 years ago | (#26931853)

Seriously, all hardware will eventually die, unless it joins the Q continuum or part vampire, demon or god... Really the summary should be. Mac user is retarded and doesn't backup. But then that would be redundant given that hes running a server on macs...

Oh don't worry it's in the 'cloud' somewhere (1)

gelfling (6534) | more than 4 years ago | (#26931881)

After all, it's the new paradigm.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>