Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Google Routes Around Outages

timothy posted more than 5 years ago | from the retrains-pigeons dept.

Google 105

1sockchuck writes "Making changes to Google's search infrastructure is akin to 'changing the tires on a car while you're going at 60 down the freeway,' according to Urs Holzle, who oversees the company's massive data center operations. In a Q-and-A with Data Center Knowledge, Holzle discusses Google's infrastructure, how it has engineered its system to route around hardware failures, and how it responds when something goes awry. These updates usually go unnoticed, but during system maintenance last month a software bug triggered an outage for Gmail."

cancel ×

105 comments

Just me? (5, Funny)

Anonymous Coward | more than 5 years ago | (#27334633)

Was it just me or did anyone else spend a few minutes contemplating how you actually could make a car that did allow you to change a flat while moving?

Re:Just me? (4, Funny)

esocid (946821) | more than 5 years ago | (#27334657)

Just you. I kept thinking about how I could use a car metaphor to describe how google...oh wait.

Re:Just me? (5, Funny)

Slumdog (1460213) | more than 5 years ago | (#27334843)

Just you. I kept thinking about how I could use a car metaphor to describe how google...oh wait.

I kept thinking about derailing a car, before I realized I was on the wrong track.

Re:Just me? (5, Funny)

tux0r (604835) | more than 4 years ago | (#27335777)

I kept thinking about derailing a car, before I realized I was on the wrong track.

I was going to reply about mixing metaphors, but then I lost my train of thought.

Re:Just me? (1)

Dekker3D (989692) | more than 4 years ago | (#27336235)

oh come on guys, i know posting here is like watching a train wreck, but.. let's stop hopping onto the bandwagon, shall we?

Re:Just me? (1)

Hordeking (1237940) | more than 5 years ago | (#27338919)

oh come on guys, i know posting here is like watching a train wreck, but.. let's stop hopping onto the bandwagon, shall we?

Better not keep going down this way. I think the light at the end of the tunnel is a train.

Re:Just me? (0)

lebscorpio (1368825) | more than 4 years ago | (#27336677)

Euh.. stop trying too hard.

It's easy (with a little help from google images) (0, Offtopic)

Ungrounded Lightning (62228) | more than 4 years ago | (#27336247)

I kept thinking about derailing a car, before I realized I was on the wrong track.

It's easy. (With a little help from Google Images...)

Car [westminstercollege.edu]

Derailer [mcn.org]

Re:Just me? (1)

PingPongBoy (303994) | more than 5 years ago | (#27345583)

All that aside, having tires is not absolutely critical to driving a car. One time I saw a Honda Civic going just fine down the street with two rims sans tires on the driver's side. Many people have no respect for their cars, but in spite of this being an older car, it was quite disturbing. The guy probably did something to shred his (bald?) tires, didn't have a spare or enough money/patience for a tow/new tires. As usual, no police were in sight.

I think the Internet is better maintained than poor people's cars. The Internet is more like a 10-axle vehicle that loses a tire not as infrequently as I used to think. Lines get cut all the time, and there's always someone splicing. It isn't juat Google worried about outages - we are all dealing with them, and barely notice.

Re:Just me? (5, Insightful)

Anonymous Coward | more than 5 years ago | (#27334757)

I thought about it for approximately 30 seconds. Then I realized that it is a bad analogy. A Google car would have hundreds of redundant wheels, changing one is easy.

Re:Just me? (5, Insightful)

Saerko (1174897) | more than 4 years ago | (#27334881)

That's what I was thinking too; and probably just function like an 18-wheeler where a tire can blow out and there's so much support that the load is still distributed adequately.

Basically, all this means is Google designs like Mack while everyone else designs like Chrysler...

Re:Just me? (0, Offtopic)

Slumdog (1460213) | more than 4 years ago | (#27335267)

That's what I was thinking too; and probably just function like an 18-wheeler where a tire can blow out and there's so much support that the load is still distributed adequately.

Basically, all this means is Google designs like Mack while everyone else designs like Chrysler...

I totally agree. Google does have a degree of hype, and infact if you compare their design to Yahoo and eBay, you may find many similarities.

A good amount of online bandwidth is consumed by google, I'd like to see an analysis of redundancy caused by google instead. Recently their search mechanism has gone haywire but noone notices.

Re:Just me? (0)

Anonymous Coward | more than 5 years ago | (#27338221)

... like an 18-wheeler where a tire can blow out and there's so much support that the load is still distributed adequately.

Unless it is one of the two front tires

Re:Just me? (1)

UncleTogie (1004853) | more than 5 years ago | (#27338519)

Unless it is one of the two front tires.

Actually, the load would still be supported if one or even both the front tires blew. It may not be under control, but that's another story....

Re:Just me? (0)

Anonymous Coward | more than 4 years ago | (#27335993)

I thought about it for approximately 30 seconds. Then I realized that it is a bad analogy. A Google car would have hundreds of redundant wheels, changing one is easy.

Not at 60mph it isn't! I don't care how many wheels you've got!

Re:Just me? (1)

sik0fewl (561285) | more than 4 years ago | (#27336149)

That took you thirty seconds?

Re:Just me? (1)

noidentity (188756) | more than 4 years ago | (#27336351)

Exactly. When a tire went out, its axle would lift up so it could be changed out safely. Actually, there would be several spare tire units ready to move down when another blew out, and there would be like 8 tires on the road all the time, so losing one wouldn't be an immediate problem. It is an interesting notion, a vehicle which wouldn't ever have to stop.

Perhaps you're thinking of a movie from the '70s? (1)

ahem (174666) | more than 4 years ago | (#27336599)

Like maybe this one? [imdb.com]

Re:Just me? (4, Interesting)

Yetihehe (971185) | more than 5 years ago | (#27334829)

Car is a bad analogy, building airplane in mid-air [youtube.com] is better.

Re:Just me? (0)

Anonymous Coward | more than 4 years ago | (#27335075)

As a former EDS employee, I'm surprised that the company hasn't tried to build an airplane in mid-air yet. Couldn't fail worse than NMCI.

Re:Just me? (1)

Idiomatick (976696) | more than 4 years ago | (#27334901)

I'd build a car with 6 wheels. If one of them popped you could have the wheel raise into some extra tall wheel well. Then change it and have it lower back to the ground after spinning up to speed. On a side note... Say 'Ronald rightly raised the right rear wheel well' three times fast.

Re:Just me? (1)

Bill, Shooter of Bul (629286) | more than 4 years ago | (#27335561)

Three fixed wheels on each side, so it could steer like a tank? I'd go one more axle to eight wheels six on the ground at a time and then deploy the fourth axle in the event of a flat, fix the tire while that axle is not spinning, and then return it to service when required. I wouldn't want to have to change a tire on a rotating axle. Or your could have just five wheels no fixed axles and deploy the fifth when one tire broke.

Re:Just me? (1)

PrescriptionWarning (932687) | more than 4 years ago | (#27334939)

duh silly, just put the car into hover mode. Sheesh you'd think Back to the Future 2 would have taught folks a thing or two about life close to 2015!

Re:Just me? (0)

Anonymous Coward | more than 4 years ago | (#27335107)

Easy, get in another car that's doing 60 and fix the first one at your leisure.

Re:Just me? (0)

Anonymous Coward | more than 4 years ago | (#27335131)

Not until I read your post. Then I imagined a second platform with wheels appearing under the car, raising it, while I repair the one that has a faulty tire. Now I'm wondering how the hell that would work. I think you've ruined the rest of my day, thanks a lot, asshole.

Re:Just me? (1)

rickb928 (945187) | more than 4 years ago | (#27335235)

It's not how well the bear dances. It's the fact that it dances at all.

Re:Just me? (0)

Anonymous Coward | more than 4 years ago | (#27335253)

I don't know about changing tires on a moving car, but these guys changed the oil on a single-engine airplane in flight:
http://untoldvalor.blogspot.com/2008/07/plane-that-flew-for-62-days-and-t-bird.html [blogspot.com]
They couldn't have stayed airborne for almost 65 days with the original oil so they routed the oil lines through the cockpit.

Re:Just me? (3, Informative)

zonky (1153039) | more than 4 years ago | (#27335389)

That problem has been solved for sometime, at least in Rallying. http://www.inforally.sibiul.ro/wrc-rally-news-10661-runflat_mousse_tyres_detail.html [sibiul.ro] At least, that was until they banned it.

Re:Just me? (1)

myowntrueself (607117) | more than 5 years ago | (#27337005)

wow brilliant!

Why was it banned? Is the mousse carcinogenic or something??

Re:Just me? (2, Interesting)

zonky (1153039) | more than 5 years ago | (#27337887)

Trying to lower costs of competing. Also, it could be argued that mousse meant that off-line mistakes were not 'punished'.

Re:Just me? (1)

theycallmejoe (1302777) | more than 4 years ago | (#27335395)

Guilty as charged.

Re:Just me? (1)

girlintraining (1395911) | more than 4 years ago | (#27335627)

Was it just me or did anyone else spend a few minutes contemplating how you actually could make a car that did allow you to change a flat while moving?

Retractable axles, like they use on dump trucks. With the wheels off the pavement, changing it becomes trivial. You need at least two sets of drive wheels though for that to work. But that's a closed-system solution. An adaptive solution would be a semi-truck that would deploy a ramp out the back. Drive the car onto the ramp and then effect the fix on the semi-truck. Then return the car to the flow of traffic. It would be dangerous though if it was one of the front tires; At 60 MPH, a normal car tire that has gone flat will shred itself in moments. In a convoy situation, it is likely that by the time the semi-truck positioned itself, the vehicle still traveling at that speed would have suffered extensive damage and be a risk to other vehicles in the area, including the rescue vehicle. Simply keeping it traveling in a straight line would be problematic, if my limited empirical evidence (episodes of Cops detailing police chases where the vehicle behing pursued has one or more damaged tires) is any indication.

Yeah... I'm a geek too.

Re:Just me? (1)

RichardJenkins (1362463) | more than 4 years ago | (#27335677)

Yes, I can do it in just 4 steps:

1. Stop car
2. Jack car up
3. Remove flat
4. Install spare

Can I get a job at Google now?

Re:Just me? (1)

mOdQuArK! (87332) | more than 5 years ago | (#27337315)

Now figure out how to do that without step #1...

Re:Just me? (2, Funny)

moxley (895517) | more than 4 years ago | (#27335835)

What are you talking about? I'm in America, and we need to find someone to blame for the flat first....

Then, maybe we can fix it..Got any nails and a hammer?

Re:Just me? (1)

rossdee (243626) | more than 4 years ago | (#27336175)

I remember something from a Popular Science|Mechnics magazine from the 1950's. This guy had heavily modified a car to keep going, it could be refueled, and change tires while moving. There were small wheels at each corner that could be jacked down and lift that wheel up, and an extended running board. Of course it didn't happen at highway speeds, just a few miles per hr. I think the goal was to cross the country without stopping for some record. Tires and other stuff were stored in a trailer towed behind the vehicle.

Re:Just me? (0)

Anonymous Coward | more than 4 years ago | (#27336237)

You are quite correct, his name was Louie Mattar and he modified a Cadillac including a shower on the running board, and an ironing board in the front seat. The car is currently in the San Diego auto museum collection, located in Balboa Park. Great place to visit by the way, tons of great museums and very scenic grounds all conveniently located right next to the San Diego Zoo.

Re:Just me? (1)

5of0 (935391) | more than 5 years ago | (#27338949)

That's exactly what I thought of, as I've been to that museum in San Diego. They've even got a little info page [sdautomuseum.org] on it, which you failed to link to. This is the internet! That's what it's for.

Already done (0)

Anonymous Coward | more than 4 years ago | (#27336597)

I did too... (1)

thenewguy001 (1290738) | more than 4 years ago | (#27336691)

I figured you'd need a car with redundant wheels that can be lowered and retracted while the car was moving. Probably also analogous to how google manages hardware downtime: redundancy.

While technically possible to make such a car, I don't see any practical use for such a system when it's just safer and more efficient to stop the car and replace the wheel.

At first I thought it might be a useful system for an armored car like the presidential limo, but the added weight of another motorized system would probably make it more of a liability than help in most situations, considering if something powerful enough to disable an armored wheel on an armored car will probably also destroy any systems adjacent to it.

Re:Just me? (0)

Anonymous Coward | more than 4 years ago | (#27336693)

No, I was thinking about his name... Urs Holzle... which I thought was pronounced Ur-s Holz-ls...

Re:Just me? (1)

Foodie (980694) | more than 4 years ago | (#27336793)

The article is a Q&A between DCK and Urs Holzle. What irony.

Re:Just me? (1)

mgblst (80109) | more than 5 years ago | (#27337605)

Wow, you guys really haven't seen the Big Bus?? Does all that, power by nuclear reactor, and comes with a pool and piano bar.

Re:Just me? (1)

Jack9 (11421) | more than 5 years ago | (#27338653)

Took me about 15 seconds to think about it. Knight-Rider style, load the car (into/onto/via grapple) a specialized truck/platform that exists to fix flats in transit or simply en-route. The car analogy is only bad because it doesn't really define the constraints based on the nature of the data, imo.

Re:Just me? (0)

Anonymous Coward | more than 5 years ago | (#27338891)

I saw it first done in this movie as a special effect... :)

http://www.imdb.com/title/tt0074205/

Great music score by David Shire!

Re:Just me? (1)

loic_2003 (707722) | more than 5 years ago | (#27340585)

You can do it with a motorbike.... [youtube.com]

Re:Just me? (0)

Anonymous Coward | more than 5 years ago | (#27341943)

Was it just me or did anyone else spend a few minutes contemplating how you actually could make a car that did allow you to change a flat while moving?

Actually, I saw that on TV about 10-15 years ago. I don't remember the name of the show.

The car was a four-door sedan with a stunt driver and two guys in the back seat.

The driver hits a ramp with the left-side tires, so the car is now balancing precariously on the two right wheels.

One guy in the back seat opens the door, climbs out, stops the back left wheel from moving, and undoes the lugnuts. Once the nuts are off, they pull the wheel off.

Then the 2nd guy passes the spare tire, and the guy puts the wheel on. Then tightens the nuts. Climbs back inside the car.

Now the driver (who has been carefully keeping the car on the two right wheels) turns sharply and the car lands back on all four wheels.

I know! I know! Pick Me!!! (5, Funny)

fuzzyfuzzyfungus (1223518) | more than 5 years ago | (#27334729)

It just treats the damage as censorship and routes around it, right?

Re:I know! I know! Pick Me!!! (1, Funny)

Anonymous Coward | more than 5 years ago | (#27334767)

It treats the routes as damage and censors them.

Re:I know! I know! Pick Me!!! (4, Funny)

fulldecent (598482) | more than 4 years ago | (#27335031)

It treats the censors as routes and damages them?

Re:I know! I know! Pick Me!!! (1)

DarkIye (875062) | more than 4 years ago | (#27335381)

It treats the censors as routes and damages around them.

Re:I know! I know! Pick Me!!! (1)

kobaz (107760) | more than 4 years ago | (#27335533)

It goes around the damages and routes the censors.

Re:I know! I know! Pick Me!!! (1)

Arthur Grumbine (1086397) | more than 4 years ago | (#27336043)

It goes around the damages and routes the censors.

It sensors the damage, and rounds the route?
...
Hey, we've almost returned full circle to TFS...

Re:I know! I know! Pick Me!!! (1)

hosecoat (877680) | more than 5 years ago | (#27337981)

it treats the routes as damage and censors them

Re:I know! I know! Pick Me!!! (0)

Anonymous Coward | more than 5 years ago | (#27341091)

We've been down this road before, and I think in the end we decided that the Internet interprets Thomas Jefferson as damage and routes around him.

Google File System Paper (5, Informative)

Anonymous Coward | more than 5 years ago | (#27334741)

To those looking for a more in-depth description, check out the technical paper on the google file system:

http://labs.google.com/papers/gfs.html

Had to read it for a search engines course in college, it's pretty darn spiffy.

Re:Google File System Paper (5, Funny)

SeePage87 (923251) | more than 4 years ago | (#27335071)

Why would we need a more in-depth description? We already got our car metaphor!

Re:Google File System Paper (1)

oldr4ver (1192469) | more than 5 years ago | (#27337527)

Because the car metaphor was meant to be humor not literal. If you want to actually learn something, then read the gfs paper. Being a former googler, I can tell you first hand that gfs is truly unique. Consider for a minute that the google infrastructure is designed to run like a single system. When something breaks in California, systems in Ireland know about it and adjust things automatically until things are fixed in Cali. EXT3, NTFS, XFS, and even the new BTrFS can't do that.

If you want to show off your amazing talent of being the net's largest tool, then just tack on another lame comment to the 1st commenter above.

Re:Google File System Paper (0)

Anonymous Coward | more than 5 years ago | (#27338181)

Wooosh !!

Re:Google File System Paper (0)

Anonymous Coward | more than 5 years ago | (#27342775)

If you want to show off your amazing talent of being the net's largest tool, then just tack on another lame comment to the 1st commenter above.

Nah, that's ok. You've already demonstrated that you are the internet's largest tool. Grow a sense of humor, you douche.

any large organization (1)

bugs2squash (1132591) | more than 5 years ago | (#27334783)

Has the same issue with rolling out updates and even though Google is (I suppose 10^100) times larger than any other company it does not mean that the same principles can't be applied. I don't see why Google should have any more problem than any other large company especially as they clearly have lots of resources and expertise to bring to bear.

Re:any large organization (1)

fuzzyfuzzyfungus (1223518) | more than 4 years ago | (#27334989)

I suspect that that is exactly why they get a (slightly puffy) profile piece written about them.

People with exotic problems and lots of resources make interesting fantasy material(hence the interest in reading about experimental fusion widgets and submarines and satellites and stuff).

People with common problems and few resources are mostly human interest/commiseration fodder("and that is how Mr. Bitter Q. Sysadmin keeps the department running for under $500/quarter, using nothing more than terrible coffee, worse attitude, and a pile of paperclips running Gentoo Linux")

People with common problems and lots of resources become "best practices" and get things like TFA written about them.

Re:any large organization (1)

TheEcho (1459195) | more than 4 years ago | (#27335461)

"a pile of paperclips running Gentoo Linux"
Can I get one of those? Does it move around when you are compiling something? At least my wife would think that would be worth staring at instead of a bunch lines of text when doing my daily 'emerge --sync && emerge -uDNav world'.

Re:any large organization (0)

Anonymous Coward | more than 5 years ago | (#27337203)

But does it run linux?

Re:any large organization (1)

SeePage87 (923251) | more than 4 years ago | (#27335105)

Probably because they handle more traffic than anyone else in the world, and their servers going down is a much more noticeable event than, e.g., GE's. And because the nature of their services probably complicates things as well.

nothing (0, Flamebait)

mekon (102072) | more than 5 years ago | (#27334865)

dont waste time reading this, holzles core answer is: well, if there is a problem, we fix it. and after that, we analyse the reason. brilliant.

Re:nothing (2, Funny)

Yetihehe (971185) | more than 4 years ago | (#27334957)

Analizing the reason first is a very good step. It could have saved me two hours today :/

Easy (1, Flamebait)

Daimanta (1140543) | more than 4 years ago | (#27334907)

Google treats outages like damage and routs around it.

Video of the car analogy (5, Funny)

Anonymous Coward | more than 4 years ago | (#27334923)

Excellent use of the car analogy, especially since it is possible to change a tire while driving a car. Youtube video [youtube.com] at 1:48.

Slightly..ahem... OT so posting anon.

Re:Video of the car analogy (1)

relguj9 (1313593) | more than 4 years ago | (#27335229)

Excellent use of the car analogy, especially since it is possible to change a tire while driving a car. Youtube video [youtube.com] at 1:48.

Slightly..ahem... OT so posting anon.

lol, all I can think of those drivers saying after they read that quote is... "ahahhaha... too easy! too easy!"

Article doesn't really say anything. (5, Interesting)

girlintraining (1395911) | more than 4 years ago | (#27335119)

You know, the article read like a press release. Hasn't slashdot whored itself out enough lately on these kinds of things? Google is so ultra-reliable, blah blah, 24x7, blah blah, commitment, blah blah, premier service partner, blah blah... I get that kind of talk enough in staff meetings. Where's the meat already!?

Why not write an article with some nice graphics saying what happens to my request from the time I hit "Search" to the time I click a result. List off all the servers it goes through, their roles, how they're monitored, etc. Give examples of failure and show the mode decisions the software makes (and where this software is running) -- show the latencies and other performance impacts as my request bounces over failure after failure. That's what I expect when I pull up an article entitled "How Google Routes Around Outages". Something useful, professionally enriching, intellectually stimulating, etc. In short, tell me why I (should) never see a "500 Internal Server Error" from Google, but I do from just about every other major website I've used.

Re:Article doesn't really say anything. (0)

Anonymous Coward | more than 4 years ago | (#27335211)

MOD PARENT UP!

article was a complete waste of time.

LINK BAIT!!!!

Re:Article doesn't really say anything. (2, Insightful)

Anonymous Coward | more than 4 years ago | (#27335731)

I would wager to say you would learn all this if you were hired on as part of google's site reliability team. Probably most of that info. you're curious about is something they're not willing to talk about in great detail for competitive reasons.

Re:Article doesn't really say anything. (2, Interesting)

Red Flayer (890720) | more than 4 years ago | (#27336179)

You know, the article read like a press release. Hasn't slashdot whored itself out enough lately on these kinds of things?

YMBNH.

This has been happening since as long as I've been lurking slashdot (2000?), and didn't go away once I set up an account (2002? maybe 2003). And from the YMBNH posts I saw when I began lurking, this has apparently been an issue since the beginning (or shortly thereafter).

At any rate, complaining about it won't do much good. There's a saying maybe it might help you to repeat:

Give me the strength to change the things I can, the humility to accept the things I can't, and the wisdom to know the difference.

Re:Article doesn't really say anything. (1)

ral8158 (947954) | more than 5 years ago | (#27337739)

Is that an Ignatian prayer?

Re:Article doesn't really say anything. (1)

Qzukk (229616) | more than 4 years ago | (#27336485)

I have to admit I was disappointed too. We recently had our colocation facility fall down on the job (turns out they have no alternate way to contact everyone should their internet fail) and I was hoping to get some insight into setting up hot sites, how a site should determine whether it can see the internet or not (clearly I can't just ping google anymore), and other things that would be useful from a technical perspective.

Instead I get "white box" and "black box" monitoring, and I have yet to figure out how to get anything but LCD panels out of google for these terms.

Re:Article doesn't really say anything. (1)

oldhack (1037484) | more than 5 years ago | (#27337637)

Yeah, I think we should, like, get out money back. Or something.

Re:Article doesn't really say anything. (1)

oldr4ver (1192469) | more than 5 years ago | (#27337639)

I could tell you, but I'd have to kill you.

Re:Article doesn't really say anything. (0)

Anonymous Coward | more than 5 years ago | (#27338597)

'Umm, cuz that'd be too easy?

Simple, really... (4, Informative)

neokushan (932374) | more than 4 years ago | (#27335185)

The key point:

When they get an outage, they check how it was caught and if it wasn't caught automatically, they figure out how to next time. Simple rule: They learn from their mistakes and don't put all their eggs in one basket.

Re:Simple, really... (4, Funny)

SeePage87 (923251) | more than 4 years ago | (#27335255)

Bollocks! I tried learning from my mistakes once, and boy did that ever turn out bad. Now I know better than to try that again.

Re:Simple, really... (0, Flamebait)

neokushan (932374) | more than 4 years ago | (#27335373)

Only because your mistake was NOT drugging the bitch.

Re:Simple, really... (1)

mgblst (80109) | more than 5 years ago | (#27337627)

I agree, if you keep making the same mistake over and over again, you learn to deal with it better, it is less of a surprise, and you are more prepared for it. You can become a real expert at that mistake, which keeps on happening. If you stop making this mistake, you might start making new mistakes....well, who knows how to fix those?

Re:Simple, really... (1)

layer3switch (783864) | more than 5 years ago | (#27337545)

But what we have here is, many baskets but with only one egg, and the problem is to make sure the egg going into the right basket "AND" getting back out from that basket while number of baskets grows!

What it sounds to me is that Google intelligence is not redundancy but rather granular task assignment with each task in mind that level of fault tolerance in result should be greater than risk of failure in result as it grows/scales.

For example (simplistic), raid 10 vs. raid 6 comes to mind. Both tolerant to equal amount of failure, what scales better with same or better tolerance to data loss? Which one has better predictable outcome? Which one gives you more time to recover and bang for the buck?

Everyone most likely has different answer to that, and I believe, Google is no different.

Re:Simple, really... (1)

oldr4ver (1192469) | more than 5 years ago | (#27337683)

Simple rule: They learn from their mistakes and don't put all their eggs in one basket.

Yes they do. Its called the Earth.

Changing tires (2, Interesting)

Spazmania (174582) | more than 4 years ago | (#27335339)

akin to 'changing the tires on a car while you're going at 60 down the freeway,'

This is not so hard. Just design the car with 4 axles instead of 2 and lift one off the road at a time. Helps if it can swivel for easy access to the lugnuts.

Re:Changing tires (1)

Arthur Grumbine (1086397) | more than 4 years ago | (#27336093)

akin to 'changing the tires on a car while you're going at 60 down the freeway,'

This is not so hard. Just design the car with 4 axles instead of 2 and lift one off the road at a time. Helps if it can swivel for easy access to the lugnuts.

I tried that once, but before I had even produced isometric drawings my tire had shredded, and my rim was trashed...

Inspiration taken from the same thing it runs on! (5, Insightful)

Thantik (1207112) | more than 4 years ago | (#27335383)

Isn't this how the *internet* is (at least in theory) supposed to work anyhow? Instead we have 90% of the cables that route the middle-east/europe running through the same canal. And I know of VERY few ISPs who actually make their systems redundant anymore. /sadface

Replacing a wheel on a car going 60mph (2, Informative)

krnpimpsta (906084) | more than 4 years ago | (#27335587)

Ok, granted they are not travelling 60mph, this is still pretty impressive.. I consider this on-topic, because maybe it is possible to do what the summary suggests (replace wheel in moving car). :)

Watch from 1:55 to 2:35:
Youtube video of guys replacing a wheel on a car while it is moving.. [youtube.com]

Re:Replacing a wheel on a car going 60mph (0)

Anonymous Coward | more than 4 years ago | (#27336265)

http://www.youtube.com/watch?v=Ojv5Vp9XREk

Doing it on a motorcycle

Re:Replacing a wheel on a car going 60mph (1)

Anubis IV (1279820) | more than 5 years ago | (#27337241)

That only works on the non-driving wheels of two-wheel drive vehicles though. To offer a quick analogy, if Google were a car, then Google would only be able to replace those servers that are not responsible for helping drive the search engine forward on the Internet superhighway.

Hoelzle (0)

Anonymous Coward | more than 4 years ago | (#27335647)

*ahem* It's Hoelzle. Just saying.

Recursive googling (1)

adrianhensler (454654) | more than 4 years ago | (#27335697)

I'm sure they just do exactly what I do when I'm at work and have a problem: they google for an answer to the problem at hand.

Oh, wait.

The above sort of leads into explaining my fear of asking google "is google alive" and the ensuing apocalypse.

Urs Holzle??? (1)

GuyFawkes (729054) | more than 4 years ago | (#27336435)

Tell me I am not the only one who read that and wondered if Google also employ I P Nightly etc

In Soviet Russia... (0)

Anonymous Coward | more than 5 years ago | (#27340003)

In Soviet Russia, outages route around Google

Sorry couldn't resist

the phrase route around (1)

nimbius (983462) | more than 5 years ago | (#27340059)

has never really seemed appropriate to me. in an infrastructure you have quorum managers, active fencing, and weighted peers...what does road construction have to do with any of it?

Amazing speed (0)

Anonymous Coward | more than 5 years ago | (#27341195)

Woohooo, going at an amazing 60 mph! Imagine that! The wind might well blow away you hat!

For Germans, talking about that speed like it was fast is really kinda cute. :-)

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...