Tier One ISPs Dying 394
xbmodder writes "Two tier one ISPs are down today. At about 23:30PST both Verio and Level 3 starting having problems with routes. According to Level 3 this is a software upgrade gone awry. Is this the end for Level 3?" Many, many reports about this are coming in, and if you're wondering why the stories were rather sparse overnight, it's because it's difficult to post them without internet access. Hope everyone else is back online too.
Could be good (Score:5, Funny)
-S
Re:Looks good now... (Score:5, Funny)
Re:Looks good now... (Score:5, Funny)
Flicker (Score:5, Funny)
Re:Flicker (Score:2)
Re:Flicker (Score:2, Funny)
Re:Flicker (Score:2, Interesting)
Re:Flicker (Score:5, Funny)
Re:Flicker (Score:5, Funny)
TEOTWAWKI (Score:5, Funny)
Re:TEOTWAWKI (Score:2)
In which way are the domes involved in the internet failing?
Re:TEOTWAWKI (Score:5, Funny)
Sure, but I feel fine.
*shrug*
Re:TEOTWAWKI (Score:4, Funny)
SICK HIM!!!
Nitpick (Score:5, Informative)
Nitpick to the max: Ye Olde Spell(ing)s (Score:3, Informative)
Re:Nitpick - gang of thugs (Score:4, Interesting)
Re:Flicker (Score:2)
flapping (Score:5, Interesting)
Takes me back... My first night on the job a rat in Berkeley chewed through the wrong cable and got himself fried -- he also happened to take the entire west-coast off the internet for the better part of a day.
Then there was the time an electrical worker got vaporized in a hole near MIT which caused quite a problem too as it overloaded the MIT power station, but the fallout wasn't nearly as bad as the day of the rat...
Clarification (Score:5, Informative)
Re:Clarification (Score:3, Funny)
Re:Clarification (Score:3, Informative)
Re:flapping (Score:5, Interesting)
OT, but it brings back memories of working at Purolator Courier in the machine room. IBM mainframe shop.
We had had trouble with the damn fire suppression all day. On third shift, around 3 AM, the trouble alarm went off (again) for the umpteenth time. One of the operators, a nervous fellow who was a little bit green, went over to the annunciator panel and opened it to see what the Trouble Might Be.
A fire technician he was not, and he apparently didn't know the difference between the trouble bell and the klaxon that would sound when a halon dump was about to occur; so he reached around the open panel door and hit the halon defeat.
Or so he thought.
It was actually the Big Red Switch.
The whole room (full of 3420 and 3480 tape drives, the 3745s, the 3800 laser printers; and the floor above, containing trivial bits like the DASD and the CPU all plunged into a deafening silence.
We all stared at each other and at the newbie BOFHeck.
A few minutes later, the phone rang. It was the Indianapolis air hub for Purolator, wondering why (when they were about to receive about 150 planes from all over the country) they didn't have anything useful displayed on their green screens.
That was a fun morning.
Ah, those were the days indeed.
Re:flapping (Score:3, Interesting)
Every Sunday night they would switch from mains to batteries to exercise the system. So at around 1am the air conditioners and lights would go out and the silence would be deafening. It always made your heart skip a beat while you checked to make sure the lights on the drives were still on. 30 seconds later the lights and air conditioning w
Halon stories (Score:3, Interesting)
Some time after my contract had ended I visited the place and it was a total disaster.
During the model change shutdown (when most of the plant maintainence and rearrangement was done) the millwrights were welding on some cableways on the ceiling of
Re:Flicker (Score:3, Funny)
Outtage Explained (Score:5, Funny)
Call me silly? (Score:4, Funny)
Is that like a bandwidth wholesaler or something?
Re:Call me silly? (Score:4, Informative)
Eg. you have your own large backbone, you own all your equipment.
In effect, a small and wholly owned internet that peers with other internets.
Re:Call me silly? (Score:2)
(hehe he said internets, George is that you?)
Re:Call me silly? (Score:2)
Perhaps you've never heard of Internet2 [wikipedia.org].
Re:Call me silly? (Score:3, Informative)
Re:Call me silly? (Score:3, Informative)
The way Tier 1 ISP is defined is mostly by its magnitude. At the time I've worked for that ISP, the rough rule of tumb was that Tier 1 ISP must have a few large capacity pipes from coast to coast at least. Must carry enough traffic so other Tier 1 ISPs can exchange the
Re:Call me silly? (Score:5, Informative)
Re:Call me silly? (Score:2)
(How could I forget to check Wiki)
Re:Call me silly? (Score:5, Interesting)
These contracts were awarded for the original 15 NSF sponsored NAPs, and to become a Tier 1 ISP, you had to have atleast DS3 connectivty to all 15 NAPs.
It's a very old and crappy definition, and I wish people would stop using it, because it is very easy to meet now adays, and most of those original NAPs are now insignificant, compared to the power of the force.
Re:Call me silly? (Score:3, Interesting)
Considering that free peering is likely only established between 2 networks that have close to a 1:1 bit exchange, this is a very high bar to meet.
What is this about? (Score:3, Insightful)
Re:What is this about? (Score:5, Informative)
The (basic) implications of this is that a good chunk of the internet as a whole is inaccessible to the rest of the internet.
Re:What is this about? (Score:4, Insightful)
They won't be back tomorrow. All gone.
Reality check: An internet outage, no matter how big, is no different than a power outage. Yeah, here in the US we would be talking about loss of power to both coasts with only the middle left running. But after the outage life goes on.
Re:What is this about? (Score:5, Informative)
When a Tier 1 provider goes down, their customers go down too. That picture on the Boing Boing page shows a list of the Tier 1 providers. Every ISP that is NOT a Tier 1, gets their access from a Tier 1.
People speculate that Level 3 is dying because they've been making some really bad decisions lately, resulting in a lot of outages. A couple of weeks ago, they actively filtered out traffic from their competetor, Cogent, over a dispute from how much to charge at the point their networks exchanged traffic (called a 'peering point'). Now this. The rumor is that the company is in financial trouble.
Re:What is this about? (Score:4, Informative)
The Cogent spat isn't over yet either - Level 3 are going to de-peer Cogent again on November 9th. They are trying to force Cogent to pay for transit, but right now it looks like Cogent holds the strongest hand and Level 3 will be once again forced to back down.
Re:What is this about? (Score:5, Informative)
Re:What is this about? (Score:4, Informative)
I predict that Cogent will do the same again as well - not lift a finger to fix the problem when they are de-peered on November 9th, and Level 3 will probably end up being forced to re-peer as customers whine that they are not getting the whole Internet and threaten to take up Cogent's free 1 year offer.
Re:What is this about? (Score:5, Informative)
More precisely, Level3 seem to own 23,000 miles of optic fiber [prnewswire.com].
The rumor is that the company is in financial trouble.
Yeah, not so much of a rumor anymore either -- Level 3 loss widens [reuters.com].
Re:What is this about? (Score:3, Informative)
true in a sense but highly misleading. A large tier 2 isp is likely to have uplinks to MULTIPLE tier 1 providers as well as many peering links of thier own.
Showing solid green now. (Score:3, Interesting)
Guess not (Score:5, Informative)
Re:Guess not (Score:5, Informative)
Comment removed (Score:5, Funny)
YES, TERRORISTS WANT TO TAKE OUR PORN!!!11! (Score:3, Interesting)
It doesn't make me wonder that. Terrorists do not give a shit about this kind of thing. To even invoke the word "terror" in this discussion is ludicrous.
- A.P.
Re:Guess not (Score:5, Interesting)
Oh please. You know, it's pretty easy to figure out if it's something likely to be attempted by terrorists or not. The simple test is does it cause mass "terror". As annoying as it might be, lack of internet access is an annoyance. Perhaps a very expensive and exasperating annoyance, but it won't cause mass terror. Terrorists prefer things like bombs, or poison gas, or disease. Some other things people get worked up about but terrorists are unlikely to attempt: sabotaging bridges and tunnels to cause traffic jams; sabotaging electricity distribution to cause blackouts; sabotaging railroad tracks, making commuters late for work!. Think DEATH, not irritation. Quit with the automatic "terrorist hysteria" already, people!
huh?? (Score:2, Insightful)
Re:huh?? (Score:2)
Re:huh?? (Score:5, Funny)
As a sophisticated veteran Windows user... (Score:3, Funny)
About a quarter of the people i know lost internet (Score:3, Funny)
Re:About a quarter of the people i know lost inter (Score:2)
Stupid, really.
Re:About a quarter of the people i know lost inter (Score:2)
To them own3d is something your dad would say.
Get with the times, daddio =)
Noticed it this morning (Score:5, Interesting)
It seems fixed now though, so no, this isn't the death of the Internet just yet.
Tier 1's dying? OMG! (Score:2, Funny)
Re: (Score:3, Funny)
This just in... (Score:4, Funny)
Tier One
/.'ed (Score:5, Funny)
I noticed it. (Score:3, Informative)
Off to the test!
Re:I noticed it. (Score:3, Funny)
I can't take the test, professor. The sites with the study guides were Slashdotted this morning!
You deserve an A for shrewdness.
Non event... for now (Score:5, Informative)
Re:Non event... for now (Score:5, Funny)
Yeah, I remember life before the internet. I used to read books, watch TV, and even occasionally go outside under that big yellow face.
*shiver*
Re:Non event... for now (Score:3, Funny)
Re:Non event... for now (Score:3, Funny)
Re:Non event... for now (Score:3, Funny)
Way to remind the rest of us we're old, young whipper-snapper!
Why in my day, we didn't have the Internet! You'd have to carry your packets to your neighbor's house so he could copy them to his computer, and that was networking. Of course without the Internet, we didn't have IP, we only had P. Thus the phrase, "I'm going to go take a P" -- as in "I'm going to take this data to my friend's house."
See, I bet they didn't teach you that in
Link in the article (Score:4, Funny)
I notice the article links back to Slashdot... I wonder is Slashdot is going to get BoingBoing'ed?
What kind of Timeframe (Score:4, Interesting)
Re:What kind of Timeframe (Score:4, Informative)
Re:What kind of Timeframe (Score:3, Informative)
History of this? (Score:2, Funny)
So much for the internet surviving a nuclear war! (Score:4, Insightful)
Is the Internet Down? (Score:5, Funny)
Overlay Routing (Score:5, Informative)
http://www.usenix.org/events/nsdi04/tech/full_pap
http://www.eecs.umich.edu/~farnam/pubs/2005-hwj-i
Time in UTC (Score:3, Informative)
But perhaps what's really meant is:
23:30 PDT = 06:30 UTC = 08:30 CEST ?
Over 100 comments so far.... (Score:3, Funny)
The main problem areas... (Score:4, Interesting)
For a breakdown, check out this view of the data [keynote.com].
Microsoft? (Score:3, Funny)
Were they upgrading to one of the Beta builds of Windows Vista Home Edition?
You insensitive clod! (Score:3, Funny)
Level 3? (Score:3, Interesting)
I don't know that they've replaced Sprint yet on my list of most sucktastic internet companies. Time was you lost connectivity to an important piece of the Internet (Like your favorite Quake TeamFortress server) and a traceroute would show the failure somewhere in the Sprint backbone. So far they've been more reliable than Sprint at their worst, at least for me.
If they go under, well Tier 1's don't ever really die. Chances are one of the other Tier 1's will buy their assets and it'll be business as usual. Usually the buyer is MCI.
Of course the true test is pretty easy -- has anyone who works at Level 3 had their paycheck bounce yet? Surely there are a few readers among their employees...
Re:Level 3? (Score:3, Insightful)
And with fewer and fewer tier 1's, is it any wonder things like this happen?
FFS, what a fucking dreadful summary (Score:5, Insightful)
No, of course not, you blithering imbecile. L3 had a 2 hour global routing meltdown. Now, it's fixed. Whilst their routes were flapping, other carriers saw transient increases in latency and some problems with reachability, to some sites. However, everything continued to work properly for non-L3 customers. Two hours later L3's routes are back and working properly. End of story, nothing to see here, move along please.
Slashdot editors, do you really expect us to believe that no-one had submitted a more coherent or accurate story than this one? Come on, for heaven's sake.
Anyway, a network engineer's view can be seen in the overnight traffic on NANOG: http://www.merit.edu/mail.archives/nanog/2005-10/ [merit.edu] "Tier One ISPs dying" indeed. Worst. Story. EVER.
Re:FFS, what a fucking dreadful summary (Score:3, Insightful)
However, L3 has been having "issues" this month that have left a lot of lower-tier ISPs in the uncomfortable position of explaining to their customers "We know the internet is down but there's nothing we can do about it." This outage really can't be good for their reputation, and I can see more potential customers taking their money elsewhere because of this.
Just because the technical issues have been fix
Re:FFS, what a fucking dreadful summary (Score:3, Interesting)
Re:FFS, what a fucking dreadful summary (Score:3, Funny)
Come on, this is SlashDot.
It should be:
X is Dying (Score:5, Funny)
Has to be said... (Score:3, Interesting)
Unixshell graphs caught my eye (Score:3, Informative)
I was looking for a Linux Virtual Host, blah, blah.
Stumbed apon these pretty pictures [unixshell.com] (near bottom of page) .
Curious, I thought, what happened to Level(3) ? I though for a second because perhaps unixshell had a peering with those people that Level(3) were in dispute with.
Nope, just one of those regular outages that make the 99.999% promises sound a little over done.
Re:it's difficult to post them without internet ac (Score:2, Funny)
Re:point of the internet? (Score:5, Insightful)
Re:point of the internet? (Score:2)
That it can't die is 90s hype
Re:point of the internet? (Score:2)
Re:point of the internet? (Score:2, Interesting)
Yes, the Internet enables/permits/allows redundant routes, but...
No, it doesn't require/demand/"enforce with any government or legal authority" redundancy at all levels.
So any smaller ISPs connected to Level3, and all their customers would have had problems reaching the rest and being reached by the rest.
(sarcasm mode)Obviously this wouldn't have happened if the EU had been in control!(/sarcasm mode)
Actually, how many of these corporations are US companies, and how many are NOT?
Re:PST? (Score:2, Informative)
Re:Isn't God trying to tell us something? (Score:4, Insightful)
Re:Yep My ISP went wrong (Score:3, Funny)