Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Verizon's Accidental Mea Culpa

Soulskill posted about 3 months ago | from the information-wants-to-be-hamstrung dept.

Verizon 390

Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).

Sorry! There are no comments related to the filter you selected.

But scarcity! (5, Insightful)

StikyPad (445176) | about 3 months ago | (#47481997)

If people don't think bandwidth is a scarce commodity, how will we get them to pay through the nose for it?!?

Level 3 - start pulling cards (3, Interesting)

emil (695) | about 3 months ago | (#47482153)

Find locations where you will hurt Verizon customers, and cut the cables. Do so publicly. Precondition repair on upgrades of Verizon's network as you direct. If Verizon doesn't want network neutrality, then punish their customers.

Re:But scarcity! (-1)

SQLGuru (980662) | about 3 months ago | (#47482155)

It looks like the Level 3 post has been pulled. It goes to their 404 page which has a link to recent posts which lists the very post linked in the article.....and the recent post link ALSO takes you to a 404.

Re:But scarcity! (5, Informative)

StikyPad (445176) | about 3 months ago | (#47482219)

Verizonâ(TM)s Accidental Mea Culpa
Mark Taylor / 18 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflixâ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizonâ(TM)s own paying broadband consumers.

His explanation for Netflixâ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know itâ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But hereâ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizonâ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatâ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3â(TM)s network interconnects with Verizonâ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So letâ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.

lvltvzw

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested â" in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something weâ(TM)ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they canâ(TM)t afford a new port card because theyâ(TM)ve run out â" even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If thatâ(TM)s the case, weâ(TM)ll buy one for them. Maybe they canâ(TM)t afford the small piece of cable between our two ports. If thatâ(TM)s the case, weâ(TM)ll provide it. Heck, weâ(TM)ll even install it.

But, hereâ(TM)s the other interesting thing also shown in the Verizon diagram. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? Could it be that Verizon does not want its customers to actually use the higher-speed services it sells to them? Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitorsâ(TM) costs?

To summarize: All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.

Why might that be? Maybe we should ask David Young.

http://webcache.googleusercont... [googleusercontent.com]

Re:But scarcity! (5, Informative)

Anonymous Coward | about 3 months ago | (#47482261)

Google's cached copy...

http://webcache.googleusercontent.com/search?q=cache:DBHDyx7n4D0J:blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/+&cd=1&hl=en&ct=clnk&gl=us

Re:But scarcity! (5, Funny)

EvilSS (557649) | about 3 months ago | (#47482595)

It looks like the Level 3 post has been pulled. It goes to their 404 page which has a link to recent posts which lists the very post linked in the article.....and the recent post link ALSO takes you to a 404.

Strange, the link works fine for me. Your ISP isn't Verizon by chance is it?

Re:But scarcity! (-1, Troll)

schnell (163007) | about 3 months ago | (#47482259)

Ugh. We're back to the same knee-jerk reactions and slanted clickbait stories. Shame on Slashdot.

This is not a one-sided story (very few stories in the real world are) as the summary attempts to show it. This isn't about wantonly deciding to screw over paying users, it's about a peering dispute. Of course Verizon has bandwidth to spare, that's not the problem. The issue is that they don't think of a much smaller ISP like Level3 as a peer, and don't want to give them settlement-free peering - they don't peer for free with lots of other ISPs for the same reason. Level3 offering to pay for the hardware is completely disingenuous, since that is a drop in the bucket of what paid transit costs as a monthly service. (By the way, Level3 played the role of Verizon to Cogent in the exact same kind of dispute [prnewswire.com] a few years ago.) But clearly if Verizon cared about the experience of its end users it would find another solution that didn't involve free peering, like CDN installations to support Netflix. So they are being jerks as well.

So Verizon doesn't want to peer for free with Level3 even though it would help their customers; and Level3 wants to peer for free instead of paying like everybody else. This goes on all day every day in the ISP world. People only notice this particular instance because unfortunately users can see the negative effects directly. Long story short, there are no white hats in this story, only gray ones on both sides, both trying to spin public perception. Stories like this one on Slashdot do no favors to a reasonable understanding of the situation.

Re:But scarcity! (5, Insightful)

ffsnjb (238634) | about 3 months ago | (#47482371)

In what world do you live that Level3 is a "much smaller ISP"? Level3 is a global tier 1 ISP, FFS.

Re:But scarcity! (4, Insightful)

Dishevel (1105119) | about 3 months ago | (#47482673)

In what world do you live that Level3 is a "much smaller ISP"? Level3 is a global tier 1 ISP, FFS.

In a world where schnell and shill sound a lot alike.

Re:But scarcity! (0)

Anonymous Coward | about 3 months ago | (#47482385)

Long story short, there are no white hats in this story, only gray ones on both sides, both trying to spin public perception. Stories like this one on Slashdot do no favors to a reasonable understanding of the situation.

There may be no white hats, but the news filters seem to be wearing a new pigment. [slashdot.org]

Re:But scarcity! (5, Insightful)

StikyPad (445176) | about 3 months ago | (#47482397)

Apples to oranges. Level3 and Cogent aren't last-mile providers; they're Tier 1 backbone providers. Tier 1 providers have things like peering agreements -- last mile providers do not. Last mile providers are (and sell) unbalanced connections, so it's impossible for them to ever have "peers."

A better way of thinking of it is that Verizon should be representing the interests of its customers, because Verizon is the gateway between the customers, and the rest of the internet. It's not doing that job -- it's trying to play both sides against each other. This is what middlemen do, of course, and they're entitled to do it, but as long as they have a monopoly (which they do), then there should be limits, oversight, and accountability.

Re:But scarcity! (2, Insightful)

fahrbot-bot (874524) | about 3 months ago | (#47482607)

A better way of thinking of it is that Verizon should be representing the interests of its customers, ...

Nice sentiment, but, unfortunately, a public corporation's responsibility is to its shareholders and their interests - which is simply $$$. (and probably executives and cushy bonuses, etc...)

Re:But scarcity! (0, Informative)

Anonymous Coward | about 3 months ago | (#47482457)

I instantly could tell this was a verizon smear post when you said that level 3 is a "small isp". I'm just going to post literally the very first lines from the wikipedia article, to show how horrible you shills are.
It operates a Tier 1 network.[1] The company provides core transport, IP, voice, video, and content delivery for most of the medium to large Internet carriers in North America, Latin America, Europe, and selected cities in Asia.[2] Level 3 is also the largest competitive local exchange carrier (CLEC) and the 2nd largest provider of fiber optic internet access (based on coverage area) in the United States.[3]
You idiots can't even take the time to have BASIC knowledge about what they are making you smear. If shills had honor, you'd probably kill yourself, but if you had any honor, you wouldn't be a shill.

Ironically the captcha is "informed". Something a shill really should be.

Re:But scarcity! (5, Insightful)

Shados (741919) | about 3 months ago | (#47482473)

The problem is still the lack of competition in the market. If everyone had the choice between 4-5 ISPs, considering the popularity of Netflix, consumer ISPs would be paying Level 3 truckloads of money to ensure Netflix works flawlessly...and the roles may even be reversed (where Level 3 tries to gouge Verizon, since they'd know Verizon would have no choice or lose a ton of customers).

But since there isn't any competition, Verizon takes their own customers hostages...

Re:But scarcity! (1, Flamebait)

sribe (304414) | about 3 months ago | (#47482573)

...a much smaller ISP like Level3...

You're fucking idiot. Shut up and get back under your rock.

No excuses left (5, Insightful)

Anonymous Coward | about 3 months ago | (#47482005)

Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.

Re:No excuses left (4, Insightful)

Stolpskott (2422670) | about 3 months ago | (#47482139)

Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.

But, but, but... regulation is the antithesis of the Capitaist way that our republican Democracy has weaned its children on since it was formed!!
I do tend to agree though - regulation of ISPs is probably the only way to deal with this.
Capitalist theory says that if an incumbent merchant/provider is too inefficient to provide a good service or if another potential merchant/provider thinks they can do a better job for a lower price, then that new provider will step in and provide said service. The threat of that is what keeps the incumbent lean and competitive, and the result is a competitive environment that is generally good for the consumer and rival providers seek to offer better deals to entice custom away from their competitors.
However, that theory assumes that there is a very low or non-existent barrier to entry into that competitive marketplace. Given the initial infrastructure setup costs and, in many cases, exclusivity contracts between providers and the municipal areas which would present the profits to drive services out into more marginal areas, the barriers to entry into the Tier 1 ISP market are prohibitive, to the point where you need to be a corporate entity the size of Google to be able to reasonably make the capital investment required.
As such, the local markets for each ISP more closely resemble non-competitive monopolies with the illusion of choice being provided by third party suppliers who typically have to by access to the resources from the incumbent monopoly - they get wholesale prices, and the consumer sees some small price reductions if the third parties can make enough money to operate by charging the consumer slightly less than the discount they got from the incumbent. But fundamentally, everything is still controlled by that original monopolistic provider, so services suck, progress is stifled because there is no incentive for change, innovation is discouraged, and the level of capacity/reliability is never going to be any more than "just barely enough so that we can maximise our profit margins".

Re:No excuses left (0)

Anonymous Coward | about 3 months ago | (#47482639)

Verizon has got to be the worst, most despicable company I have ever had the dis-pleasure of dealing with. If they were to implode in a massive bankruptcy, I would dance on their stinking grave.

Re:No excuses left (4, Insightful)

Virtucon (127420) | about 3 months ago | (#47482667)

Regulation for the public benefit = good. Examples: Public Utilities, Healthcare, Agriculture, Air Quality/Environmental Protection.
Regulation for the sake of Regulation = bad. Examples: 70,000 + pages of IRS Regulations and 30,000 pages of tax code written by special interests and bureaucrats.

PR needs to talk to tech (5, Insightful)

MrDoh! (71235) | about 3 months ago | (#47482009)

Was obvious people were going to figure out everything Verizon was saying is BS, and that they'd continue to get bad press about this. You'd think the PR droids spouting this stuff would talk to their tech people and listen. But they probably said "look, just give us a pretty graphic right?" "But, techs will see through your spin" "Leave that to us" "But it'll make us look even worse" "You don't get paid to deal with this" All too predictable, and the same techs are probably still being yelled at.

Re:PR needs to talk to tech (2)

arbiterxero (952505) | about 3 months ago | (#47482189)

It is painful how true that is.

Most of the time they get away with it, I'm just ecstatic that they didn't this time.....and soooo badly......

Connect with a VPN (5, Interesting)

LinuxFreakus (613194) | about 3 months ago | (#47482047)

Just connect to a VPN first and then use Netflix. You'll be able to clearly see how much Verizon is throttling. I've been using this as a workaround for a while now. I'm not sure why more people don't think of pointing this out when Verizon's tech support people claim there is no throttling.

Re:Connect with a VPN (5, Interesting)

i.am.delf (1665555) | about 3 months ago | (#47482077)

You can also escape this bottleneck using an IPv6 tunnel to he.net.

Re:Connect with a VPN (4, Funny)

jythie (914043) | about 3 months ago | (#47482103)

I wonder if Netflix themselves could provide such a service, maybe even run it through Comcast? Now there would be a fun bit of PR... "yes, your netflix connection runs better if we route from your Verizon DSL through Comcast, imagine how much better it would be if you just switched to Comcast?"

Re:Connect with a VPN (4, Funny)

nine-times (778537) | about 3 months ago | (#47482525)

Nothing is better when you switch to Comcast.

Re:Connect with a VPN (5, Insightful)

Anonymous Coward | about 3 months ago | (#47482115)

According to tfa, they actually aren't throttling. Throttling implies that they are deliberately shaping traffic inside their network to limit your bandwith.

What they are really doing is deliberately creating a bottleneck at key peering locations through negligent inaction when it comes to upgrading infrastructure.

Small difference, I know, but very important when you actually talk about throttling, and likely the argument they would make if the FCC took them to court over it.

Re:Connect with a VPN (0)

Anonymous Coward | about 3 months ago | (#47482317)

According to tfa Verizon is deliberately applying an artificial limit of 40Gbps to the bandwidth of traffic from Netflix as it enters their network. That seems to fit your definition of "throttling".

Re:Connect with a VPN (0)

Anonymous Coward | about 3 months ago | (#47482657)

It's a 40Gbps limit caused by hardware limitations at the interconnect point, not by specific routing inside their network. There's nothing artificial about that. Can it be upgraded to provide far more capacity? Yes. Should it be? Yes. Is Verizon doing it? No.

As I said, very small difference, but potentially a very important one should this go to court.

Re:Connect with a VPN (3, Insightful)

Talderas (1212466) | about 3 months ago | (#47482629)

What you're doing is using a VPN connections which has an different inbound interconnect than the one which the majority of Netflix traffic comes in on. Verizon is 100% correct. There's no throttling going on because the connection that is being "throttled" as you put it is actually at 100% utilization and congested causing dropped packets and bad performance.

In Verizon's defense (4, Funny)

NotDrWho (3543773) | about 3 months ago | (#47482049)

Netflix has *yet* to pull up a dump-truck full of money to Verizon HQ.

Re:In Verizon's defense (4, Informative)

_xeno_ (155264) | about 3 months ago | (#47482097)

Actually, they did [bloomberg.com] . Verizon has just yet to deliver. Apparently they don't expect to deliver until the end of the year [arstechnica.com] in any case.

Re:In Verizon's defense (5, Funny)

Taeolas (523275) | about 3 months ago | (#47482125)

Actually, they did [bloomberg.com] . Verizon has just yet to deliver. Apparently they don't expect to deliver until the end of the year [arstechnica.com] in any case.

Which this article seems to implies it takes Verizon a year to send a technician to 7 cities to connect up a few cables between routers. (And / or maybe install a couple of cards). Maybe Verizon should stop having their techs travel by horseback, they might get it done faster.

Re:In Verizon's defense (2)

arbiterxero (952505) | about 3 months ago | (#47482223)

Horseback?

From Verizon, I'd be surprised if they were even given shoes.

End Of the Year is a schedule determined carefully by verizon so that Verizon can figure out how to offer their own competing service "Without the problems of netflix!" ....or possibly because Verizon has to force netflix into a complex solution to hide the fact that it's a simple problem.

Re:In Verizon's defense (2, Funny)

Anonymous Coward | about 3 months ago | (#47482265)

Maybe Verizon should stop having their techs travel by horseback, they might get it done faster.

Gotta support those buggy whip manufacturers.... ;-)

Re:In Verizon's defense (4, Funny)

jythie (914043) | about 3 months ago | (#47482123)

I wonder how many pennies would fit in a dumptruck....

Re:In Verizon's defense (0)

Anonymous Coward | about 3 months ago | (#47482179)

Dump truck delivery of the pennies is too slow. I prefer the shotgun barrel delivery method, preferably aimed at Verizon executive's heads.

Re:In Verizon's defense (1)

Hillgiant (916436) | about 3 months ago | (#47482443)

Never underestimate the bandwidth of a trebuchet loaded with a dumptruck full of pennies.

We are talking about bandwidth, right?

Re:In Verizon's defense (4, Informative)

Hillgiant (916436) | about 3 months ago | (#47482429)

http://www.wolframalpha.com/in... [wolframalpha.com]

$260,000. I'm sure Verizon loses that in the couch cushions every other month.

Re:In Verizon's defense (1)

Jason Levine (196982) | about 3 months ago | (#47482661)

After some Google searches, it looks like a dump truck would be about 27 cubic yards [yahoo.com] . A penny is about 0.44 cubic cm [yahoo.com] . This gives us about 46,900,000 pennies in a dump truck (rounding down to the nearest hundred thousand) or $469,000 worth.

Re:In Verizon's defense (1)

Anonymous Coward | about 3 months ago | (#47482321)

Defending an ISP?

Tool.

I disagree (2, Interesting)

Dishwasha (125561) | about 3 months ago | (#47482093)

We all know most top tier network providers are running over multiple bands of fiber just sitting there idle. What Verizon is saying is Level 3 has not worked out an agreement with Verizon to upgrade capacity. The physical part is the easy part; it's just about upgrading port usage. Now, if Level3 is paying for X bandwidth and they're not getting X bandwidth because Verizon hasn't upgraded their equipment, I'm sure Level3's lawyers would be all over that.

Re:I disagree (3, Insightful)

putaro (235078) | about 3 months ago | (#47482215)

Level 3 doesn't pay Comcast for bandwidth. Why should they? Comcast customers have already paid Comcast for the links to their house and they're the ones pulling data from Level 3. Level 3's customers pay Level 3 to deliver to the edge of their network. As the Level 3 post points out, the cost for Verizon to add more bandwidth between the Level 3 network and the Verizon network is minimal.

Re:I disagree (2)

arbiterxero (952505) | about 3 months ago | (#47482247)

This is a peering agreement, not a service agreement. Even if it were a service agreement, then it should be Verizon paying Level 3 so that verizon's customers can access the content they want.

Peering agreements don't usually pay each other because both networks gain advantage from the peer-connection.

Re:I disagree (4, Insightful)

SighKoPath (956085) | about 3 months ago | (#47482345)

Part of the issue is that Verizon is a last-mile network, and does not sell symmetric bandwidth to its subscribers. So, the typical agreement between providers - where they each send about the same amount of traffic to each other and upgrade the interconnects to handle that traffic - will not work between Verizon and Level 3. Verizon (and the vast majority of other last-mile providers, including Comcast) will NEVER have a balanced interconnection with Level 3, because the home subscribers can all download far faster than they can upload.

Really, it's Verizon's customers who are causing all this bandwidth usage, so it should be Verizon ensuring that their interconnects can handle the requested bandwidth. If anything, Verizon (and Comcast) should be paying Level 3 for additional download capacity... but we all know that is never going to happen.

Re:I disagree (1)

Bill, Shooter of Bul (629286) | about 3 months ago | (#47482389)

Yeah, that would make sense. Verison's blog just had a small hidden reference to that. Its not immideatly obvious, probably because thats a difficult thing to explain to people not versed in peering agreements.

Re:I disagree (-1, Troll)

Charliemopps (1157495) | about 3 months ago | (#47482413)

More specifically this is what's going on:

Level3 to Netflix: "If you get a 10gig trunk through us and only us we'll give you half off!"
Netflix to Level3: "SOLD!!! Hell yea!"
Verizon to level3: "Our traffic from netflix moved over to Level3 last night... very strange, anyways we need to increase our capacity..."
Level3 to Verizon: "Ok, that will be $X"
Verizon to level3: "um... That's 300% higher than any other provider out there..."
Level3 to Verizon: "suck it... your monies are belong to us"
Verizon to Netflix: "We need to talk about peering. Level3 is totally trying to rip us off."
Netflix to Verizon: "No thanks, we got a great deal. By the way, Level3 sends their regards"
Verizon to Netflix: "ok, we're not going to increase capacity with Level3, if you want to serve your customers properly, you're going to have to buy a truck from a more affordable peer. We can even offer you a deal on our network if you'd like."
Netflix to the world: "Net Neutrality! People unite!"

Re:I disagree (0)

Anonymous Coward | about 3 months ago | (#47482585)

We all know most top tier network providers are running over multiple bands of fiber just sitting there idle.

There are a number of times where "we" are overly simplistic, or just plain wrong. Especially long haul (or even just outside the door of the hubsite). First, most providers target a no more than a 50% utilization on a link to minimize the issues of bursts of traffic (and lost packets), and to insure that if a link is cut (and cuts are more common than anyone would like), when the traffic fails over to another the traffic will "fit" in the remaining capacity. Second, while in reality, the large ISPs own (or have IRUs) a lot of fibre, a lot of that fibre is unlit, and lighting it up for long haul usage is not always just plugging in another router optic port due to the engineering to insure that other links can handle the (failover) capacity, and to add in the amplifiers along the route to get the signal to the other side, and sometimes installing the (very) expensive DWDM gear to light it (which all of the big boys do). Network engineering at ISP scale is not always as simple as plugging in another port.

Blog post gone? (1)

dbrueck (1872018) | about 3 months ago | (#47482143)

L3's blog still has a summary blurb, but the link to the actual post gets a 404 - did they take it down or did they just link it wrong? Anybody have a cached copy?

Re:Blog post gone? (0)

Dega704 (1454673) | about 3 months ago | (#47482193)

I'm seeing the same thing. Maybe Verizon realized their screw-up and took it down. Hello Streisand Effect.

Re:Blog post gone? (2)

Anonymous Coward | about 3 months ago | (#47482337)

Its L3's blog, not Verizon's.

Re:Blog post gone? (0)

Anonymous Coward | about 3 months ago | (#47482225)

Google still has it cached.

Re:Blog post gone? (0)

Anonymous Coward | about 3 months ago | (#47482241)

Google cache for the level3 post: http://webcache.googleusercontent.com/search?q=cache:blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/

Re:Blog post gone? (1)

Anonymous Coward | about 3 months ago | (#47482243)

This link is working for now...

http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/?utm_source=dlvr.it&utm_medium=tumblr

Re:Blog post gone? (0)

Anonymous Coward | about 3 months ago | (#47482255)

Google has it cached:
http://webcache.googleusercontent.com/search?q=cache:DBHDyx7n4D0J:blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/

Re:Blog post gone? (0)

Anonymous Coward | about 3 months ago | (#47482423)

(Blog) posts that have not been "reviewed" by Corporate Consul and/or appropriate C level staff that points out that a large customer's emperor is naked tend to get disappeared. Sometimes the poster ends up leaving to explorer other career opportunities.

I will also point out that the post was a bit flippant, and that while the optics costs may only be a few thousand, the router cards (depending on what particular router Verizon has in the hubsite) may be tens, or even hundreds of thousands of dollars (usually multiple ports, but still not just a few thousand). And for each additional "input" port, there has to be a "output" port on the other side, and then the downstream router needs a "input" link... and so on and so (and if you have to pay for fibre to be trench to get outside the hubsite to some location, you are looking at many many millions). So the investment may not be quite so minimal as the post suggests.

Answer needed (-1, Troll)

Kohath (38547) | about 3 months ago | (#47482145)

Why should Verizon do the upgrade? Why would they want to? To make Level 3 happy? To make Netflix happy? What is their incentive?

The only answers I've seen for this on Slashdot are:
- the government should threaten Verizon and force them to operate the network contrary to Verizon's best interests,
- the government should seize Verizon's network
- no answer, just crying about how you're entitled to better Netflix video quality

Got anything better?

Re:Answer needed (1)

Anonymous Coward | about 3 months ago | (#47482165)

How about because Netflix paid them to do it [bloomberg.com] ?

Stop whining about things we have a legitimate reason to bitch about!

Re:Answer needed (4, Insightful)

ledow (319597) | about 3 months ago | (#47482199)

Customers.

You're paying for a service, and nowhere does it say that they will discriminate against a particular service, such as Netflix.

It's obstructive business, against your customer's best interests, for no particular reason. It will also violate any given "net neutrality" laws that are / may come into effect.

Those laws are the answers. The reason for their existence is this sort of unnecessary posturing. And governments make companies do a lot of things against a company's best interests - all the time. It would be in the company's best interest to not pay tax, screw over its customers, not ship goods that have been paid for, be monopolistic, collude with others to enforce market prices, etc. The laws are brought in to stop that shit in the PEOPLE'S best interest, not the company's.

Not saying it's anywhere near perfect, but your post seems to want to back a corporation screwing over its customers and then (falsely) blaming its competitors and random third-party companies for that.

Re:Answer needed (1)

Kohath (38547) | about 3 months ago | (#47482325)

Customers.

They seem to have very limited interest in what their customers want for Netflix streaming quality. What is their incentive to care?

Those laws are the answers.

I covered that with "the government should threaten Verizon and force them to operate the network contrary to Verizon's best interests".

"I want it and my government friends have guns..." Is this the best we can do?

Re:Answer needed (4, Insightful)

chihowa (366380) | about 3 months ago | (#47482513)

"I want it and my government friends have guns..." Is this the best we can do?

The reason Verizon can stay in business despite having "very limited interest in what their customers want" is because of municipal and state granted monopolies, federal grants and subsidies, and the reason they even exist at all is because of a government approved corporate charter. Why is "government friends with guns" an acceptable argument for them getting their way, but not an acceptable argument against it?

Re:Answer needed (0)

Kohath (38547) | about 3 months ago | (#47482651)

The reason Verizon can stay in business despite having "very limited interest in what their customers want" is because of municipal and state granted monopolies...

I know. So a different answer might be to break up the monopolies and tell local governments that they can't make long term monopoly deals any more.

Why is "government friends with guns" an acceptable argument for them getting their way, but not an acceptable argument against it?

It's not good in either case. We should head in the other direction.

Verizon can afford more government friends than you can. Do you honestly foresee a time when they won't? If not, maybe you shouldn't want things to be decided based on who has more government friends?

Re:Answer needed (5, Insightful)

doug (926) | about 3 months ago | (#47482217)

Sure. The content streaming from Netflix has been requested by Verizion customers. They've paid for access to the internet, which includes Netflix. They are the ones being throttled. Basically Verizon is trying to double dip here - get money from regular customers plus shaking down more from content providers. If Verizon really cannot handle the flood of Netflix content, shouldn't they raise the cost to the consumers to build out the Verizon network?

Re:Answer needed (2, Informative)

Anonymous Coward | about 3 months ago | (#47482257)

...because maybe Verizon's paying customers want it and expect to get what they pay for?

I know. Serving your customers. Totally alien concept.

Re:Answer needed (3, Insightful)

Wookact (2804191) | about 3 months ago | (#47482295)

Sure, the governments should break up the ISP monopolies that restrict access to only one or two ISPs. They should also try to reduce any barriers to entry in the market for new competitors to spring up. This would increase competition and force verizon to better their service to retain their customers.

The issue with this is that its good for consumers but bad for investors, and we all know who our esteemed congress men actually represent.

Re:Answer needed (1)

Kohath (38547) | about 3 months ago | (#47482355)

This is a different answer. Thanks.

Re:Answer needed (2)

Kohath (38547) | about 3 months ago | (#47482489)

Perhaps a rule where cable or satellite TV providers are prohibited from operating centralized peering points. If Verizon had to buy their bandwidth from upstream providers, they wouldn't be able to choke L3. And L3 would have to bid against Verizon's upstream providers to get Netflix's business.

Essentially, less economic centralization in the network infrastructure would provide for more opportunities for competitive bidding all along the chain. Everyone would end up with more customer-focused incentives.

Re:Answer needed (1)

arbiterxero (952505) | about 3 months ago | (#47482301)

because Verizon is being paid by their customers to access this data.

Why should verizon have any peering agreements with anyone?

Why not just operate their own network that doesn't peer out to the internet as a whole?

As a customer of an ISP, I'm not paying for the ISP to deliver me to their own internal network. I'm paying them to deliver my data to the peered connections. ALWAYS.

If my ISP has one peered connection that's beyond capacity, I expect them to upgrade it, as it's the cost of doing business.

And especially when it's cheap. This is chump change to them, and Level3 even offered to pay for it..... why is verizon saying no?

Re:Answer needed (1)

Bill, Shooter of Bul (629286) | about 3 months ago | (#47482485)

The traffic isn't uniform in both directions. When it is, its an easy agreement. When its not, someone usually pays. Level3 offered to pay for the hardware to make additional connections.

But thats kind of like me telling verision that I'll pay for the router and won't charge them for using their network and expecting them to provide me with free service. There isn't enough of an incentive to allow them to agree to my free internet for Bill proposal. Same thing for most networks with unequal traffic patterns. Netflix is so big that any network they buy transit from, will cause their data flows to become unbalanced.

Re:Answer needed (1)

hAckz0r (989977) | about 3 months ago | (#47482311)

How about because customers are paying them for Internet service, and going to Netflix is where they want to go? People don't want to pay for bad service do they? The real problem here is Verizon is a competitor to Netflix, and Verizon is not only being allowed to be anti-competitive but also hoping to get paid for it.

Re:Answer needed (1)

berashith (222128) | about 3 months ago | (#47482347)

to make their customers happy. that is it. The customers are purchasing something, then finding out that they cant get what they thought they paid for. When customers find out that they can get some things at the speeds that they paid for , but not others, then it should be their ISPs job to provide that service. If there was true competition then this would happen quickly. Verizon doesnt really compete, so they can play these bullshit games.

Lets reverse your question. Why should people not be entitled to better netflix quality? Netflix provides enough infrastructure. Verizon claims to not have congestion issues WITHIN their network. People have PAID for a level of delivery that they arent getting. Your word entitled says a lot. this isnt a favor, this is what has been paid for.

Now lets play stupid analogy. If the only way to deliver a package was UPS, and you were forced to pay for 2 day shipping on everything, but it took a month to get the delivery complete, wouldnt you be pissed? I would. I paid for a service, and got a fraction of it, I was ENTITLED to the service, as our contract doesnt say 2 day delivery unless the delivery company just isnt in the mood, and in that case fuck you.

Re:Answer needed (1)

silas_moeckel (234313) | about 3 months ago | (#47482367)

The government should threaten them because they are not providing the service their customers have paid for and they advertised. The Internet works because tier 1's all connect to each other and upgrade ports as they get saturated (good ISP's are proactive, ok ones last minute and crappy ones only after the link is saturated). While this is related to net neutrality it's primarily a false advertising issue, if they change their wording to correct that it's a DPUC (or similar) issue as they are no longer providing internet access per their monopoly contract.

Re:Answer needed (5, Insightful)

bill_mcgonigle (4333) | about 3 months ago | (#47482433)

Got anything better?

Remove the laws and regulations holding back community fiber projects.

Re:Answer needed (1)

bleh-of-the-huns (17740) | about 3 months ago | (#47482455)

How about to make their fucking customers happy. I pay Verizon (because my only other choice is Comcast, and I hate them more). I request a service, I expect my provider to give me access to this service. Netflix pays L3, L3 is their service provider. Service providers peer, that is the way the internet has always worked.

Verizon's Response (5, Funny)

CanadianRealist (1258974) | about 3 months ago | (#47482157)

Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more

Verizon's response was "Ok, but these cards tend to wear out pretty quickly so we'll need you to pay that amount each month. 5,000 streams may sound like a lot, but they don't last very long. A person watches a few movies a week, maybe a couple of youtube videos per day, that's like 20 streams in one week, and that's only one customer. Before you know it, you've used up all 5,000 of those streams and the card needs to be replaced."

"Oh yeah, and if it's coming from Netflix then we're using twice as many streams. We use one stream from Netflix to us, then another stream from us to our customers. Maybe you should really pay us that amount every week."

Re:Verizon's Response (0)

StratiKat 虎 (3754083) | about 3 months ago | (#47482435)

if (ignorant) { Obviously the 5000 streams are at any one instance, not how many in total they can support in their lifetime. The lifespan of the cards are not measured by stream count (which is arbitrary anyone because a stream has no predetermined length of time). } else if (Sarcastic || facetious) { Not very apparent - seems confusing. Clarification would be helpful. }

It's not just Netflix that is suffering though (2)

bleh-of-the-huns (17740) | about 3 months ago | (#47482159)

I have FIOS... Yes my Netflix performance is piss poor, but so are the connections to other services that just happen to use the same transit providers as Netflix.

Particularly the VPS providers that I was using (I just switched due to the latency). I have 2 VPS providers, 1 in Reston, 1 in the UK. The one in Reston is just down the street from Verizons datacenter (used to be UUNET), but the provider to the VPS company I use was Cogent, heavy latency right at the peering point.

Of course, Verizon likes to blame Netflix for picking crappy transit providers, but had it been Company XYZ instead of L3 and Cogent, Verizon would have done the exact same thing to XYZ and let the peers saturate.

I did manage to switch to a different VPS that does not use Cogent or L3, and I have consistent low transit times, which I use as a VPN endpoint. Seems to do the trick (I have been doing this long before any people started publicizing using VPN's to get around Verizon and Comcasts shenanigans, mostly to keep Verizons prying eyes from monetizing my internet behavior, not to keep gov spying eyes out. If VZ wants to pay me [no, not give me a discount on already overpriced service, but give me cold hard cash] for my browsing and internet habits, then I will more than be happy to let them snoop)

Salesmen vs Engineers (4, Funny)

minstrelmike (1602771) | about 3 months ago | (#47482163)

What happened was a bunch of salesmen and marketers at Verizon asked how they could explain the network throttling.
They obviously didn't understand the presentation so they assumed no one else would either.

Verizon criminals (-1)

Anonymous Coward | about 3 months ago | (#47482169)

First, this is a violation of the Telecommunications Act of 1996 [wikipedia.org] .

Secondly, we are seeing the rip off of the US taxpayer. And I think Verizon should pay ALL of the taxes they should have paid: [pbs.org]

Over the decade from 1994-2004 the major telephone companies profited from higher phone rates paid by all of us, accelerated depreciation on their networks, and direct tax credits an average of $2,000 per subscriber for which the companies delivered precisely nothing in terms of service to customers. That's $200 billion with nothing to be shown for it.

And thanks to the Hobby Lobby ruling (pierced the corporate veil), it looks like that the CEO should go to jail for tax evasion.

L3 blog post that has now disappeared (2, Informative)

Anonymous Coward | about 3 months ago | (#47482175)

Verizon’s Accidental Mea Culpa Mark Taylor / 17 hours ago

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s own paying broadband consumers.

His explanation for Netflix’s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know it’s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But here’s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon’s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that’s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3’s network interconnects with Verizon’s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So let’s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.

lvltvzw

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they’ve run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it.

But, here’s the other interesting thing also shown in the Verizon diagram. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? Could it be that Verizon does not want its customers to actually use the higher-speed services it sells to them? Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitors’ costs?

To summarize: All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.

Why might that be? Maybe we should ask David Young.

Re:L3 blog post that has now disappeared (2)

greenreaper (205818) | about 3 months ago | (#47482273)

Here's a Google cache, including the diagram: http://webcache.googleusercont... [googleusercontent.com]

I knew this for years (0)

Anonymous Coward | about 3 months ago | (#47482181)

My company has Verizon and Cogent (and others). Our cogent offices have an absolute terrible time maintaining any type of stable throughput to our midwest data center running on Verizon. Often times we get speeds of several KB/sec for hours on end between them over our 100/200mbit connections. I've called Cogent and Verizon and complained for years. The bottleneck is the handoff from Cogent to Verizon. Verizon will not budge and claims no responsibility.
  Interesting tidbit is the average speed for the last year shows Verizon --> Cogent is about 2x-10x the speed compared to Cogent --> Verizon on the same circuits. Verizon must not like to take on incoming Cogent data.

Help me understand (1)

Mycroft-X (11435) | about 3 months ago | (#47482197)

So maybe someone can explain this to me because I don't entirely get it.

Right now Level 3 doesn't pay Verizon any additional money for the data being sent its way (yes, requested by Verizon customers, but transport is usually paid by the shipper -- when I order a physical product I pay for shipping to the vendor, who pays the transporter).

The reason Level 3 doesn't pay any more is because they are using settlement-free links established to provide basic bi-directional communication between the two networks. Because of the way they are using them, these links (which are set up to provide balanced access) are saturated in one direction while only 30-60% utilized in the other direction.

The point made by both companies is that fixing the congestion is a simple matter of hooking up a couple ports (which would increase the utilization of Verizon's network).

Level 3 wants Verizon to agree to expand the settlement free ports to allow for the imbalance of traffic. Verizon says "our settlement free ports are sufficient for normal traffic, and if you want to avoid congestion for the additional traffic you are charging Netflix to carry then you're going to need to purchase additional ports and pay for that traffic."

Neither wants to budge and so they fight a PR war about it. Level 3 says "It's just a couple ports and a little cable" while disregarding the downstream impact on Verizon's network. Verizon says "Level 3 is taking undue advantage of our mutually beneficial arrangement and wants us to help them do it for free."

Is this accurate?

Re:Help me understand (1)

drinkypoo (153816) | about 3 months ago | (#47482281)

Because of the way they are using them, these links (which are set up to provide balanced access) are saturated in one direction while only 30-60% utilized in the other direction.

That's true, but you have the responsibility backwards. The peers that download and don't upload are the problem. That's Verizon, not Netflix.

Re:Help me understand (1)

Shados (741919) | about 3 months ago | (#47482339)

Netflix pays level 3 to get their bits from their servers all the way to the edge. Customers pay to get bits from the edge all the way to their house. Level 3 and Verizon make an agreement for the parts where those 2 networks touch each other.

Verizon is saying: "The direction of bits matter. Because our customers are paying to receive bits and not to send bits, YOU owe US more money. If our customers paid to SEND that much bit instead of receiving them, then we'd owe you money....like if it changes anything on the network".

Thats bull. Customers pay for a certain amount of bits coming from any edge to their house. Who cares exactly where it came from?

Re:Help me understand (0)

Anonymous Coward | about 3 months ago | (#47482341)

transport is usually paid by the shipper -- when I order a physical product I pay for shipping to the vendor, who pays the transporter

But the Internet is not something that you just dump something on. It's not a big truck. It's a series of tubes.

You pay your ISP for your service. Whoever you're connecting to pays their ISP for their service. That's the way we've always done it.
The only ones who want to change this system are greedy ISPs.

Assuming Verizon were the end-user.... (0)

Anonymous Coward | about 3 months ago | (#47482363)

Actually it's more like a company paying for x telephone circuits to their PBX, only Verizon made most of them 'party lines', so even though you can connect you're now having to talk and listen over both your fellow employees as well as their customer's replies in order to get anything done. However if you do a 'long distance' callout via calling card then call from there to your customer you get a perfectly free and clear line instead of a 'party line' :)

I hope this makes for a relatively accurate, concise, and entertaining telephone analogy.

Help me understand (0)

Anonymous Coward | about 3 months ago | (#47482459)

It seems more like:
L3 provides bulk backbone data haulage to its customers, which they pay for. It provides the resources to deliver them to their destination. It will hand over packets to whichever ISP provides service to the packets' destination. All this, it does.

Verizon is supposed to provide a certain bandwidth to its customers, which they pay for. These are the end users. It does provide a link meeting bandwidth requirements between the customer and Verizon's internal network, which includes some services like Verizon's pay TV. However by failing to upgrade an at-capacity interconnect for which it is responsible it fails to provide the full bandwidth to their subscribers when they visit certain sites beyond Verizon's internal network, such as Netflix.

L3 appear to be doing all they are contracted to - delivering data to its destination ISP - and at a high bandwidth.

Verizon do appear to be falling short of their obligations as they have contracted to provide consumers a package based on provision of a certain bandwidth, but are failing to meet the terms of that contract at least for data from some parts of the internet by failing to upgrade parts of their infrastructure. The fact that this would increase Verizon's network utilisation is neither here nor there - if that is the cost of providing their clients with their contracted bandwidth then that is what they are bound to do.

Level(3) Page now returning 404 (1)

Trevers (143804) | about 3 months ago | (#47482211)

Is this a resurrection of the Slashdot effect? Or did someone higher up at Level(3) notice this and pull the blog post?

Solution (0)

Anonymous Coward | about 3 months ago | (#47482221)

Verizon seems to be saying that L3 needs to pay more because the bandwidth flowing into Verizon's network far outweighs the bandwidth flowing into L3's. While this is fundamentally idiotic for a last-mile ISP for any reason than money grabbing due to the asymmetric up/down bandwidth offered to last-mile customers, let's play their game. It sounds like all we need to do is send an equivalent amount of traffic back to Netflix, and everything will be right in the world. Even better, if we send more traffic to Netflix than we receive, L3 ought to be able to get money from Verizon because that traffic is out of balance in the other direction now.

Level 3 blog post unavailable (1)

strangeintp (892348) | about 3 months ago | (#47482305)

Just tried going to the link, it's been 404'ed.

Original article is 404 (3, Informative)

fok (449027) | about 3 months ago | (#47482307)

Here is a copy of the text, just in case:

Verizon's Accidental Mea Culpa
Mark Taylor / 23 hours ago

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix's customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon's own paying broadband consumers.

His explanation for Netflix's on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know it's bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post , I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But here's the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon's depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that's what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3's network interconnects with Verizon's in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So let's look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.

lvltvzw

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested â" in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we've been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can't afford a new port card because they've run out â" even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it.

But, here's the other interesting thing also shown in the Verizon diagram. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? Could it be that Verizon does not want its customers to actually use the higher-speed services it sells to them? Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitors' costs?

To summarize: All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.

Why might that be? Maybe we should ask David Young.

Re:Original article is 404 (1)

EvilGrin5000 (951851) | about 3 months ago | (#47482335)

Ack, didn't see your comment until after I had replied above: Here's a cache link so you can see the image/diagram : http://webcache.googleusercont... [googleusercontent.com]

ugh (1, Interesting)

Charliemopps (1157495) | about 3 months ago | (#47482315)

I get tired of being the only person on slashdot that understands this...

The problem isn't the interconnect. The problem is between the local remote that feeds your house and the Central Office. When that much data comes from Netflix all at the same time, the remotes trunks can't handle it. Upgrading THOSE trunks costs a fortune. Throttling netflix at the peer reduces load on those trunks without affecting other services. That's what's going on and why Verizon (and others) are throttling Netflix. They have no other way of targeting netflix traffic directly without sending the FCC into a tizzy.

Netflix-------> Level3-------> Verizon core network-------> Verizon local CO----(the problem is here)---> remote-------> your house

You can argue that Verizon should fix that trunking themselves, we'd have a different argument then... but what this "Story" is about isn't even what's wrong. You can't just look at one section of the cheapest part of Verizons network and claim how easy it would be for them to fix. They've got a huge multi-billion dollar network to maintain and the front door to that network is the least of their concerns.

Re:ugh (4, Interesting)

Shados (741919) | about 3 months ago | (#47482387)

So when Netflix decided to pay Comcast, they were able to upgrade all of those remote trunks in ~24 hours, even though they cost of fortune?

Victory for Verizon! Now they'll charge more (1)

AnontheDestroyer (3500983) | about 3 months ago | (#47482379)

Dear Verizon Customers,

We were recently told we needed ~$100,000 worth of equipment. In order to pay for this we're going to need to add an interconnect service fee of $5/month to your bill. Sorry. Try a competitor? Hah, suck it.

Thanks,
Verizon CEO

PS We have also sued the company who told us this for saying it publicly, and we will most likely charge them more money, too.

Level3 BLOG post unblocked (sort of) (0)

Anonymous Coward | about 3 months ago | (#47482383)

The link to the LEVEL3 Blog post has been blocked...but modifying the link a bit [level3.com] pulls it back up again. Blog post reproduced below (with relevant image here [level3.com] )

VerizonÃ(TM)s Accidental Mea Culpa
Mark Taylor / 23 hours ago

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns NetflixÃ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by VerizonÃ(TM)s own paying broadband consumers.

His explanation for NetflixÃ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know itÃ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But hereÃ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as VerizonÃ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatÃ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network Ã" IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3Ã(TM)s network interconnects with VerizonÃ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So letÃ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.

Network diagram [level3.com]

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested Ã" in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something weÃ(TM)ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they canÃ(TM)t afford a new port card because theyÃ(TM)ve run out Ã" even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If thatÃ(TM)s the case, weÃ(TM)ll buy one for them. Maybe they canÃ(TM)t afford the small piece of cable between our two ports. If thatÃ(TM)s the case, weÃ(TM)ll provide it. Heck, weÃ(TM)ll even install it.

But, hereÃ(TM)s the other interesting thing also shown in the Verizon diagram. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? Could it be that Verizon does not want its customers to actually use the higher-speed services it sells to them? Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitorsÃ(TM) costs?

To summarize: All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.

Why might that be? Maybe we should ask David Young.

it wont matter... (0)

Anonymous Coward | about 3 months ago | (#47482515)

the fcc will soon rule in favor of big multi-service providers, such as verizon and comcast, and allow the decimation of the hope of network neutrality so that they can stifle online-based competition in favor of their own video services.

Robber Barons guarding the bridge (1)

SlashDread (38969) | about 3 months ago | (#47482537)

Thou shall not pass

Confusion (0)

Anonymous Coward | about 3 months ago | (#47482633)

I am a bit confused as to how this all works, perhaps someone could clear things up for me.

Why should Level3 pay Verizon for access to their networks, and not Verizon pay Level3 for access?

In this case, Level3 is the supplier of the goods (Netflix data), Verizon and its customers are the consumers of the goods.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?