Announcing: Slashdot Deals - Explore geek apps, games, gadgets and more. (what is this?)

Thank you!

We are sorry to see you leave - Beta is different and we value the time you took to try it out. Before you decide to go, please take a look at some value-adds for Beta and learn more about it. Thank you for reading Slashdot, and for making the site better!



Comcast Kisses-Up To Obama, Publicly Agrees On Net Neutrality

Drakonblayde Private Links != Paid Priority (258 comments)

Full Disclosure: I am a network ops engineer for Comcast.

Anyone who believes that buying private links into a providers network is the same as your traffic getting paid priority knows jack shit about network ops. In the case of Comcast, Netflix traffic gets no special priority once it's on the internal network. The direct links simply lets them bypass the naturally occurring bottlenecks that occur at internet peering points.

Now I'm sure a bunch of people (who are not network engineers) are going to argue over the wording and philosophy as to whether or not buying paid links into a providers network constitutes priority or not. It's not. In network operations, priority is a very specific concept. It means that you treat one class of traffic better than others, usually to the detriment of other classes of traffic. As an example, e911 voice traffic has the *highest* priority on the Comcast network.

Comcast does not treat Netflix traffic any better than anyone else's traffic. Nor is it treated any worse. It is forwarded as Best Effort within the Comcast network.

The only difference that buying direct links in meant was that they got to skip the congestion in the peering points. Comcast has alot more bandwidth internally and once traffic makes it into the network, congestion is not usually a problem (things do break, redundant links become saturated, etc. It's a big network, but in normal operation mode, congestion doesn't exist). What little prioritization we do has alot more to do with latency than with congestion (ie, your phone call is more important than your massive porn transfer, since voice is alot more sensitive to delay than bulk data transfer).

about two weeks ago

ISPs Violating Net Neutrality To Block Encryption

Drakonblayde I think the part that scares me.... (149 comments)

Is that techdirt did virtually no research on the issue, they just passed along what Golden Frog said in their filing.

Which brings me to the *really* scary part.

A company which provides VPN service should reasonably expect to have a clue when it comes to network operations.

Not only did this company not have the chops to figure out that 'someone may have incorrectly configured a firewall!', oh no. They decided to compound their inadequacy by including it in a filing to the god damn FCC.

So many levels of failure involved in this.

about a month and a half ago

Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling

Drakonblayde Re: Alternative explanation (398 comments)

Actually, when Sub2 joins, his system can start buffering the show 10-minutes in, and while doing so, send a request to resend the first 10 minutes of the video. Netflix servers save bandwidth, and both users watch the video they wanted, when they wanted.

That's not how Multicast works. The source has absolutely no idea who the members of the stream are. Only the edge routers know that. IGMP joins do not get forwarded up the multicast tree, only PIM joins. There would have to be some additional intelligence built into the app that would tell the stream to start adding additional data for the buffered client to the stream. And then you *still* have the problem that all that additional buffering data? Gets delivered to *EVERY SINGLE MEMBER OF THE STREAM*. So you end up saturating and transiting data that not everyone needs. This is a large part of the reason why no one uses PIM Dense mode, the way it works, alot of unnecessary traffic gets flooded.

So sure you could have the late joining client join another stream to get the first 10 minutes they missed, staying in the old stream while they buffer so that when they're caught up they can just drop the catchup stream and stay in the original multicast. Or you could unicast the first 10 minutes to them.

Or, you could just unicast the entire damn thing, which is alot less complex operationally, and doesn't require your upstream providers and their customers to essentially reconfigure their entire networks.

Now, that's just the engineering side. Let's look at the business side. Doing that would require effort from software developers, network engineers, and network operators. All to improve Netflix's experience. For the providers that Netflix's customer base connects to, there's no incentive, especially since they likely already have complex multicast implemetations of their own (which will usually be the case for any provider that hosts video). They will expend alot of man hours, possibly some opex and capex for absolutely no return. If they're publicly traded, this will make shareholders unhappy.

For the transit providers, they have even less incentive to make this go. If it did decrease Netflix's bandwidth usage, it would directly effect their revenue, unless Netflix was stupid enough to agree to pay the same amount of money for less bandwidth used.

about 4 months ago

Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling

Drakonblayde Re:Alternative explanation (398 comments)

I suspect that whats going on is that Netflix put the majority of their traffic on Level3 and Level3 is trying to charge Verizon an exorbitant rate for enough bandwidth to handle that peer. Verizon said "No" and told Netflix to go with another peer. So Verizon has plenty of bandwidth, Netflix has plenty of bandwidth... it's where those peers are located that's the problem.

Ok, but you're wrong.

Level3 has admitted they have settlement free peering with Verizon. Level3 does not pay Verizon anything. Verizon does not pay Level3 anything.

Netflix pays Level3. This is why Level3 gives a shit about this situation.

What's going on is that Verizon is trying to cut out the middleman. Verizon wants Netflix to pay them to get traffic into their network instead of paying Level3 to deliver traffic into the Verizon network. Why? Because they don't make any money from Level3.

Naturally, Level3 is all in a huff about Verizon trying to fuck with their revenue stream.

about 4 months ago

Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling

Drakonblayde Re: Alternative explanation (398 comments)

There are at least three underlying problems for the congestion issue - one is the DMCA and related copyright laws that prevent any sort of sane caching, the general fear of multicast that everybody on the Internet still seems to have (half a million unicast streams of the same show is insane - where are the global warming people on this?), and the grants of monopolies and/or prohibitions on competition that prevent local competition.

I agree with you that half a million unicast streams can be nuts, when it comes to on demand content, multicast is a non-starter (we'll ignore the fact that multicast is sorely lacking in security features and would require some serious re-engineering on many networks to work... there's a very big reason why inter-domain multicast routing is not seriously employed, and it has nothing to do with fear).

Think about how this traffic flows -

Sub1 wants show A and starts playing it on Netflix.

10 minutes later, Sub2 wants show A as well.

What happens if this stream is multicast? Well, Sub2 gets the show 10 minutes later, assuming he's joining the same multicast group as Sub1. Sub2 is not happy, he wanted to watch the entire show.

How to get around this? Well, ok, so Netflix could just mux the feed for Sub2 inside the feed for Sub1, and presumably, the client would be able to tell which parts of the feed were for which sub. However, the problem is that the data for Sub2 would still be delivered to sub1, sub1 would just throw it away and pay attention to it's own data. However the data for both feeds have to transit the same backbone, which drives capacity usage up. This is also unsustainable, as eventually, as more subscribers joined, the feed would grow so large that it would saturate the downstream of all the subscribers receiving it and eventually lead to packet loss.... which would lead to loss of video, stuttering, etc. All the same shit thats going on now.

Ok, so muxing different streams into the same feed for the same show isn't going to work.

So Sub2 could just start getting the feed over a different multicast group, that would solve the ever growing feed problem!

Except that if you do that, there is functionally no difference between sending the traffic to a multicast destination and a unicast destination.

So sure, while sending half a billion unicast streams seems insane, especially since alot of those will be watching the same show, the fact is that a very small percentage are going to be watching the same show at the exact point in the show. For the most part, they will all be at different parts. Multicast is a solution when the data for the given event is live, or when it's linear (ie, this is the point we're at, and there's no going back). On Demand services does not fit either of those profiles.

Multicast has it's place in the on demand world, but only on the backend. It's wonderful for things like distributing a new asset to all of the streaming sources, or for filling caching servers that will the streaming boxes will need to pull from, but multicast is simply not a workable or superior implementation when it comes to delivery of content to subscribers.

about 4 months ago

Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling

Drakonblayde Re: Alternative explanation (398 comments)

You do not do networking for a living very well then.

The problem isn't how Verizon is egressing traffic to Netflix, it's in how Verizon is allowing traffic from Netflix to ingress to their network. Once the Netflix traffic makes it into their network, I have no doubt it is being delivered over the most efficient route.... which means jack shit when the ingress point is such a tight bottleneck. Inbound traffic engineering is a very different animal from outbound traffic engineering.

You're not incorrect in that the fault lies with Verizon, but I seriously doubt you actually understand why or how.

about 4 months ago

Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling

Drakonblayde Re:Could be a different route involved for the VPN (398 comments)

You do not understand how BGP works.

The problem isn't the data that verizon is sending to netflix, it's the data netflix is sending to verizon. Verizon messing with routing policy on netflix's announced prefix's wouldn't have an effect on verizon's streaming speeds. The traffic flows from Netflix to Verizon.

Therefore, in order to influence streaming speeds, Verizon would have to change their routing policy on how they announce their own routes in order to influence which links netflix traffic can come in on. The problem is, there's no way within BGP for Verizon to say I want Netflix traffic to only come in over these specific links. It would influence *all* traffic from that peer. Routing policy is destination based, not source based, and not source-destination based. By simply announcing the routes are more preferable over Level 3 saturated links, that forces traffic Level3 delivers for those prefixes to come in over those links.

Sure, Level3 could do some traffic engineering of their own and ignore or mutate some parts of Verizon's route announcements, and force that traffic in over unsaturated links Verizon may have with Level 3 (if there are any), but Level3 is a middleman. Doing so would take them out of their middleman status and put them firmly on Netflix's side. Verizon's likely response would be to immediately de-peer Level3.

The only folks who can effectively change how the traffic reaches Verizon's network is Netflix. They determine their outbound routing policy, but only up until their own border. Once it transits to another AS, it will be forwarded according to the upstream AS's routing policy. If Netflix wants to avoid saturated Verizon-L3 interconnects, the only thing they can do is not send traffic to Level3 for Verizon prefixes. They could easily modify their inbound route policy to send traffic for Verizon's prefixes via another peer. This is something that Level3 does not want, because it effects their revenue, hence their seeming to take sides with Netflix on the matter. It's one thing for Level3 to have an opinion on what Verizon is doing, that doesn't really effect operations. The second you change operations to try and force that opinion, well, you're likely to invoke the Law of Unintended Consequences.

Now, for whatever reason, Netflix has decided to go ahead and keep sending Verizon traffic to Level3. The reality is that if Verizon has decided to be douchebags about this, then they can do the same thing for whatever peer the traffic is ingressing through. Maybe all of Netflix's other peers ultimately transit to Verizon via Level3 anyway, which would make any change of forwarding policy moot.

About the only way for Netflix to solve this is to go ahead and cut out the middleman and just pay Verizon directly for interconnects into their network. This is what Verizon wants: another revenue source for traffic their going to deliver anyway. This is what Level 3 does not want: When you cut out the middleman, the middleman makes no money.

Netflix has already done it with Comcast and AT&T, so it's not surprising that Verizon wants in on this action as well, and will continue to be douchebags about it.

In the meantime, savvy customers can come up with their own solutions in order to avoid having netflix traffic destined for them coming in over saturated links. VPN and tunneling are two perfectly valid solutions.

about 4 months ago

Comcast Carrying 1Tbit/s of IPv6 Internet Traffic

Drakonblayde Re:Advantages? (146 comments)

NAT provides "security" because it is actually impossible to hack a computer behind a NATing router, without A) hacking into the router (in which case a firewall doesnt matter), or B) having the end user poke a hole / port forward through the NAT (which they could do with a firewall).

Oh that's not true at all. Just because you can't access the IP assigned directly to the computer from outside of the NAT hardly means you can't communicate with the device. NAT doesn't imply packet inspection or anything of that nature.

Far too many people confuse NAT as a security product because it's often paired with limited firewall capability in consumer grade routers. If you can do sufficient packet analysis (like, say, compromising an outside host that the NAT'd box talks to), you can communicate with the device. Malicious packets are translated inward just fine.

If the end user of a NAT box does something stupid, like install software that (innocently or not) leaves the host exposed behind the NAT, the NAT might as well not be there.

NAT is an ugly hack for address conservation, nothing more. Those who trumpet it as a security benefit are severely lacking in their TCP/IP skillset.

about 4 months ago

Microsoft Wants You To Trade Your MacBook Air In For a Surface Pro 3

Drakonblayde Maybe this is Microsoft's way of saying.... (365 comments)

they can't afford Apple products either. After all, if I had a Surface Pro 3, and I could sucker someone into giving me a Macbook Air for it, I'd do it in a heartbeat.

about 5 months ago

Cisco Opposes Net Neutrality

Drakonblayde Re:Its all about sales (337 comments)

While Cisco is primarily concerned with selling network gear, that doesn't make them wrong.

Believe it or not, every once in awhile, you can tell the truth and it will actually further your own agenda.

about 6 months ago

Cisco Opposes Net Neutrality

Drakonblayde Re:TFA is a lie (337 comments)

I got here late, but TFA is a lie. Stating the obvious (voice and HTTP are not "equal" to the client nor provider), doesn't make an official Cisco stance against Net Neutrality. In fact, most Net Neutrality proposals (every one I've seen officially submitted in Congress), would have allowed for such action. No Net Neutrality has yet prevented reasonable traffic grooming. It's designed to prevent Comcast from running a VoIP service with premium QoS and deliberately lowering the QoS of all other competing services. To keep all competing services at the same level is "neutral".

Net Neutrality is not "traffic neutral" It's "provider neutral" at least so far in every bill I've read. And that's the best way. Why force every packet to be the same when we know they are inherently not?

I wish I could mod you up. You have a proper understanding of what net neutrality is about, rather than what it's been perverted into.

about 6 months ago

Cisco Opposes Net Neutrality

Drakonblayde Re:Title is a bit sensationalist... (337 comments)

There is a difference between intra-domain and inter-domain prioritization and the operational futility of the latter.

inter-domain prioritization is hardly futile. ISP's don't own the entire world, nor is the entire world directly connected to one network. Customers use applications that are time sensitive and not owned by the provider. Customers expect that, if they want to view a video, for example, that the video actually plays and isn't choppy, or doesn't stop to buffer every 5 seconds. This is a crapload more important than how fast your Google search results load.

In this case they are warranted. Cisco's statements cannot possibly be applied to the real world without picking winners and losers.

In what way? Cisco is not saying Comcast should prioritize Netflix over Hulu, or vice versa. Cisco is saying that, yeah, ISP's should be able to prioritize Hulu and Netflix over, say, Facebook.

Let me put it this way - by insisting you treat everything the same, you're also picking winners and losers. Services and applications which need priority access (ie, very low latency and/or jitter) in order to work correctly or reliably are losers in the 'all should be equal' philosophy.

Or, let's illustrate this a little more colorfully - since the interent is often compared to a highway, that analogy will fit. Let's say you've got a gunshot victim in the back of an ambulance and he needs to get to a trauma center immediately. Do we expect the ambulance to follow the posted speed limits?

Or let's go another way - I have a limited amount of money. Tomorrow, I have to make the decision to pay the rent, or to pay my internet bill so I can make that nights World of Warcraft raid.

We already recognize the intrinsic need for priority in our everyday lives in order to get things done.

Some things are more important than others. The same concept applies to network operations, and trying to deny that is what's operationally futile.

about 6 months ago

Comcast Converting 50,000 Houston Home Routers Into Public WiFi Hotspots

Drakonblayde Re:Who owns them? (474 comments)

That's actually easy to answer - QoS markings. They're not just for traffic prioritization. It's pretty easy to use them for access control and accounting as well. Pubic WiFi traffic is marked one way, subscriber traffic another. Setting your own QoS markings won't get around that either - the modem will just strip them and set the proper ones.

about 6 months ago

Comcast Converting 50,000 Houston Home Routers Into Public WiFi Hotspots

Drakonblayde Re:In addition (474 comments)

The flip side of that is that you have access to the wireless mesh as well if you want it. You're not only providing it, you can consume it as well.

And Comcast is not charging you for power. You have to provide power to the cable modem in order to use it, and you're paying that fee whether you provide your own, or you take theirs.

I do agree, however. Leasing a modem is stupid. A Motorola SB 6121 costs about 80 bucks at your local big box store and is fully compatible with the Comcast network (do not believe the sales people or the install techs when they try and make you doubt whether or not the modem will work with the Comcast network - it will. Comcast supports every cable modem vendor you can find in Best Buy and it's like). It'll pay for itself in under a year. Now, if you're using one of the Wireless gateways, then the cable modem is also acting as a router, so you'll need to buy one of those as well if you don't already have them, which takes you a little longer to reach your break even point. However, most folks only change service providers when they move, and most folks don't move frequently, so chances are you'll hit the break even point before then.

If someone has been paying a cable modem rental fee for like 5+ years, then that person is terrible at math.

about 6 months ago

Comcast Converting 50,000 Houston Home Routers Into Public WiFi Hotspots

Drakonblayde Re:Who owns them? (474 comments)

Ehh, Comcast's business practices tend to suck, but their technical people do a good job. I think they were the first large-scale residential provider in the US with DNSSEC and IPv6 for example.

In any case, they are already doing separate channels for separate services (I believe that's how they implement voice service for example), so this will just be turning up another channel.

Nope. Comcast Digital Voice is straight IP, straight VoIP. It's not another service channel, the modem just has a priority queue for VOIP traffic.

There are separate RF channels for different products in the sense of video and VOD, but that's because they travel over different infrastructure at the hubsite/headend (it's not combined until it heads down to the customer facing nodes).

The one exception to that is the new X1 boxes. The channel guide is delivered through the CMTS rather than through a small freq OOB channel like it's been traditionally done (this is why if you reboot an X1 box, you don't have to wait for hours for the channel guide to repopulate). But even this isn't coming in via your internet cable modem - the X1 boxes simply have another cable modem inside of them that ranges up just for that information.

about 6 months ago

Comcast Converting 50,000 Houston Home Routers Into Public WiFi Hotspots

Drakonblayde Re:Who owns them? (474 comments)

This is accurate. How fast your speeds can go depend on three things -

1. How many channels are turned up on the CMTS side. Each DOCSIS 3.0 downstream supports ~42mbit/s and each upstream supports ~30mbit/s

2. How many channels your modem can bond. For example, if your cable modem can bond 8 downstreams, and 4 upstreams (which is pretty common with modern cable modems), you can have a theoretical maximum of 383 Mbit down and 122 Mbit up. Your modem will bond to as many channels as it can (ie, however many the CMTS offers, or the maximum amount it can bond to if the CMTS offers more than that)

3. The modems bootfile. This is where the rate limiting comes in. If you're bonding 8 down and 4 up, but you only purchased 30 down and 4 up as part of your plan, this is where they enforce that - the bootfile will have the QoS settings that the modem enforces.

There's a 4th factor that those who purchase top tier internet quickly find out - their router (assuming they haven't leased a cable modem that's also acting as a gateway). The shitty little Linksys you bought ten years ago is *not* going to be able to handle 100mbs of downstream, since the packets are all forwarded in software, and they didn't exactly put powerful procs in those things.

about 6 months ago

Cisco Opposes Net Neutrality

Drakonblayde Title is a bit sensationalist... (337 comments)

Of course a network vendor is going to point out that some packets needs preferential treatment over others. It's something they've worked to engineer into their product lines because their customers demand the capability to do so. For an ISP, 911 VoIP packets are a much higher priority than World of Warcraft packets.

Too many folks are caught up in the idea that prioritization is bad. There's a difference between between the philosophy of Network Neutrality and the operational reality of packet prioritization.

Saying Cisco opposes Net Neutrality just because they're pointing out some simple truths on how network operate today is like saying Glock supports terrorism just because they make guns.

Of course, if the title weren't sensational, no one would probably read it.

It saddens me that Slashdot seems to have decided that they need to resort to the same tactics as the National Enquirer

about 6 months ago

Comcast CEO Brian Roberts Opens Mouth, Inserts Foot

Drakonblayde Re:Truth in labeling, truth in advertisement law. (343 comments)

Yes, they did.

Why? Because there was no benefit for Comcast to do so.

This isn't like Comcast refusing aid to vets, the homeless, or kicking kittens, or anything like that.

The colocation appliance *only* helps Netflix. For Comcast, it increases OpEx for no benefit. That tends to make shareholders mad.

So what responsibility does Comcast, a for-profit entity, have to help another for-profit entity (and a direct competitor to Comcast's own streaming service) contain their operational costs. Would it be nice? Sure. But Comcast isn't in business to be nice. It's in business to make money.

The idea that Netflix is some sainted company that deserves to be treated better than any others mystifies me.

about 6 months ago

Comcast CEO Brian Roberts Opens Mouth, Inserts Foot

Drakonblayde Re:He also forgot to mention... (343 comments)

I actually know quite a bit about routing. Enough to know that you don't call an interconnect a 'route'. A route is a destination prefix with a valid next-hop. But go right on ahead and think you know all about internet routing just because you read a blog entry or two.

And yes, you do let interconnects get congested if there's no business case for upgrading them.

I'm not saying I disagree with your opinion. As an operations wonk, it pains me to see capacity issues and my natural inclination is to fix it.

However, that shit costs money, and not lunch money either. This is when the business side interfered with the tech. Unless there's a good business case for it, those links aren't getting upgraded until there is.

In this case, there was no business case to do so. If Netflix was complaining about the quality of service because of the saturation on the Cogent interconnect, all they had to do was alter their routing policy to send the traffic for Comcast's prefixes out their Level 3 links instead. It's a trivial and often performed piece of BGP traffic engineering. Netflix decided not to do so (because Level 3 is a crapload more expensive than Cogent) and make a public stink about it.

Even after the public stink failed, Netflix *still* decided not to send the traffic out Level 3, opting to purchase direct links into the Comcast network (and, shortly thereafter, into AT&T's network as well. Strangely enough, people don't seem to have an issue with AT&T telling Netflix to go fuck themselves, just Comcast). That decision should tell you a couple things - Bandwidth ain't cheap. It's cheaper than it was 10 years ago, on a per mbit/gbit cost, but the amount of traffic crossing has scaled up even while prices have been scaling down. And it should tell you just how expensive Level3 is to actually use.

You're basically saying that, just because the Cogent link was saturated, Comcast should have instantly gone ahead and upgraded their links with Cogent, nevermind that the guys doing the complaining had links to another Comcast transit provider, who's links *weren't* saturated.

When bandwidth costs are the clear majority of your OpEx, you think twice about doling out CapEx and additional OpEx if there is another option. You would make a horrible network operator.

about 6 months ago


Drakonblayde hasn't submitted any stories.


Drakonblayde has no journal entries.

Slashdot Login

Need an Account?

Forgot your password?