Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

ISP Fights Causing Netflix Packet Drops

Soulskill posted about 7 months ago | from the never-attribute-to-malice... dept.

Media 289

An anonymous reader writes "We've been hearing more and more reports of ISPs throttling Netflix and other high-bandwidth services lately. The ISPs have denied it, and even Netflix itself seems to believe them. If that's the case, what's going on? Well, according to this article, the blame still lies with the ISPs. While they may not be explicitly throttling connection speeds, they're refusing to upgrade network connections as they demand more money from content distributors. For example, Netflix pays Cogent to distribute their internet traffic. Cogent has an agreement with Verizon to exchange traffic — which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement. Verizon wants more money from Cogent, and one of their negotiating tactics is simply to stop upgrading their infrastructure so that service degrades. 'There are about 11 Cogent/Verizon peering connections in major cities around the country. When peering partners aren't fighting, they typically upgrade the connections (or "ports") when they're about 50 percent full, Cogent says. ... With Cogent and Verizon fighting, the upgrades are happening at a glacial pace, according to Schaeffer. "Once a port hits about 85 percent throughput, you're going to begin to start to drop packets," he said. "Clearly when a port is at 120 or 130 percent [as the Cogent/Verizon ones are] the packet loss is material."'"

cancel ×

289 comments

Sorry! There are no comments related to the filter you selected.

Yes, it is true! (-1)

Anonymous Coward | about 7 months ago | (#46306275)

I'm a preposterous penis pumping extremist! Bow before me, Slashdot!

Re:Yes, it is true! (-1)

Anonymous Coward | about 7 months ago | (#46306329)

What say you about this?

Re:Yes, it is true! (0)

bobbied (2522392) | about 7 months ago | (#46306525)

I'm a preposterous penis pumping extremist! Bow before me, Slashdot!

Dude, you are hard core..

Can confirm (2)

schneidafunk (795759) | about 7 months ago | (#46306301)

I've been streaming netflix and noticed a lot of stoppage & buffering. I ended up upgrading my wireless router thinking that was the problem. Nope, it's still happening.

Re:Can confirm (3, Informative)

dave562 (969951) | about 7 months ago | (#46306339)

Oddly enough Netflix is running well for me on Time Warner in southern California. I am no fan of Time Warner, but House of Cards (the major hot button that people seem to be using as the yard stick for performance this week) is done buffering HD by about half way through the opening credits.

The majority of the articles that I have seen all seem to mention Verizon, with little to no mention of the other ISPs. Maybe the real issue is a fight between Verizon and Cogent? If that is the case, what is the solution? Do we recommend allowing Verizon to acquire more content producers so that they stand to benefit by pushing their content back the other way, thereby restoring the in/out ratio?

Re:Can confirm (1)

schneidafunk (795759) | about 7 months ago | (#46306533)

Well that's interesting to hear, and I do have Verizon.

Re:Can confirm (3, Insightful)

msauve (701917) | about 7 months ago | (#46306557)

Why should Verizon expect to have an even in/out ratio? They sell the vast majority of their customers asymmetrical connections.

sell is the key word. Cogent not paying Verizon (3, Informative)

raymorris (2726007) | about 7 months ago | (#46306621)

They SELL connections to their customers. Cogent isn't paying Verizon. The peering they're talking about is an even trade with neither company paying the other. When it ceases to be an even trade, it's time for negotiations.

Re:sell is the key word. Cogent not paying Verizon (2, Insightful)

Anonymous Coward | about 7 months ago | (#46306715)

Yes, they SELL connections to their customers, so the incoming flood of traffic from Cogent has already been paid for by Verizon's customers that are trying to watch their shows on Netflix. Verizon is trying to double-dip. So what if the traffic is asymmetric? If it's that big of a problem, the continued performance problems caused by Verizon's intransigence could be solved by a massive reduction in the customer base once they find that they can't watch Netflix.

Re:sell is the key word. Cogent not paying Verizon (1)

0123456 (636235) | about 7 months ago | (#46306779)

From what I understand, peering agreements are typically based on a similar amount of data going both ways, so both sides benefit from just connecting their networks together with no need for additional payments. But, if 90% more data is coming in than going out, then the network sending the data is going to have to pay the network that's receiving it. Which means Netflix will have to pay their ISP more so they can pay to deliver the data.

Re:sell is the key word. Cogent not paying Verizon (1)

Anonymous Coward | about 7 months ago | (#46306835)

Which makes sense if the "sending" network is sending broadcast packets or something, but users of "receiving" network are paying "receiving" network to connect them to "sending" network, so peering should be a wash.

Re: sell is the key word. Cogent not paying Verizo (0)

Anonymous Coward | about 7 months ago | (#46306721)

In the isp consumer space we pay for what we download.

So Verizon can pay for what it downloads thanks.

Re:sell is the key word. Cogent not paying Verizon (5, Insightful)

msauve (701917) | about 7 months ago | (#46306755)

Customers PAY for those connections. Verizon's customers are paying to receive traffic from the Internet. Whether that's slashdot or Netflix doesn't matter, it behooves Verizon to deliver the service their customers are PAYING for.

Re:Can confirm (1)

Anonymous Coward | about 7 months ago | (#46306781)

Verizon's service consists of a hell of a lot more than FiOS. The vast majority of their bandwidth comes from symmetrical business fiber connections. Plenty of content is also hosted on Verizon connections as well. Prior to Netflix, there was significantly more traffic going out from Verizon's network than in.

Re:Can confirm (5, Informative)

schnell (163007) | about 7 months ago | (#46306833)

It has been a while since I was in the peering game - that was back when UUNet meant something and please get off my lawn - but here's the general idea:

To provide access to the Internet - either as a content host or a content consumer - you need a full view of routes to get to all other ISPs where your sources or destinations are. To get the routes, you need "peering" with the other ISPs or you need to buy "transit" from some other ISP that gets their routes via peering or transit.

Some ISPs are, frankly, more important than others because they provide access to more subscribers or more content than others - they used to be called "Tier 1" ISPs. Peering is valuable because it's traffic you're exchanging for free that you could otherwise be charging a lot of money for. Tier 1 ISPs generally agree to peer with each other because they all need each other, and it makes economic sense for them to say they're all on an equal footing - they were "peers." The economic rationale was because Tier 1 ISPs had to pay for large national or global networks, while Tier 2 or 3 ISPs had small or regional networks and that the Tier 1 ISP was bearing most of the cost of delivery. Traffic ratios were preferred to be equal (content vs. users) for peering, because if you're a content host with one datacenter and some outbound circuits, your cost is far less than having a big national network to serve end users - so web hosting/colo provider ISPs had a harder time getting peering with the big consumer/business access providers. The Tier 2 or Tier 3 ISPs would peer with each other freely because they had equivalent footprints, etc., but the big guys knew that access to their network was extremely valuable and it would be foolish to give it away for free.

So if you're a smaller ISP, and *you* need the Tier 1 more than *they* need you, don't expect to get peering. The ISP will tell you to buy transit from them, or at least buy transit from someone else who does (and the fewer "hops" to get from you to them, the better for your customers). Cogent may host much of Netflix, but they are by no means a Tier 1. This may no longer be the case - like I said, I have been out of the Tier 1 ISP world for years - but at least historically Cogent was known as a bottom feeder of the industry. They charged dirt-cheap rates but ran a crappy network and skimped on their upstream connections to cut costs.

So what's happening here most likely is that Cogent has either bought transit from Verizon and doesn't want to buy more and says "peer with us, we won't buy more." Or Cogent does have peering with Verizon but VZ has said, in effect, "you are not our 'peer'" and beyond a certain amount of peering bandwidth, you should start buying." Cogent is using Netflix to try to argue that "our content is more important to Verizon customers than the other way around," and Verizon is saying, "Um, nope." I won't say who I think is right or wrong here, but this is not the first time Cogent has had peering fights with other large ISPs [google.com] and I think you can see a pattern here.

Re:Can confirm (0)

Anonymous Coward | about 7 months ago | (#46306355)

VZ just also spent a huge chunk of money buying VZW. No spending is going on other than 'necessary'.

Re:Can confirm (1)

rwa2 (4391) | about 7 months ago | (#46306415)

We could also, you know, confirm by looking at packet loss stats from Cogent to your ISP's carrier
http://internethealthreport.co... [internethealthreport.com]

Re:Can confirm (0)

Anonymous Coward | about 7 months ago | (#46306447)

Me too, but sometimes it works fine. I haven't figured out if I watch "battlestar gallactica" at 11pm when everyone else is asleep, or if "transformer's rescuebots" is on a particular netflix streaming server that is susceptible to all this. Does anyone know enough about netflix's arch to comment on the possibilities?

Re:Can confirm (0)

Anonymous Coward | about 7 months ago | (#46306491)

That doesn't confirm anything from this article.

Re:Can confirm (1)

jest3r (458429) | about 7 months ago | (#46306613)

Same here. For the past week or two I have been experiencing all sorts of glitches, stoppages and buffering through Netflix. My local ISP on demand service is fine though. Before last week everything was awesome!

Called my ISP about it - they said contact Netflix.

Maybe Netflix should add net neutrality to the House of Cards story arc to get the word out???

Re:Can confirm (1)

suutar (1860506) | about 7 months ago | (#46306689)

I haven't been getting stoppage/buffering with Netflix, but I've been thinking I was getting lower video quality than I'm used to, which could be a less severe result of packet loss. I have definitely been getting a crapload of buffering with Amazon Instant Video, though.

as always streaming sucks, torrents rule (0, Troll)

Anonymous Coward | about 7 months ago | (#46306311)

so it takes an extra hour or to to get a movie, what do I care?

thanks to everyone paying for netflix and going to movies, if I wasn't poor I would totally join you in spending money on entertainment
(after medical bills, home improvements, saving for retirement, and entertainment I can't download for free)

Re:as always streaming sucks, torrents rule (1)

kthreadd (1558445) | about 7 months ago | (#46306347)

Well, the fact that it's illegal is a bit of an obstacle.

Re:as always streaming sucks, torrents rule (2, Insightful)

Anonymous Coward | about 7 months ago | (#46306387)

That didn't stop the NSA

Re:as always streaming sucks, torrents rule (1)

SuricouRaven (1897204) | about 7 months ago | (#46306395)

It would be if copyright law was remotely enforceable online.

Network vs Content providers (5, Insightful)

future assassin (639396) | about 7 months ago | (#46306317)

If companies provide network access they should not be be allowed to be a content provider. Too much conflict of interest and they can concentrate on properly managing and not OVERselling their network.

Re:Network vs Content providers (2)

dave562 (969951) | about 7 months ago | (#46306373)

In your mind, where does Cogent fall on the network provider / content provider divide? They are providing network access, but they are also (as far as I know) providing the infrastructure for Netflix's CDN.

Re:Network vs Content providers (4, Informative)

thule (9041) | about 7 months ago | (#46306397)

Cogent likes to think of themselves as a pure bandwidth company. No frills bandwidth for a great price. No content, no VoIP, nothing. They have colocation data centers, but that came when they purchased a company for their network.

Re:Network vs Content providers (1)

suutar (1860506) | about 7 months ago | (#46306825)

I would put Cogent on the network provider side. They store bits, they shovel bits, they don't care whether those bits encode Star Trek or Futurama or Dale and Tucker Versus Evil. (Unless they have a stake in Paramount that I haven't heard of :)

Re:Network vs Content providers (5, Interesting)

Lawrence_Bird (67278) | about 7 months ago | (#46306489)

That isn't even the real point, though I understand where you are going with it. The real issue here is that Verizon, Comcast, Cablevision, etc all have agreed to provide their customers with X/Y data connections for a hefty monthly fee. They are refusing to do what it takes for those customers to be able to use what they have paid for. In effect, we are back to the early internet days of ISPs oversubscribing dialup lines except now it is oversubscribing routing equipment during peak hours.

Re:Network vs Content providers (1, Informative)

Pinhedd (1661735) | about 7 months ago | (#46306799)

The congestion isn't on the customer-facing side of the ISP's network, it's at the peering exchange with Netflix's carriers. They make no promise about that.

Re:Network vs Content providers (1)

ducomputergeek (595742) | about 7 months ago | (#46306553)

We really missed the boat with not having local governments at the city/county level build the infrastructure. Where I grew up we had an electric coop. My Dad still lives there. Other than the main lines, it was all buried cables. Power outages were extreme rarities. His rates have been between $.08 - .11 per kilowatt hour. Hell a few years ago the rates went DOWN after the coop paid off some debt. In the city our rates are about .145 per kilowatt hour and increasing to .16 per kilowatt hour here soon. Last state I lived in started to force Utilities to open up their lines to competitors. My electric rates there went down from $0.18 per KW/hr to $.12 when that happened.

When I moved into the city, it was a major utility company, overhead wires, and if we got some ice or wind the power would go out.

Where my Dad lives, water & sewer is maintained by the city, and now contracts out with the county for water. His bill is about 1/3rd less than what our bill is closer to the city and our water is owned by a private utility.

Where it made sense we should have had cities or counties putting in fibre and allowed rural areas to form wireless coops. Then lease out service to whomever at a fixed cost per line.

Re:Network vs Content providers (0)

Anonymous Coward | about 7 months ago | (#46306727)

WT ever loving fuck is a wireless coop? Is that like electric fence for chickens or something?

Look man, do you think government makes up electricity by pulling it out of thier asses and then gives it away to the citiens out of their feelings of beneficience?

Man where can I get some of what you are smoking.

*This* is what government do when you give them power. Maybe not today or tomorrow, but if you give them power, this is what you get in the end.

Every. Fucking. Time. Word.

http://www.independent.co.uk/news/world/europe/ukraine-crisis-fresh-clashes-break-out-despite-president-viktor-yanukovychs-truce-with-opposition-9140428.html

But yea, computergeek says nah, this could never happen here. Sure.

Re:Network vs Content providers (1)

bluefoxlucid (723572) | about 7 months ago | (#46306743)

Studies show buried lines have more down time than overhead lines.

Re:Network vs Content providers (1)

Obfuscant (592200) | about 7 months ago | (#46306747)

We really missed the boat with not having local governments at the city/county level build the infrastructure.

No, we didn't, and this is not an example of that kind of problem anyway. It's an issue of a peering relationship that has changed from "peer" to a heavy bias towards traffic going one way. In the early days of the net peering companies decided not to try to charge each other for packets based on the idea that things would balance out. It was also harder to track that kind of stuff. It would cost more for accounting than any imbalance would cost. That's not true anymore since there are concentrated streaming video sources pushing large volumes of uncacheable content out.

Where I grew up we had an electric coop.

Where I grew up there was a coop. It's not a city or government operation, it was a coop. Customer owned. "Coop" is short for "cooperative", as in the customers cooperatively operating the company.

Netflix should know better (5, Interesting)

Spazmania (174582) | about 7 months ago | (#46306335)

Cogent has a long history of instigating peering disputes with other networks. Normally I'd complain about Verizon's behavior but this is Cogent we're talking about. They have -zero- credibility.

Re:Netflix should know better (2)

ffsnjb (238634) | about 7 months ago | (#46306467)

Agreed, every time I look at any various looking glass to see why something is broken on the internet, it's Cogent dropping the ball... Remember this [slashdot.org] ?

Every time I have to troubleshoot a broken internet problem, and it goes up to Tier 1 land, check the looking glass... It's Cogent being stupid. As soon as this story hit... check. [internethealthreport.com] Sure enough, Cogent-Level3 has ~3% packet loss the last 24 hours, with stupidly high latency. Screenshot [imgur.com] .

Re:Netflix should know better (3, Insightful)

Shinobi (19308) | about 7 months ago | (#46306657)

Yeah, Cogent really is trying every dirty trick they can to go past any contract limits etc and freeload, and then they cry loudly to the media when they get told to stick to the contract.

Dumb design (1)

Anonymous Coward | about 7 months ago | (#46306351)

Any streaming protocol that doesn't adapt to available bandwith and allow for the odd dropped packet is broken.

Re:Dumb design (1)

jspoon (585173) | about 7 months ago | (#46306511)

Netflix does adapt to low bandwidth by the obvious contingency of showing me terrible low bitrate streams. I'm on FIOS and used to get consistent beautiful HD so I feel entitled to complain.

Chromecast Vs. Roku (2)

TrippTDF (513419) | about 7 months ago | (#46306357)

I have a 1st gen Roku, and a Chromecast. When I stream Netflix with the Roku, I seem to top out at 2, maybe three quality bars. While there's no on-screen metric for the same stream in Chromecast, the picture is noticeably better. Perhaps the Chromecast is getting a higher-quality, lower-bandwidth stream, or there's some sort of throttling based on the streaming device going on?

Re:Chromecast Vs. Roku (1)

sixsixtysix (1110135) | about 7 months ago | (#46306719)

or there's some sort of throttling based on the streaming device going on?

This wouldn't surprise me in the slightest. Sometimes, they'll actively block certain devices. Just look at hbo go. I can stream on my computer or iphone app over wi-fi (or cellular), but, and this is the most retarded thing ever, it will not work on my roku because comcast and hbo go don't have a deal (which I don't get why they'd even have too, considering the previous 2 services work fine and both require linking my comcast and hbo accounts).

Net Neutrality laws? (3, Insightful)

thule (9041) | about 7 months ago | (#46306363)

I've been saying this for ages! Even mentioned this here on slashdot. Peering is peering. They are not degrading performance by configuration, they just let the link get congested. How do any of the proposed net neutrality laws address this issue? Answer is, they don't. To me that means that Net Neutrality laws are about something different than neutrality. More likely with government regulation, it becomes Net Control. With that, increased stiffing and limiting reaction to market dynamics, not improving it.

Re:Net Neutrality laws? (1)

dave562 (969951) | about 7 months ago | (#46306445)

You are mixing apples and oranges.

Peering agreements have been the same forever. As long as there is nearly a 1:1 ratio between the providers, everything is fine. The issue comes up when one side is using more bandwidth than they are giving in return.

Netflix is breaking the long standing status quo. Last I checked, they accounted for ~30% of ALL of the traffic on the internet. Obviously that is going to skew the metrics, and that is why Netflix is trying to push their own CDN. I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs. What they should do is run the numbers and figure out what costs more; "overage" charges from Cogent, or eating the cost of paying to deploy their CDN hardware and network links to the other Tier1 ISPs.

Are you suggesting that net neutrality should address situations like this? Are you saying that it is a good idea to have the government force a business to eat the cost of supporting someone else's business model? To me, that sounds like a big fat subsidy for Netflix at the expense of everyone else.

Netflix should get benefit from desirability (4, Insightful)

SuperKendall (25149) | about 7 months ago | (#46306495)

Netflix is breaking the long standing status quo. Last I checked, they accounted for ~30% of ALL of the traffic on the internet. Obviously that is going to skew the metrics, and that is why Netflix is trying to push their own CDN. I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.

Why? A lot of people might only get internet, or faster internet anyway, BECAUSE of netflix.

If I were not streaming stuff on Netflix I might very well just use a cellular internet connection and not get cable internet at all. Netflix is helping the ISP's make money, and Netflix should gain some benefit from that fact as a result.

Re:Netflix should get benefit from desirability (1)

dave562 (969951) | about 7 months ago | (#46306625)

Do you have any hard data on how many people are actually purchasing high speed internet specifically for Netflix? I have not seen any data, but my gut feeling is that the number is a fraction of a percentage of people who have high speed connections. Would you agree that by and large, the majority of people viewing Netflix are people who already have high speed connections?

Right now Netflix is getting to benefit at the expense of ISPs. Do not get me wrong here, I do not like the way the ISPs are handling it and I think it is disingenuous that they are not keeping their circuits up like they should be. My issue here is that Netflix is not a good champion to use to make the case for ISPs being tightfisted when it comes to circuit maintenance.

Netflix partnered with Cogent. They could have partnered with Level3 or any other Tier1 provider and they would be in the same situation. The peering status quo has broken down. Cogent needs to pay to make up for the lop sided transfer ratio. Cogent will most likely pass that cost on to Netflix. Netflix will have to raise their prices to offset the cost hike. Until that happens, Netflix is able to provide artificially low rates to their customers at the expense of other ISPs. Bandwidth is not free so someone has to pay for it.

To use a real world example, when builders, either residential or commercial plan new communities, they have to do environmental impact studies. One of the essential components is usually a traffic study. If they are going to go for a high density development, like a massive multi-story apartment complex, most of the time they have to work out a deal with the city to widen nearby roads, install traffic lights, widen the sidewalks, and do all sorts of other things to mitigate the impact. Right now, the internet does not have the equivalent of that, and Netflix is benefiting from it.

Imagine if your neighbor turned his house / apartment into a concert venue because there were not any ordinances to prevent it. All day long, you could not find a place to park, could not have guests over because they could not park, suffered brown and black outs because the music equipment kept blowing out the local transformers, etc. How would you handle that?

Re:Netflix should get benefit from desirability (1)

0123456 (636235) | about 7 months ago | (#46306813)

Do you have any hard data on how many people are actually purchasing high speed internet specifically for Netflix?

The only reasons I need megabit Internet are downloading games from Steam and GOG, and watching videos. Games, I don't much care if I have to leave the PC running overnight while it downloads, but streaming videos are either high enough bandwidth to play, or they're not, and will stutter and buffer or drop to low quality.

So I suspect many people are buyig high-speed Internet just to watch videos, be that Netflix, Youtube, or pirate sites.

Re:Net Neutrality laws? (1)

Lord Lemur (993283) | about 7 months ago | (#46306521)

Hasn't Netflix been offering to put their equipment inside of any ISP that will have them, since forever?

Re:Net Neutrality laws? (4, Informative)

Anrego (830717) | about 7 months ago | (#46306529)

Yeah, but it's not like Netflix is using a free service. They are paying for that bandwidth. I assume they are paying quite a bit. More importantly, someone is selling it to them.

Re:Net Neutrality laws? (1)

firex726 (1188453) | about 7 months ago | (#46306759)

Also Netflix has made the offer available to setup CDN servers in the local providers facilities to ease the BW usage on their peers. Google uses this; but Verizon, Time Warner, and Comcast do not, who then whine about the BW use.

Re:Net Neutrality laws? (3, Informative)

heypete (60671) | about 7 months ago | (#46306595)

I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.

All the peering details are here [netflix.com] . In short: they don't charge anything. They offer direct interconnects to Netflix's CDN for free, free peering at major internet exchange points, and free, Netflix-managed hardware caches to ISPs to avoid duplicate network traffic (the vast majority of traffic stays within the ISPs internal network). For the hardware caches the ISP needs only provide power and network connectivity.

There's really no reason for ISPs to wrangle with Netflix -- there's plenty of options to avoid congestion.

Re:Net Neutrality laws? (1)

compro01 (777531) | about 7 months ago | (#46306791)

There's really no reason for ISPs to wrangle with Netflix

Verizon has two [verizon.com] reasons [verizon.com] to wrangle with Netflix.

Re:Net Neutrality laws? (1)

H3lldr0p (40304) | about 7 months ago | (#46306637)

Except there has never been anything close to a 1:1 relationship. There couldn't have been because even since the IDSN days the up/down ratio of what we could get was always in favor of the down. So there's never been enough traffic coming from the ISPs to even approach parity.

In fact, it's been the stupid ISPs have been using this as a club against the other players. It's of their own making, and now they're choking on it. In fact, Nexflix is even willing to give ISPs servers to take the congestion away from that part of the network, something that's been consistently rejected. The ISPs are the problem. In specific their greedy, overly Wall Street focused, stupid management is the problem.

So, no. They don't get any sympathy from me in this. They aren't being held accountable by their customers, they aren't being held accountable by what little law is in place, and they certainly aren't being held accountable by the market. If they were, they'd be out of business by now.

Re:Net Neutrality laws? (1)

dave562 (969951) | about 7 months ago | (#46306681)

Except there has never been anything close to a 1:1 relationship. There couldn't have been because even since the IDSN days the up/down ratio of what we could get was always in favor of the down. So there's never been enough traffic coming from the ISPs to even approach parity.

Do you have any data to support that? On the last mile connections, there is an imbalance between up and down. On residential versus commercial, there is definitely an imbalance between up and down. But at the highest levels, the content farms and co-located customers, I thought it was pretty even.

I am pulling these examples out of my butt in an attempt to provide some context. I have no idea who has hosting agreements with who. But if you have NBA.com on ISP1 and NFL.com on ISP2, and MLB.com on ISP3, the traffic between the two would be roughly equal. Again, probably a bad example, given the different sizes of the audiences for each, the fact that the seasons do not overlap, etc. The point that I am trying to make is that most of the major ISPs all host high traffic farms, those farms are not centralized with one specific ISP, so far the most part, there is roughly 1:1 between the various providers.

Re:Net Neutrality laws? (1)

marka63 (1237718) | about 7 months ago | (#46306655)

Requiring 1:1 traffic flows is a stupid way to organise peering agreements. It doesn't matter if it is 1000:1 as long as both sides are getting benefit from the peering. If you have content you want to deliver and eyeballs that want content you peer.

As for transit providers and eyeball networks. The transit providers are being paid to deliver content and the eyeball networks are being paid to deliver content. Failing to upgrade interconnects is taking monies under false pretences and that has another name "fraud".

Re:Net Neutrality laws? (1)

MozeeToby (1163751) | about 7 months ago | (#46306731)

Netflix is breaking the long standing status quo.

Here's what you don't understand: Cogent (on behalf of Netlix) wants to send packets to the ISP. The ISP (on behalf of their customers!) wants to receive those packets. No one is getting shafted. Both sides of the agreement are already getting what they should be wanting. Now the ISPs are coming along and saying that the peering agreements are only about upstream bandwidth? Why? The ISP's customers are the ones who requested the packets that Cogent is trying to deliver to them.

Re:Net Neutrality laws? (1)

sixsixtysix (1110135) | about 7 months ago | (#46306763)

Isn't someone always using ~30% of ALL internet traffic? If it's not netflix, it's torrents or youtube. These 3 must use 99.999999% of all internet traffic from what people post here.

Re:Net Neutrality laws? (2)

kqs (1038910) | about 7 months ago | (#46306693)

Very logical. In related news, many things can make you ill, not just toxic waste. Therefore, any laws which discourage dumping toxic waste on your property are not about health, and are probably just about governmental control of companies.

Peering limits are certainly used as a rough form anti-net-neutrality, but they're not ideal; they have the pinpoint accuracy of a sawed-off shotgun at 100 yards and they are very obvious. The proposed laws tend to target the subtler, better directed forms. Don't let the perfect be the enemy of the good.

Natural monpolies (2)

bob_super (3391281) | about 7 months ago | (#46306385)

Water, then gas, then electricity, phone, and now internet backbone are Natural Monopolies.
Private companies should not operate in Natural Monopolies because they just add overhead cost as they operate for profit.
Obviously, with the American public convinced of the lie that government is just waste, and that private companies magically reduce waste (at the bottom, for sure), we all keep on shelling massive amounts of cash for basic services. Cash that would better be invested in keeping the US moving forward, rather than lining the pockets of terrible profiteers like my 401k...

Re:Natural monpolies (0)

Anonymous Coward | about 7 months ago | (#46306543)

"Obviously, with the American public convinced of the lie that government is just waste, and that private companies magically reduce waste (at the bottom, for sure), we all keep on shelling massive amounts of cash for basic services. Cash that would better be invested in keeping the US moving forward, rather than lining the pockets of terrible profiteers like my 401k..."

lolwut?

'the lie that government is just waste'

OMFG.

http://blog.heritage.org/2014/02/06/billions-wasted-flawed-obamacare-exchanges/

"The federal taxpayers have spent over $5 billion to build Obamacare’s exchanges. Unfortunately for taxpayers, these investments have not delivered a good return and are ongoing.

Despite the muddled launch of many exchanges, the federal government is still doling out grant money, giving out over $200 million to nine states last month. Fourteen states and the District of Columbia are running their own Obamacare exchanges and they all received federal grant money to help build them—a total of over $4 billion. California alone received $1 billion to set up its exchange."

Billions bob_super, billions with a B, and do remember, this is not even healthCARE, it's an internet shopping cart, which isn't even FUCKING NEEDED if they just say "hey citizen buy health insurance that is Obamacare compliant and claim any applicable subsidy on your tax return" but no fuck you they WANT to build the exchange becuase Obama's wife has buddies that need the graft and then; the shit DONT even WORK *and* your PII is sent automatically to hackerz in Ukraine, *and* ain't no one seeing no $2500 reduction in health insurance costs (hahaaahahahahaaaa, hahahahaaaa, bwaheahaha hahahahah FUCK YOU) *and* the fucking doctors listed as in network *aint* in the fucking network.

But yea, you must be right and government is always more efficient, super bob. We'll go with that.

Re:Natural monpolies (1)

bluefoxlucid (723572) | about 7 months ago | (#46306773)

Huge massive thing with 8 billion functions screws up 1 function. See? Huge massive thing is all waste, no efficiency.

Re:Natural monpolies (0)

Anonymous Coward | about 7 months ago | (#46306807)

WT ever loving F are you talking about?

Re:Natural monpolies (0)

Anonymous Coward | about 7 months ago | (#46306563)

Well, no, none of those are natural monopolies. Those are state chosen monopolies because if the government did not force citizens to accept the utilities being dug under their land, some people would refuse or only permit the digging if it came with some exorbitant personal payment. This way, only the government gets the exorbitant payment, but everyone gets access to the utilities at some monopoly-declared price points.
The governments involved only have incentive to allow another provider of any utility if they are convinced that the kickbacks from two companies will actually exceed the kickbacks from one company that doesn't want competition.

Re:Natural monpolies (0)

Anonymous Coward | about 7 months ago | (#46306623)

Water, then gas, then electricity, phone, and now internet backbone are Natural Monopolies.

You forgot roads.

Sadly if this wasn't an oligopoly, it wouldn't be (0)

Anonymous Coward | about 7 months ago | (#46306393)

The sad fact is, if Verizon wanted to improve customer satisfaction they could easily join Netflix's Open Connect program: https://signup.netflix.com/openconnect
Instead they decide not to and customers are screwed, because we don't have an alternative provider available. Long term solution is for Congress to get off it's ass
and mandate a national fiber installation that will be leased out to any network provider that's willing to run an ISP.

"Lopsided"? (2)

K. S. Kyosuke (729550) | about 7 months ago | (#46306423)

Cogent has an agreement with Verizon to exchange traffic — which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement.

I'm not quite sure I understand what's "lopsided" about that. It still flows through both of the two networks, and the companies of the receiving end are still the ones cashing in from end users, and they still very much like their users paying extra money for extra data transfers, don't they?

RE: "Lopside"? (0)

Anonymous Coward | about 7 months ago | (#46306477)

Peering arrangements are typically ran such that there are equal amount of sending and receiving traffic on both sides.
With the advent of Content Delivery Networks this is about impossible to achieve unless you are talking about two Tier1 ISPs exchanging traffic such as Level3 and Cogent.
Basically, Verizon are claiming that because Verizon customers want to watch Netflix, Cogent should pay them to allow the traffic.

Re: "Lopside"? (1)

K. S. Kyosuke (729550) | about 7 months ago | (#46306583)

Isn't Netflix sufficiently large to be able to afford having direct connections into multiple major ISPs? Obviously, not being US-based, I have little idea how these things work in the US (we're pretty happy with a single peering nod in our tiny little country) but it would at least eliminate the peering disputes.

RE: RE: "Lopside"? (1)

Anonymous Coward | about 7 months ago | (#46306647)

They do offer multiple POPs which will allow direct access as well as caching servers, but due to US ISPs also being in the business of selling similar services and a lack of competition, it makes more financial sense to hamper competition and drive profits to their services. IE Verizon has Redbox which they will gladly sell to consumers.

Re:"Lopsided"? (1)

silas_moeckel (234313) | about 7 months ago | (#46306739)

It's lopsided, Verizon is more and more and eyeball network where people "consume" bandwidth they got there by pricing themselves out of most of the transit market. That puts them in a somewhat unique position that there are a tier 1 peer but should not be. They are trying to leverage the you need access to the eyeballs more then we need access to your content. Regulating Tier 1's is not easy they never really have been anything but self regulated but as they expand/develop they have a conflict of interest. Remove the conflict require the entities to only sell transit without a discount to anybody.

This is what you get with 'net neutrality' (1)

roman_mir (125474) | about 7 months ago | (#46306451)

This is 'net neutrality' for you - if a service is eating up all of the available bandwidth, who is supposed to pay for the degradation in quality of service exactly? Net neutrality is basically a price control and we know what price controls do: create shortages of supply.

Re:This is what you get with 'net neutrality' (1)

AvitarX (172628) | about 7 months ago | (#46306653)

It's not a price control, nothing about net neutrality says they can't charge their consumers for accessing extra data, and that is who they should charge, and it shouldn't matter the source.

Obamaville. We haz it. (-1)

Anonymous Coward | about 7 months ago | (#46306463)

http://www.dailymail.co.uk/news/article-2564858/Baltimores-people-woods-Inside-hidden-homeless-camps-milk-crates-wooden-doors-tarps-outskirts-town.html

And Big Media in the US is like snooze***snooze***snooze! What? Never heard of it. George Bush hates black people! Halibuton!!!

Meanwhile the EBT cards still work... today... good for porn, booze and skittles fer sure.

Oh and it's going to reduce your health insurance costs $2500 a year. Increase or decrease I mean, whatever, not really important because no doctors will be accepting Obamacare insurance anyway.

Fuck you. Pay me.

Re:Obamaville. We haz it. (0)

Anonymous Coward | about 7 months ago | (#46306519)

Was that bushville 8 years ago? Clintonville 16 years ago? Greater Bushville before that? Reganville?

But yeah, homeless people are all Obama's fault.

Re:Obamaville. We haz it. (-1)

Anonymous Coward | about 7 months ago | (#46306591)

A fair point, I will give you that. But the real point is that the media is ignoring this. If it was Bush this would be on every front page newspaper and on ever talk show on TV for weeks and months on end. You know it. I know it.

But it's not Bush, it's Obama, one of the chosen ones, a fellow traveller, the statist they all love and support.

Hence it's crickets.

You do understand this is not news we see, it's just state propoganda.

Depressions and economic collapse is a pretty easy thing to deal with when you are still employed. You are basically fucked if you are not. But hey, this is statism, gotta break a few eggs, right?

Don't worry citizen, *this* bunch of statists know better than all the others that have failed, I'm sure they will get it right this time and utopia is just around the corner.

Now get back to work, you need to pay for this shit.

Re:Obamaville. We haz it. (0)

Anonymous Coward | about 7 months ago | (#46306581)

imho you weigh 350+ pounds

Re:Obamaville. We haz it. (-1)

Anonymous Coward | about 7 months ago | (#46306639)

All those people homeless (and yet no doubt on government assistance to boot) and we see

http://blog.heritage.org/2014/02/06/billions-wasted-flawed-obamacare-exchanges/

"The federal taxpayers have spent over $5 billion to build Obamacare’s exchanges. Unfortunately for taxpayers, these investments have not delivered a good return and are ongoing.

Despite the muddled launch of many exchanges, the federal government is still doling out grant money, giving out over $200 million to nine states last month. Fourteen states and the District of Columbia are running their own Obamacare exchanges and they all received federal grant money to help build them—a total of over $4 billion. California alone received $1 billion to set up its exchange.

Many state-based exchanges have struggled with similar website problems as the federal healthcare.gov site, and some are still barely operational despite being four months into the open enrollment period."

And I'm not saying that the state should just give a whole crap load of money to the homelsee, obviously that would solve nothing. But just think of how absurd this whole thing is, we are in a depression and yet we see millions upon millions, billions of real dollars pissed down the drain - laundered to Obama's and others radical statist crony's if you want to know the truth, and we see homeless and unemployed and welfare and disability and illegal aliens and on and on the graft goes. The circle of life.

And Obama says; "Fuck you. Pay me." Pay up sucka.

Where is your Network Neutrality God now? (1)

SuperKendall (25149) | about 7 months ago | (#46306469)

So how exactly is Network Neutrality going to fix the ACTUAL problems we are having instead of the problems we are pretending we have?

It's not. You can't force a company to buy more bandwidth, unless you want to expand Obamacare to mandate ISP's by a "platinum" bandwidth package from back haul providers.

Incentive Structures (1)

resistant (221968) | about 7 months ago | (#46306481)

Basic incentive structures are a serious problem that will only grow worse over the years. The more successful become edge businesses such as for-profit streaming video providers and monetized social networks that go heavy on large images and video advertising, the more the data carriers will enviously demand a cut of the big pie that other businesses are dividing up.

I wish I knew of a good solution to this problem that still shows reasonable respect for the free market. Perhaps it's time to give heavy federal and state tax incentives to businesses that do absolutely nothing *but* provide consistently reliable, high-speed data-transmission channels. That means no end-user content provision or end-point Internet connections for individuals, businesses or other organizations. Let the innovators scramble to implement gigabit capabilities that end-point ISPs can resell to their customers. No doubt there are sorts of problems with this idea, but my brain has nothing better right now. :/

Re:Incentive Structures (1)

number17 (952777) | about 7 months ago | (#46306767)

Didn't the free market invent bandwidth caps with overage fees to cover those users who download the most content? This was due to some absurd amount of 90% of the traffic was from 5% of the users who were downloading pirated material.

Fast forward 12 years [tribemagazine.com] and its the average user who is sucking down that bandwidth.

Slashdot BETA (0, Offtopic)

giorgist (1208992) | about 7 months ago | (#46306483)

What happened here ? How can I go back to the old Slashdot ?

Re:Slashdot BETA (0)

Anonymous Coward | about 7 months ago | (#46306547)

Click "parent" in a comment and soulskill will come and magically take you back to the old site...

Instructions (0)

Anonymous Coward | about 7 months ago | (#46306627)

Look at the URL for this page. Remove the "-beta". Hit Enter. Rejoice.

And this is how it's supposed to work (3, Informative)

Anonymous Coward | about 7 months ago | (#46306487)

Look, for years and years, peering agreements have been based on the idea that peering between partners would have roughly equal inbound and outbound traffic patterns. When one partner starts pushing more traffic out than in, it signals an imbalance in the connection, and actually the burden.

Let me explain: internet routing is based on autonomous system (AS) hops. Cogent is an Autonomous System, so is Verizon, Level3, etc. If I'm connected to one AS, and I need to send a packet to someone on another AS, the router within my AS attempts to deliver that packet to the closest exit point for that AS. So for example, if I'm sending packets from Cogent to a user on Verizon, then Cogent will offload the transport of the packet to the closest point possible (usually in the same city). The idea being that Cogent doesn't know where the user is on verizon, so they just want to send it to verizon using an Exterior Gateway Protocol (e.g. BGP - which is AS hope based), and then verizon will use their Interior Gateway Protocol (e.g. something like OSPF) to deliver it most efficiently inside of their network.

And therein lies the problem with unequal inbound and outbound peering situations. If Cogent is 80% inbound and 20% outbound with a specific peer, that is usually shouldering the burden for most of the transport distance and cost for that packet. If it's 50/50, then the burden is the same, and they're equal peers.

In the internet routing world there are tier1 providers, tier 2 providers, etc etc based on how many peers they have. But of the tier 1's - not everyone is created equally. Cogent has Tier 1 status, and their POPs are all interconnected, but they don't have as many geographic POPs as say a Verizon. As a result, they dump packets to their peers as local as possible and those packets are routed across the internet by the peers. Their piers on the other hand, for the burden that *should* be cogents have fewer locations that they peer with cogent due to their limited # of POPs. So even for the outbound to cogent traffic, they end up shouldering a disproportionate amount of that traffic transporting it to one of cogent's pops.

And then you have companies like netflix. Netflix buys bandwidth from these low cost tier1 providers, who are low cost because they don't share the transport burden with the real tier 1s. And when congestion happens, and the real tier1's tell cogent "sorry you're out of bounds withe contract because you don't have an even inbound / outbound ratio" cogent decides not to pay the penalties for the uneven traffic patterns. Instead, they let congestion and packet drops happen. The Netflix comes along, and says "Oh yeah, this provider, they're terrible for netflix traffic - they don't upgrade their pipes." They do this knowing full well the economics of internet peering. How do they know full well? Because they don't even want to pay cogent. All the while that they're letting the tier1 take the heat for the crap peering situation, they're approaching said tier1 saying "Hey, nice network you got there. It would be a shame if someone publicized bad information about its performance. Hey, you know how you can prevent that from happening? Join open connect! All you have to do is host our hardware, provide power, and data center space, and cooling, oh and connectivity for free and your network will look great for netflix customers who won't have to suffer through this peering situation you have going on with Cogent."

And now, Netflix, not being a back bone at all, has managed to make you look bad and then tell you in order to look good you should host their hardware and give them free transport... And because /. is /. Netflix comes out looking like the victim here...

RE: And this is how it's supposed to work (0)

Anonymous Coward | about 7 months ago | (#46306569)

Verizon, Comcast, and other last mile providers are far from Tier1. Since Verizon is requesting data from Cogent, they have the obligation to either upgrade or route the traffic through another peer. I would highly doubt that many last mile providers are anything more then just that "eyeball networks" with lopsided traffic to begin with. The old peering models are essential useless when discussing a balanced peering arrangement between and CDN and Eyeball network.

Dispute between companies affecting service (0)

Anonymous Coward | about 7 months ago | (#46306499)

What are their customers, chopped liver?

Cogent, the UDP of ISPs (1)

tokencode (1952944) | about 7 months ago | (#46306555)

Cogent is to ISPs what UDP is to network protocols.

It's not just Netflix Traffic (0)

Anonymous Coward | about 7 months ago | (#46306585)

It is affecting ALL traffic. I have a customer who cannot reach our servers during the afternoon because of the amount of traffic and packet loss. So the customer is getting screwed here because Verizon and Cogent are fighting it out.

We're Surrounded by Morons. (2)

VortexCortex (1117377) | about 7 months ago | (#46306599)

If I gave you a decentralized network capable of surviving nuclear war, routing packets around cities that vanished in moments, and you then built a World Wide Web of Data Silos then I'd be within my rights to call you all fucking morons!

STORE AND FORWARD. How the hell can't you realize this is the decentralized solution that the Internet needed, not a centralized Server / Client cluster fuck? What is collocation? IT'S WHAT YOU GET FOR FREE WITH STORE AND FORWARD, idiots. Oh there's all that Youtube, Netflix, etc., bandwidth? Well, what if I told you I could reduce your peering bandwidth to ONE COPY of each resource? That way if your neighbor recently watched a show or cat video you could download it directly from them or the upstream cache they're connected to? A router should only ever have been one part of the node, the cache is an essential part, as part of any strategy for load distribution. Ah, but you don't need to keep a copy of each resource at each node if you index the data via distributed hash table -- it's not rocket surgery fools.

How else do you think the Space Internet will work? NASA's DTN and even old-timey HAM Packet Radio are smarter about data than the fucking web! Unlike the WWW, they use Store and Forward. HTML allows lightweight dynamic pages to pull in heavy static resources, if only assets had a <... hash="SHA-512/Base64:MDVjMjg4YmY2ZGV...hMGEx==" hmac="Base64:..."> then secure pages could pull in unsecured static resources and browsers could verify their validity without "mixed content" warnings -- Oh but that would mean the architects would have to ACTUALLY KNOW WHAT THE FUCK THEY WERE DOING. I'm surrounded by morons. I mean, you could even just use your HTTP-AUTH proof of knowledge to key your ciphers and eliminate the Certificate Authority MITM, but noooo... Morons, I Say! MORONS!

Re:We're Surrounded by Morons. (2)

SuricouRaven (1897204) | about 7 months ago | (#46306699)

I really couldn't agree more. That would be a great way to do things. It wasn't built in from the beginning because it wasn't practical until recently - storage just cost too much. It isn't being built in now because the existing models work-ish, and no influential organization has been willing to lend their backing to deploying the new technology.

I did have the idea of encoding the hash addresses as a 'magic directory' in HTTP - eg, http://your.server.com/CANary/... [server.com] //filename. That way any browser or software aware of the address form can run a SHA search of reachable caches, while any non-aware software (or if the cache search fails) just interpret it as an HTTP address and get the file the old way. Even if there are no caches in reachable, it'd still save a lot of IMS requests when re-visiting a website.

Making personal or household devices public cache servers is probably not the best idea though. Privacy concerns - it wouldn't take long for someone to write a script that queries a specified cache for the top thousand video hashes from the popular porn sites and see if they have it stored already.

Re:We're Surrounded by Morons. (1)

SuricouRaven (1897204) | about 7 months ago | (#46306713)

Slashdot mangled my example link. Huh. Here's another example:
http://birds-are-nice.me/CANar... [birds-are-nice.me]
Incidentally, if you can tell me what that song is, I'd be very grateful. If it's pre-1963 it's public domain in the UK and I can add it to me 'legal to use' collection.

Bad article or am I a Pollyana? (1)

mythosaz (572040) | about 7 months ago | (#46306615)

Cogent sells transit for an average price of $1.31 per megabit, though large volume users like Netflix get discounts.

Wut?

Netflix offers a colo/CDN bandwidth saver for ISPs (1)

Karl Cocknozzle (514413) | about 7 months ago | (#46306661)

Netflix has a program where they'll colocate some servers containing a content cache on a segment of the ISPs network so that their peering connections aren't getting beaten to death--why wouldn't these companies get involved in such a program other than as a means to squeeze more money from Netflix, their subscribers, or both.

Re:Netflix offers a colo/CDN bandwidth saver for I (2)

sexconker (1179573) | about 7 months ago | (#46306861)

Netflix has a program where they'll colocate some servers containing a content cache on a segment of the ISPs network so that their peering connections aren't getting beaten to death--why wouldn't these companies get involved in such a program other than as a means to squeeze more money from Netflix, their subscribers, or both.

Because you have to agree to Netflix's terms to host those things.
Everything from physical access requirements to the ol' "By the way we may host other, non-Netflix content on these things in the future, and we'll charge people for the privilege, but you'll still have to treat it as Netflix data and not expect any money for carrying it on your network".

Netflix muscled their way into favorable agreements (both with and without those storage boxes at the ISPs) when they trotted out Super HD. Now that they have a lot of those agreements in place, Netflix opened it up so (just about) everyone gets Super HD, and they aren't making any noise over it anymore.

Taking The Other Side This Time (1)

Bob9113 (14996) | about 7 months ago | (#46306679)

Cogent has an agreement with Verizon to exchange traffic â" which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement. Verizon wants more money from Cogent, and one of their negotiating tactics is simply to stop upgrading their infrastructure so that service degrades.

I'm a pretty rabid net neutrality guy, and as a disclaimer I've never worked for ISPs or long haul providers so I may have my head up my ass. But in this case, I'm tending to fall on the Verizon side of the argument. Peering is supposed to be a two-way street, and if it isn't, the peering is subject to negotiation. It may not be fair for Cogent to take the relatively easy money from giant, centralized Netflix outbound data handling and leave Verizon doing the more costly work of parcelling that data out to millions of endpoints without giving Verizon a piece of the action.

The problem of Verizon having a non-competitive (or reduced competition) supply of consumers to negotiate with is still there, and the outcome ultimately will be Netflix paying more for their connectivity, but at least the payment is being coupled to the right transaction.

RE: Taking The Other Side This Time (0)

Anonymous Coward | about 7 months ago | (#46306797)

Quick question. How can you have equal traffic when you have a network on one side that is basically consumers (Verizon) and a network on the other side that is basically suppliers (Cogent)? I'm sure all those DSL customers are serving up a lot of content. Considering Cogent's notorious supply of Porn and now Netflix, it should all just balance out.

Netflix vs Google (2)

careysb (566113) | about 7 months ago | (#46306687)

It seems that ISP's are so concerned with Netflix's bandwidth suck that they try to get away with throttling. What about Google? Supposedly Google's web crawlers account for the largest single chunk of Internet bandwidth. (Ok, educate me)
--
Sent from my IBM 360 mainframe

Fair is fair (0)

Anonymous Coward | about 7 months ago | (#46306705)

So the ISPs want more money, that's fair as long as they pay the content providers for all the content they make available that provides the ISPs with so many customers. That's fair right?

Supply and Demand (1)

recharged95 (782975) | about 7 months ago | (#46306729)

simply to stop upgrading their infrastructure so that service degrades.

And who's going to pay for upgrading their infrastructure? Verizon might as well pass the costs upstream. And with net neutrality, we're going to see more of this.

CDN's? (1)

DoofusOfDeath (636671) | about 7 months ago | (#46306803)

Does anyone know if/why Netflix doesn't just have CDN servers located directly on Verizon's network?

It seems a little moronic to stream stuff from far away, when Netflix's streaming content is a large but basically fixed quantity of data.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>