Why is IPv6 broken?

It's broken, first and foremost, because not all network providers who claim to be tier 1 are tier 1.

Even worse, some of these providers run 6to4 relays or providers to home users. A user has no choice which provider is running their 6to4 relay…so, they might end up using a relay that is run by a provider who doesn't peer with their intended destination. I don't think the IETF saw that one coming. But the result is to make 6to4 even more broken. Now, I know some people want 6to4 to die, but while it still exists in some form, user experience is worse than it could be. The temporary fix is for any provider to run their own 6to4 relay for their own customers (assuming that they themselves have full connectivity).

Right now, unless you buy transit from multiple tier 1s, and do so with carefully chosen tier ones, you have only part of the IPv6 internet. Many tier 1s are unsuitable even as backup connections, since you still want your backup connection to have access to the whole internet! Good tier 2 providers might be an excellent choice, sine good providers have already done this leg work and can monitor their providers for compliance.

– by Bob Network at http://mailman.nanog.org/pipermail/nanog/2011-July/038149.html on 2011-07-09

Routing table size has nothing to do with completeness of routes. Google may be one route, through aggregation. And SmallCo may advertise a large route through one provider, and, due to traffic engineering, a smaller route through a second one - in many cases, anyone that had the large route would be able to contact SmallCo, even without the smaller route being present. So routing table size doesn't work. In addition, some providers aggregate their routing tables to reduce routing load and such. Others intentionally don't or deaggregate it intentionally so that they can brag about having bigger routing tables. What you need to ask is: “How many /64s can you get to from your network, and how many of these /64s are reachable from at least one other major provider (you don't care about internal-only networks, after all)?” They can give you that information, but many won't want to.

It's also not about technical people not getting along. It's about business players trying to make money, but not just that either. It's also about ensuring that providers don't end up assuming more than their share of costs for a link. Just because you have a common peering point doesn't mean that turning peering on would reduce your costs. In some cases it may increase costs tremendously, particularly on your long haul backbone links, because the other party would like to take advantage of an attitude of trust on the internet.

That's why we end up with peering policies and contracts.

Let's take Hurricane. This is no different than other providers…basically, they want to say, “We shouldn't need to pay for IPv6 transit from anyone.” This is what Cogent said on IPv4 a few years ago. Google used to say this too for IPv6, not sure if they are still saying it. Basically, “We know we're big enough that you won't want to screw your users by not peering with us.”

A small network couldn't do this tactic - a 100 node network who said to the IPv4 tier 1s: “Hey, I'm in the Podunk Internet Exchange, so are you, so I'm going to peer from you so I don't have to buy any bandwidth for my web server (placed in the Podunk exchange).” Sure, they would like to - it would save a ton of money if their site got lots of hits. I mean, who wouldn't want free connectivity?

In IPv6, we're going through what we settled years ago in IPv4 - who has to pay who to connect. After all, even free peering connections have a cost in manpower, debugging, traffic engineering, documentation, etc.

Some players who aren't getting free interconnection to tier 1s in IPv4 want to get it in IPv6. So they've worked to attract lots of users, and done so under the guise of “We like IPv6 and want to promote it.” Others have not bothered with trying to attract the users, but have said, “We're too big for you to not want to give us connectivity for free, since it would piss off your users if you don't” (Google did this at one point in the past, may still be doing it). The Google example is basically trying to use a monopoly position to force business decisions.

Now, HE, Google, and others would want you to think, “Hey, IPv6 is all new, and these $#@! other providers just want to make a buck on something they have no right to.” Well, perhaps. But what they aren't saying is, “We can turn on BGP for IPv6 on our existing connections to other providers, with no cost to us, and actually have full connectivity.” The issue isn't about cost today - nobody is charging extra for IPv6 in addition to IPv4 on a pipe where you already buy IPv4 bandwidth. And Google and HE already buy IPv4 bandwidth. What they are thinking of is the future, 15 years from now, when there is no IPv4 - in that future, IPv6 isn't insignificant bandwidth, it's everything. Wouldn't it be nice to be a tier 1 and not pay for that? Of course! And certainly one can argue for or against the current tier 1 club's exclusivity. But it's the way the internet works right now, for better or worse. In the meantime, in pursuit of this future, today's customers are screwed by these providers trying to position themselves to make more profit margin down the road.

Which is better for the customer? A system where they are screwed today so that their provider can have a better negotiating position in business discussions OR a position where they do whatever they have to take to provide the customer with full connectivity? (To HE's credit, they are giving away transit today on IPv6, so it's not like you are losing anything of value by not having the full internet routing tables, but it's a huge reason to not pay HE anything in other services, such as data center colocation - go with a provider that you pay and which gives you what you pay for - full transit).

Lots of people who aren't running big networks don't understand peering. They think, “Doesn't this benefit everyone if everyone exchanges traffic?” Maybe, on a pure level, but the business doesn't work that way.

I'll give you an example. Let's say you are a little ISP, and located in Virginia, near a major peering point. You say, “All the tier 1s are there, I can pull fiber to that peering point, which is only a block away, and have free internet, other than the cost of the line.” So, let's say you run the line, and, let's say that all the tier 1s agree to let you peer for free, since they want your traffic too. Now, let's say your user downloads 1,000 TB from a server in California, on Qwest's network.

You paid, let's say, $15,000 for your piece of fiber going a block. You needed to hire contractors and buy permits and such, after all. So you shared in the costs of letting the user get to the server. What did Qwest pay? Well, they dug trenches, pulled fiber, negotiated with cities, counties, and states, paid taxes on their work, lit this fiber, etc. It cost a lot because they went a lot further than your one block. And a lot more than $15,000.

You say, “So what! Their customer benefits too!” That's true, but let's go a bit further. Let's say you have a network that extends to California - you by DS3s from Sprint to do it. There's some cost in that, but your user in Virginia would need more bandwidth than your DS3s. So you decide NOT to peer in California, just in Virginia. That way you don't have to upgrade your lines for your Virginia user. Maybe you even legally break your company into two entities, so that you can peer in California and Virginia both, but you can say with a straight face, “We only have Virginia offices for this user - the other company is a separate entity, and not the entity that owns either the server or the end user.”

In other words, you found a way to shift most of the traffic burden and infrastructure costs to Qwest, away from your user.

This is why Qwest has some sort of peering policy. Among other things, it will require multiple exchange points, and Qwest will probably say they will send traffic to the closest peering point, to minimize their costs. You get to do the same (more on that later).

Let's say that you currently buy bandwidth from NTT - you're not big enough to get free peering from everyone, but Qwest agrees to peer with you. Of course Qwest and NTT also have a business relationship, to give each other free peering. If Qwest gives you and many other customers free peering, however, you'll send less traffic across NTT's network. That might be good from a technical standpoint, but NTT now is selling you a smaller pipe - and making less money. In effect, Qwest undercut NTT's business and lowered NTT's profits on the connection. How will NTT respond to that, when they were also giving free peering to and from Qwest? Well, they might decide that Qwest isn't a very nice partner and tell Qwest, “Pay us for transit or get lost.” That could be ugly - both NTT and Qwest could lose, but Qwest, if they actually care about stable service, won't want to risk it. So generally you don't give peering to anyone who is a customer of one of your free peers. You don't hurt their business. In fact, it's often a requirement in the peering connection, legally. (that said, you could argue whether or not there is an abuse of monopoly here…that's a different issue)

Going one further, let's say you have the server, and Qwest has the end-user. That doesn't change anything - the economics are still such that Qwest has the cost, you don't. That said, it's convention that the person receiving the traffic pays for most of the backhaul.

What's the path between your host and a remote server? How do you find it? If you said “traceroute”, you might be right, but are probably wrong. You need to trace route both sides.

Every provider on the internet is trying to minimize costs. This means that you want traffic to leave your network and go to the destination network with as little distance traveled as possible, because costs go up with distance. It's cheaper to increase the size of pipes within a city to get to a peering point than to increase your backbone pipe size. So, peering contracts typically specify that you dump traffic to the peer as soon as possible. That means the person receiving the traffic generally pays more. It also means that any traffic that crosses an AS boundary almost certainly travels a completely different path each way. In many cases, one third party provider may be used in one direction, another in the other direction. So seeing packet loss on your traceroute at some random tinet router doesn't mean that this router is the cause of any problem, since the return path for that packet from that provider's router might actually cross yet another network that is never transited in either direction for your network connection. (I'm ignoring that most large providers also don't always send ICMP reliably BECAUSE they limit this intentionally to spare the router CPU from overload - it takes router CPU to generate an ICMP TTL exceeded, but it doesn't take router CPU to forward a packet - so traceroute or ping indicating loss at a router doesn't mean anything in itself - the path itself likely has zero percent loss).

So, here's the scenerio.

Let's say a user and a server are on two seperate networks, U (user) and S (server).

Let's say they both utilize transit provider T. So the path could be: U – T – S. S buys an OC12 from T, while U buys a T1.

But let's say that the user has a second transit provider, BIG, who is a free peer of T. He bought an OC3 from BIG. So there's another path between U and S: U – BIG – T – S. Likely this path is much faster than U – T – S.

So, the path for the traffic to S goes U – T – S.

Now, what path does the traffic from T's router go, when T's router generates an ICMP TTL exceeded in response from a traceroute from a user? Does it go straight over the T1 line, or does it go over the peering connection to BIG and then to the customer? The answer, it turns out, depends on network configuration and policy. Let's say it goes out over the T1, but the T1 is congested. It will look like the congestion is at the connection between BIG and T, because this is the first hop that will show packet loss. BUT…the congestion is actually at the U's connection to T, which is irrelevant to the actual traffic path between U and S. So the user, at this point, calls up BIG and T and bitches about “Your peering congestion is congested” when the real problem is that traffic completely unrelated to the user's problem is going via a congested path that is never used for connectivity between U and S.

If you add several providers into this loop, you can end up with a situation where traffic uses Sprint in one direction, but never hits a Sprint router in the other. This is actually very common. A user with slow downloads might be experiencing packet loss on the path from server to user, but not the other way around. In other words, the problem is a provider that never shows up on the user's traceroute!

Remember that the providers hand off the traffic as soon as possible to their peer. So, whoever receives the larger amount of traffic needs the bigger cross-country (or trans-oceanic) links. If one side transmits a T1's worth of data, the other side transmits an OC48's worth of data, only one needs the OC48s across the country - the one receiving the traffic. That's why you hear about “traffic ratios”. If the traffic is even both ways, both sides have to pay for the same amount of cross-country infrastructure to carry that traffic. So most providers won't peer with someone for free that sends, say, 10 times the amount of traffic that they will receive. It would end up costing a lot of money

Some providers want to do away with traffic ratio policies, mutliple location peering, not providing free services to the other's customers, etc.

THAT is why you can't ping some sites from your HE tunnel. It's not just that providers won't peer. It's also that providers have rules to keep themselves from getting screwed.

Certainly, there's ways around some of this (for example, traffic ratios - if I make sure my network is used for the cross-country traffic I send, not yours, then I've addressed that concern at a bit of increased expense for myself). But it's generally not worth doing until the size of the providers is sufficiently large. Other things don't have a good technical fix, like not peering with your peer's customer - that's a business rule.