11

TLDR Question:

Who pays for traffic when it transits multiple providers


Original Question

I know there are a great many sites out there with this question and answer, but as yet I have not been able to understand them. If I am in CT and want to contact someone (a server) in CA, it is likely that my data will be transferred through at least three ISPs - my ISP, their ISP and some other(s) in between. The same is true if a connection is routed overseas to the UK and then to Germany and so on.

It seems to me too marvelous to be true to say that these companies who have built these huge networks are "just fine" with letting traffic though neither from or to their customers. Especially someone situated in Oklahoma or who ran a US-Europe or US-Asia underseas cable, the majority of the traffic won't be from their customers and there is no way for them to be paid for thru-traffic.

I could go on and on with this, but you can imagine yourselves all the questions and complexities. Maybe it is just relatively cheap to pass along this traffic once it reaches a "major artery" and even to lay underseas cables? Or maybe a little government funding and corporate charity supports the whole thing?

I know this might be the wrong site for this, but I am looking for someone who actually knows something about the topic.

12

EDIT: I forgot to mention this - if you're interested, there have been books written about this topic. I highly recommend Bill Norton's The Internet Peering Playbook. Available in print or digital copies - it's pretty much the de facto text on this kind of stuff.

DISCLAIMER: The examples used here are hypothetical only - I have zero insight into the business relationships of the carriers mentioned in the below post.

The Internet is really controlled by the peering coordinators of the Tier 1 ISP's around the world.

In the Internet connectivity business world, there's transit and peering. There are also various forms of peering (paid peering, settlement-free peering, etc) but for the most part, these are what Internet connectivity as a whole boils down to. Yes, it is complex because when it comes down to it, there is a cost involved with getting your bits from Oklahoma to Germany and back. It's not just technical jargon and routing policy - there are significant economic and business drivers involved as well.

The term "buying transit" just refers to a contract where you're paying a provider money to get bits off of your network and on to someone else's network. Keeping to your example, let's say there are three providers involved with you sending traffic from your local office in Oklahoma to a branch office in Germany. Let's say your local provider is Hurricane Electric. Hurricane Electric gets the traffic to Germany via Level3. Level3 connects to a major German incumbent provider - DTAG, aka Deutsche Telekom AG. DTAG will then get your traffic to its final destination if the local office is buying transit from DTAG, or from someone else that relies on DTAG. So the connectivity would go something like this:

Local office -> HE -> Level3 -> DTAG -> local German ISP -> remote office

HE is not a tier one carrier, so they likely will buy transit from Level3, or may have paid peering with Level3. Level3 and DTAG are both tier one carriers and thus are most certainly peers, and the local German ISP that your remote office connects to would likely buy transit from DTAG.

Forgetting the routing involved and only thinking about the money, one needs to understand traffic ratios. Typically the big tier one carriers (like Level3) will not peer (referring to settlement-free peering here) with others unless the traffic ratios are somewhat even. Unless you're another big carrier, this will almost certainly never be the case. Most local ISP's and content providers are called "eyeball customers", because their ratios are outbound-heavy. Taking that into a business context, Level3's network would be utilized much more than the content provider or local ISP's network. At this point, you're right - Level3 does not benefit from peering with a local provider or a content provider, and it simply wouldn't happen.

However, if you are a very large ISP with a great deal of customers and also a good chunk of originated prefixes, but you and Level3's traffic ratios just aren't quite there yet, then paid peering may be an option - often much cheaper than buying transit.

Now what if you're not a big carrier? Do you benefit from peering with other people, even if they aren't carriers or ISP's? Yes. When you buy transit, you're usually paying for a set cost per megabit, billed at 95% percentile usage rates, with a minimum commitment. If you're a content provider, it's in your best interest to peer with as many people as possible, because this just brings the cost per megabit of your transit down, because when you peer with folks, you have a direct route to their network, and you no longer need to rely on transit for your bits to reach their network.

Also playing into all of this is routing policy of these tier 1 carriers. Typically these carriers will do hot-potato routing. What that means is if Level3 peers with DTAG in multiple geographic locations, Level3 will likely dump off the traffic destined to your remote office at a DTAG peering point as quickly as possible, rather than carrying the traffic on its own network to a DTAG peering point that is closest to the final destination.

You also run into situations where carriers neglect peering ports in order to strong-arm their peers into doing something they want. In other situations in peering relationships, one carrier will simply de-peer with the other carrier, causing devastating impact for a significant portion of Internet users (see Level3 and Cogent).

To answer the original question - the TL;DR answer is all networks involved with getting the traffic from the source to the destination, pay in some way or another. The longer answer is that if you're going to buy transit, make sure you're buying the transit from a reputable carrier. You're not just paying for uptime and availability, you're also paying for the carrier's connectivity as well.

  • Interesting. I did look up Level 3 and Cognet and I finally understand about Net Neutrality. – user1122069 Dec 2 '14 at 16:59
  • If one peer is only downloading and the other peer is only uploading - in terms of thru traffic (as an example) - which side gets the burden to pay? Or is it roughly equivalent? – user1122069 Dec 3 '14 at 2:36
  • Thanks for the definitions provided here. I was scouring the internet for a definition explaining what an "eyeball network" was until I found this. The other stuff you included helped fill in a few additional gaps also. – SeldomNeedy Nov 9 '15 at 5:28
  • You're very welcome! Grab Bill Norton's book as well. Goes even more in depth and provides fantastic strategies if you're a company that connects to the Internet in any way shape or form. – John Jensen Nov 9 '15 at 23:07
7

Who pays for traffic when it transits multiple providers?

Ultimately, you do.

John Jensen did a fantastic job explaining how this works overall, but I suspect a diagram could help. I will borrow the example providers he used.

Let's assume you have a web server colo'd in an HE.net facility; assume you pay HE.net $500/month for a 100Mbps co-colocation. All HE.net's transit and upstream provider's fees are passed through to HE.net's customers.

When someone on Teo L.T's network pulls a page from your server, the traffic has to go through HE.net, (potentially) Level(3), DTAG, and Teo LT.

Sample topology

As a Tier 2 provider, HE.net needs to buy transit from Level(3) (Link {D}) and HE.net also buys bandwidth from an IXP (Link {E}). Those transit / IXP fees are paid for by HE.net's customers and any other revenue streams that HE.net might have (such as selling off some HE.net spare fiber capacity).

An imaginary breakdown of who pays who for peering, transit and IXP colo is shown below...

Link | Link Type | Buyer      | Seller
---- + --------- + ---------- + ----------
{A}    Peering     N/A          N/A
{B}    Peering     N/A          N/A
{C}    IXP         Level(3)     IXP
{D}    Transit     HE.net       Level(3)
{E}    IXP         HE.net       IXP
{F}    IXP         DTAG         IXP
{G}    Transit     Teo L.T.     DTAG
{H}    Transit     Google       Level(3)
{I}    Transit     Google       DTAG

Caveat: This diagram may have no resemblance to real peering / transit / IXP relationships; purely for purposes of illustration.

  • (It's just "LT", as in the country code...) I'm curious though, what would it look like with research networks like GÉANT & LITNET? Is it mostly the same? – grawity Dec 1 '14 at 15:43
  • 1
    How would you expect this to change without net neutrality, if at all? – generalnetworkerror Dec 2 '14 at 19:17
  • 1
    It means very little (from what I understand now), except that certain companies would pay more for their bandwidth because they "use a lot of it". It's just a gimmick for the cable company to charge you twice - once for the high bandwidth connection and the second time (indirectly) when you use it. – user1122069 Dec 3 '14 at 2:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.