How to architect a globally distributed website, from an IP address perspective

Question

I would like to have a website that is loaded fast from any point in the world. From my understanding, you need to take advantage of the data center "regions" at places like Google Cloud or AWS. That's as far as my understanding goes.

What is missing is how exactly to implement it (at a high/diagram level here, not at the code level).

For example, on Google Cloud for a specific "project" you have the choice of IPv4 or IPv6, of "Premium" vs. "Standard" ("Traffic traverses Google's high quality global backbone, entering and exiting at Google edge peering points closest to the user." vs. "Traffic enters and exits the Google network at a peering point closest to the Cloud region it's destined for or originated in."), and of Global vs. Regional. If you select IPv6 you are limited to Premium. If you select Global on IPv4 you are limited to Premium. I am not sure how this works, not at the level of detail of Google's specific system, but what generally is going on here. I don't see how a "global" IP address can be better than a regional one, since the regional one is closer to the request source.

On AWS, they don't have a "Global" option, all IP addresses are IPv4 and they are regional.

That was just some background for the main question.

My question is how to architect a system to take advantage of regional data centers. Just generally, at the IP level. I am wondering how it goes from my domain to the regional IP address. Or if it is like Google Cloud's "Global" IP address, how you could then integrate regions into it. Or if that's backwards, how to conceptualize of this. I saw the diagram below, but it only explains how a domain is mapped to a single region-independent IP address. I don't see how regions are actually implemented / come into play.

Basically I would like to know at a high level how I should organize my IP addresses and servers to take advantage of regional data centers. So far my thinking is of having servers in different regions with their own copies of data. But then I get lost when thinking about IP addresses and domains. If I use the AWS model, it seems I reserve an IP per entrypoint server per region. But then I don't see how the domain name figures out which IP/region to select. If I use the global Google Cloud model, I don't see how I can add regional servers.

Answer 1

What you are asking about is called anycast. Basically you assign the same addressing in multiple locations around the world, and the Internet routing protocol (BGP) will know about all of them, but it only installs the one with the best metric (closest, by Internet) into the routing table of a router.

Each router maintains its own routing table. Anycast is simply that the same network is advertised from multiple places. A router receiving multiple advertisements for the same network will choose which advertisement to place in its routing table based on the metrics of the routing protocol. This results in the closest (from the perspective of the routing protocol metrics) destination being used by a router.

For example, the primary way that BGP determines which of multiple destinations is the closest by the number of ASes in the AS_PATH attribute. If a router receives five advertisements for the 8.8.0.0/16 network, it will choose one to place in its routing table, and it will route the traffic destined for 8.8.8.8 to that network. If the site with that network goes down, the entry in its routing table will be withdrawn, and the next closest destination will be placed in its routing table.

Different routers in different locations will each make their own independent determinations as to which destination is closest. Anycast is simply advertising the same network from multiple sites and letting each router determine for itself where to send traffic destined for that network.

Anycast can be used by both IPv4 and IPv6 because it is the routing table and routing protocol that determine where to send traffic.

Answer 2

There are several ways to achieve your goal of "a website that is loaded fast from any point in the world"

Perhaps the simplest and most common is to have multiple domain names. When any of your sites is hit, use geolocation to determine the location of the IP address and redirect the traffic to the closest regional site.

This has the benefit of also enabling you to all the user to override the automatic redirection if they like, as they can send a cookie or flag to tell you 'no, i really want the uk version'

You can also have a single site for your server side code, while keeping content such as images nearer the user and just using different links depending on the users geolocation.

In terms of DNS, there is a thing called anycast routing where the dns server is aware of multiple possible IP addresses it can return and monitors which has the lowest ping/ least hops for the requesting IP address. This does have some downsides though and should be used with caution.

eg. https://aws.amazon.com/blogs/aws/new-aws-global-accelerator-for-availability-and-performance/ or. https://cloud.google.com/dns/

You can also simply program your DNS to return different IPs based on the source of the DNS request. However, you will be seeing the IP address of the users DNS server, rather than the user.