5

With anycast, it would be possible to have the same IP address at multiple locations in the world, without the need of a proxy or VPN that forwards the packets to its final destination.

If you would just use IP spoofing to change the source IP address to for example 6.7.8.9, normally you would not be able to just receive the response packets for this IP address, as the Border Gateway Protocol determines a different path back for the response packet that you normally won't be able to intercept.

What does anycast do so you can reliably receive IP packets meant for this particular IP address? Since the shortest paths BGP relies on can change, which mechanism ensures that when you send a request at location A that has IP address 6.7.8.9, you will receive the entire response back to location A and you won't receive a response with also a few packets delivered at location B that has IP address 6.7.8.9 and another few packets delivered at location C that also has IP address 6.7.8.9?

For DNS it might not matter, but Anycast is also used by CloudFlare for the content delivery network, which would be HTTP(S) over TCP and not just UDP Packets.

Edit:

How about this scenario: Server A and B have the same IP address with anycast. Server B connects to an external server that is very close to server A and very far away from server B, since server A is closer, the response will be routed back to server A, isn't it possible to always route this type of requests back to server B, since server B initiated the connection to the external server?

1
  • Did any answer help you? if so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could post and accept your own answer.
    – Ron Maupin
    Jan 5, 2021 at 23:03

2 Answers 2

5

...which mechanism ensures that when you send a request at location A that has IP address 6.7.8.9, you won't receive a response with also a few packets delivered at location B that has IP address 6.7.8.9 and another few packets delivered at location C that also has IP address 6.7.8.9?

That's really the point of anycast. When you send something to an anycast IP address, the routing protocol will route the traffic to the nearest (according to the routing protocol) network for that address. If there is a problem where a link to one of the networks goes down, the traffic will automatically be rerouted to the next nearest site. You, as the sender, don't care which one gets the traffic. That's why it is so useful for things like DNS and NTP.

It's not like an individual can just start advertising a network on the public Internet. Someone advertising a network on the public Internet better own that network. If you have a BGP relationship with ISP(s), they will find out if you are advertising a network you don't own, and you will be summarily cut off from BGP peering with the ISP(s).

Anycast is also used by CloudFlare for the content delivery network, which would be HTTP(S) over TCP and not just UDP Packets. If you are in middle of uploading some data and the data get scattered over different servers, would the only option be retransmitting it and hoping it goes to the right server next time or does anycast have an option to prevent this?

TCP cannot get scattered over different servers. TCP is a connection-oriented protocol, and it creates a connection. If a link to a server to which you have a connection goes down, you would need to create a new connection to the next server in line. Anycast is nothing more than using the routing protocol to get the traffic to the nearest network with that network address.

If anycast is used and if the path to server A is the shortest path for one fraction of a second and then the path to server B is the shortest path for another fraction of a second and then the path to A is the shortest again for another fraction of a second, etc. it would get scattered over different servers. It takes several seconds before it goes to the timeout state and TCP would require several retransmissions attempts to be made before it goes to timeout.

That would be a serious network problem if the routing changes that often, especially on the Internet. BGP will notice flapping, and it will penalize the flapping link, and the link will get dropped from routing updates. You would be hard-pressed to find what you fear in the real world. ISPs take stability very seriously, and they will block partnerships which display instability.

How about this scenario: Server A and B have the same IP address with anycast. Server B connects to an external server that is very close to server A and very far away from server B, since server A is closer, the response will be routed back to server A, isn't it possible to always route this type of requests back to server B, since server B initiated the connection to the external server?

You are really trying to make this a lot more complicated than it is. You don't initiate a connection from an anycast address to any old address on the Internet. Anycast addresses are destination addresses. If a host with an anycast address needs to contact a different host on the Internet, it uses a non-anycast address, either with a secondary address on the interface, or, more likely, a different interface. Don't confuse the anycast address with the host itself.


Seriously, there are some very, very smart people who do nothing all day but think about this sort of thing. RFCs must be peer approved before they are released as proposed. If nobody has any real objections, they are promoted to RFCs. The whole world has these under a microscope from the very inception of the idea, through, and after the release of the RFCs. If there really was a flaw in anycast, it would have been discovered, and solutions proposed, years ago.

There are multiple RFCs for anycast: RFC 1546, Host Anycasting Service (November 1993), RFC 4786, Operation of Anycast Services (December 2006), and RFC 7094 , Architectural Considerations of IP Anycast (January 2014). These RFCs will explain how Anycast works.

7
  • Anycast is also used by CloudFlare for the content delivery network, which would be HTTP(S) over TCP and not just UDP Packets. If you are in middle of uploading some data and the data get scattered over different servers, would the only option be retransmitting it and hoping it goes to the right server next time or does anycast have an option to prevent this?
    – Jomad
    Aug 8, 2016 at 1:19
  • TCP can't get scattered over different servers. TCP is a connection-oriented protocol, and it creates a connection. If a link to a server to which you have a connection goes down, the connection will time out. You would need to create a new connection to the next server in line. Anycast is nothing more than using the routing protocol to get the traffic to the nearest network with that network address.
    – Ron Maupin
    Aug 8, 2016 at 1:22
  • If anycast is used and if the path to server A is the shortest path for one fraction of a second and then the path to server B is the shortest path for another fraction of a second and then the path to A is the shortest again for another fraction of a second, etc. it would get scattered over different servers. It takes several seconds before it goes to the timeout state and TCP would require several retransmissions attempts to be made before it goes to timeout.
    – Jomad
    Aug 8, 2016 at 1:31
  • That would be a serious network problem if the routing changes that often, especially on the Internet. BGP will notice flapping, and it will penalize the flapping link, and the link will get dropped from routing updates. You would be hard-pressed to find what you fear in the real world. ISPs take stability very seriously, and they will block partnerships which display instability.
    – Ron Maupin
    Aug 8, 2016 at 1:34
  • How about this scenario: Server A and B have the same IP address with anycast. Server B connects to an external server that is very close to server A and very far away from server B, since server A is closer, the response will be routed back to server A, isn't it possible to always route this type of requests back to server B, since server B initiated the connection to the external server?
    – Jomad
    Aug 8, 2016 at 1:59
3

Ron did an absolutely wonderful job of explaining some of the concepts here. This post mentioned the fact that CloudFlare use Anycast, I work for CloudFlare so I'd like to explain how it works reliably.

Firstly it's important to remember that network traffic is re-routed reasonably infrequently and if it happens in your region, it will likely happen when you aren't even awake. As a protocol TCP is quite resilient, so you won't see any ill-effects as a result of this.

TCP cannot be scattered, as a protocol that simply won't work. TCP is an ACK-based protocol; this means that TCP packets are acknowledged by sending back a packet with an acknowledgement bit set. The TCP Header may then contain a Acknowledgment number (of 32 bits in length) which effectively acts as an acknowledgment receipt of all the previous bytes; if this is inaccurate then a connection close may be sent from the server to the client. In TCP, when the RST flag in a TCP header is set. A TCP Reset effectively leads to the connection being immediately killed.

If a TCP connection was scattered and re-routed mid-communication; the new server would not understand the ACK being sent to it with the request and would effectively close the connection to the client. The client would then be able to re-attempt the request which would then be completed by the new server they would be routed to through BGP.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.