While the industry is preaching cloud in every market, and users, whether eagerly, reluctantly or unwittingly start falling in line everywhere, few seem to talk about the actual fabric of the Internet and the ever remaining complexity that undeniably sits between a customer and their mission critical cloud services.
As a result there is precious little acceptance left for connectivity issues of any kind; never mind that even a fairly large service provider in the northern outskirts of Europe, can influence the global flow of traffic between backbone providers or alleviate transatlantic congestion, to about the same extent as the nearby conference hotel, where said service provider host their customer training sessions, can keep United Airlines (Only due to a "pressing, unforeseen and otherwise undisclosed matter of passenger safety", of course. I'm sure you understand.) from cancelling the customer team's outbound flight, despite the hotel's obvious and undisputed obligation to make good the non-refundable room reservations.
In slightly simpler terms, customers increasingly expect that service providers take it upon them to ensure continuous service availability and performance, at least beyond the customer's own Internet uplink. With the competition never being further away than one click of the mouse, I feel multihoming is rapidly becoming an absolute requirement for even the smallest provider of SaaS or other Internet dependant service - Save being built on top of third party infrastructure, in which case the same requirement falls to said third party.
Unfortunately, BGP is as complex, cumbersome and scary as ever, and just does not seem like a good fit for a relatively small operation without a team of dedicated wan people. As such, I've been looking hard for a fairly robust, simple and inexpensive solution that will allow dynamic routing via two Internet Service Providers without running full BGP from your own AS.
Now, I think I finally have an idea for a usable design that will at least keep dynamic routing protocols at an arms length, and for the most part, within the respective domain of the two ISPs.
The setup makes use of a couple of notable mechanics of the
BGP routing protocol and the
F5 Networks BigIP appliance. First out, BGP will give heed to the most specific applicable route object. This allows both ISPs to announce the larger
/22 route in addition to a more specific
/23 route each, effectively cutting the block into two halfs during normal operation. As the two
/23 routes combined account for the entire
/22 block; as long as both ISPs are reachable, the
/22 routes should be ignored, thus letting traffic flow according to the
/23 routes to each of the sites along the green markers.
If, say, the link to
ISP 2 goes down, AS 200 should withdraw both routes (or withdraw the small route and downgrade the other, as they can still announce on behalf of it's peer
AS 1000) and traffic will start to flow to the
/22 route already announced by
AS 1000, with any luck, following the red marker.
BigIP appliances are reachable from both
ISP gateways in the same layer 2 segment, the rerouted traffic will be bridged across the
Lan 2 Lan links, and end up at the same
BigIP appliance as before, although they will arrived on one of the two ports facing the
Lan 2 Lan rather than the one facing the
ISP 2 gateway.
Here the second of he two mentioned mechanics come into play.
BigIP will by default transmit replies on the network port where a given request originated, and similarly (or so I hope) direct traffic to the originating gateway. Admittedly, I have not yet tested the part about replies being forwarded to the originating router; I suppose it could be necessary with some sort of rule to update the
static gateway route entry, maybe use a simple local routing protocol or perhaps even let a
VIP interface on
BigIP (or if possible, on the remaining healthy
ISP gateway) take over the IP address of the unhealthy
In this specific layout, the fact that the two ISPs already have direct peering seem to add to the robustness of the solution. Most likely these are already announcing our routes to their respective peers on behalf of each other. This can both help reduce the immediate impact of an outage and speed up propagation of changes necessary to regain full reach-ability. If the
AS 200 of
ISP 2 becomes mostly unavailable, though somehow maintains connectivity to
Site B as well as the
BGP session to
ISP 1, any traffic destined for
ISP 2 that arrives at
AS 1000 will simply be forwarded to
AS 200 and join the path of the green marker from there.
Of course, if any of these scenarios are reversed and faults occur on the
ISP 1 side instead, effects mirroring those previously outlined should present themselves, and traffic should similarly start to flow along the path of the blue marker.
As for the rest of the design, it should be possible, beneficial even, to combine this with
DNS round-robin to distribute initial requests across the two sites. The setup is also meant to
tolerate failure of any internal component up to and including a full site blackout.
So. Think this should work ok? Any reason it should not be done this way? Did I miss any big gotchas? See any possible improvements? Or perhaps even have an altogether better alternative? I am, as they say, all ears :)