While the industry is preaching cloud in every market, and users, whether eagerly, reluctantly or unwittingly start falling in line everywhere, few seem to talk about the actual fabric of the Internet and the ever remaining complexity that undeniably sits between a customer and their mission critical cloud services.
As a result there is precious little acceptance left for connectivity issues of any kind; never mind that even a fairly large service provider in the northern outskirts of Europe, can influence the global flow of traffic between backbone providers or alleviate transatlantic congestion, to about the same extent as the nearby conference hotel, where said service provider host their customer training sessions, can keep United Airlines (Only due to a "pressing, unforeseen and otherwise undisclosed matter of passenger safety", of course. I'm sure you understand.) from cancelling the customer team's outbound flight, despite the hotel's obvious and undisputed obligation to make good the non-refundable room reservations.
In slightly simpler terms, customers increasingly expect that service providers take it upon them to ensure continuous service availability and performance, at least beyond the customer's own Internet uplink. With the competition never being further away than one click of the mouse, I feel multihoming is rapidly becoming an absolute requirement for even the smallest provider of SaaS or other Internet dependant service - Save being built on top of third party infrastructure, in which case the same requirement falls to said third party.
Unfortunately, BGP is as complex, cumbersome and scary as ever, and just does not seem like a good fit for a relatively small operation without a team of dedicated wan people. As such, I've been looking hard for a fairly robust, simple and inexpensive solution that will allow dynamic routing via two Internet Service Providers without running full BGP from your own AS.
Now, I think I finally have an idea for a usable design that will at least keep dynamic routing protocols at an arms length, and for the most part, within the respective domain of the two ISPs.
The setup makes use of a couple of notable mechanics of the BGP routing protocol and the F5 Networks BigIP appliance. First out, BGP will give heed to the most specific applicable route object. This allows both ISPs to announce the larger /22 route in addition to a more specific /23 route each, effectively cutting the block into two halfs during normal operation. As the two /23 routes combined account for the entire /22 block; as long as both ISPs are reachable, the /22 routes should be ignored, thus letting traffic flow according to the /23 routes to each of the sites along the green markers.
If, say, the link to ISP 2 goes down, AS 200 should withdraw both routes (or withdraw the small route and downgrade the other, as they can still announce on behalf of it's peer AS 1000) and traffic will start to flow to the /22 route already announced by AS 1000, with any luck, following the red marker.
Since both BigIP appliances are reachable from both ISP gateways in the same layer 2 segment, the rerouted traffic will be bridged across the Lan 2 Lan links, and end up at the same BigIP appliance as before, although they will arrived on one of the two ports facing the Lan 2 Lan rather than the one facing the ISP 2 gateway.
Here the second of he two mentioned mechanics come into play. BigIP will by default transmit replies on the network port where a given request originated, and similarly (or so I hope) direct traffic to the originating gateway. Admittedly, I have not yet tested the part about replies being forwarded to the originating router; I suppose it could be necessary with some sort of rule to update the static gateway route entry, maybe use a simple local routing protocol or perhaps even let a VIP interface on BigIP (or if possible, on the remaining healthy ISP gateway) take over the IP address of the unhealthy ISP gateway.
In this specific layout, the fact that the two ISPs already have direct peering seem to add to the robustness of the solution. Most likely these are already announcing our routes to their respective peers on behalf of each other. This can both help reduce the immediate impact of an outage and speed up propagation of changes necessary to regain full reach-ability. If the AS 200 of ISP 2 becomes mostly unavailable, though somehow maintains connectivity to Site B as well as the BGP session to ISP 1, any traffic destined for ISP 2 that arrives at AS 1000 will simply be forwarded to AS 200 and join the path of the green marker from there.
Of course, if any of these scenarios are reversed and faults occur on the ISP 1 side instead, effects mirroring those previously outlined should present themselves, and traffic should similarly start to flow along the path of the blue marker.
As for the rest of the design, it should be possible, beneficial even, to combine this with DNS round-robin to distribute initial requests across the two sites. The setup is also meant to tolerate failure of any internal component up to and including a full site blackout.
So. Think this should work ok? Any reason it should not be done this way? Did I miss any big gotchas? See any possible improvements? Or perhaps even have an altogether better alternative? I am, as they say, all ears :)
