BGP, multipathing, and possibly OSPF

Question

I'm setting up a second site for redundancy and DR.

The goals are as follows:

• ISP fault-tolerance from either site: HostA, for example, can reach the internet in the event that ISP-A(primary) and ISP-B go down.
• Similarly, servers will continue to be accessible from the internet in the event that 1 or 2 connections are lost.

Notes:

• We already have Router-A connected via BGP among two ISPs. Either ISP can go down and we can continue to function, so assume that the initial BGP/ARIN steps are completed.
• EDIT: sites interconnected via 1 or more 1gbps private lines
• EDIT: Routers are also firewalls in this case

My questions:

• How do I handle the default gateway situation?
  • Do I point clients in their respective sites at their respective router? What if that router loses connectivity?
  • What about asynchronous routing? Say ISP-A is our primary and traffic comes in destined for a server on Site B (who is pointed at Router-B for default GW), won't that cause issues? Ideally we wouldn't have such a situation even if it means designating one router as default GW for all and have it route to the second router in the event it loses it's ISP connectivity.
  • I was thinking I needed something like OSPF to propagate the awareness of ISP link failures, but I'm not clear on what it would help.

Thanks.

Answer 1

This might be a bit long winded, but I'll try to address most of your issues in one go. Wish me luck!

You can have your upstream service providers advertise a default route (0.0.0.0) through BGP to your routers. When you configure OSPF on your BGP routers, you can use the command "default-information originate." As long as you're getting the default route from your ISP (which only happens when you have connectivity to that ISP), you'll advertise the default route into OSPF. As soon as you lose that default route from your ISP, you'll lose the OSPF default route.

You'll need to establish BGP peering between all of your internet routers (the ones connected to your ISPs). This is an internal BGP or iBGP relationship and needs to be a full mesh. Per your diagram, you only have two internet routers, so a full mesh is simple. There are no tricky steps to peer on iBGP versus eBGP; just a simple neighbor statement. When BGP sees that your neighbor has the same AS number, it forms an iBGP relationship instead of an eBGP one.

Preferring one route over another is a bit tricky, particularly where BGP is concerned. Load balancing is done using route maps and traditionally with as-path prepending. There are a few ways to do this. I'd like to add a caveat that if you aren't filtering outgoing route advertisements, you'll end up advertising ISP A's routes to ISP B and C and so forth which will turn you into a transit AS and you'll end up piping some A <-> B, A <-> C, and B <-> C traffic through your network which is probably not what you want.

Here's one go at your load balancing:

! Set an IP access list that matches your BGP-advertised network

ip access-list standard 1 permit a.b.c.d mask a.b.c.d

! Set an IP as-path access list to only allow advertising of YOUR network

ip as-path access-list 1 permit ^$

! Make a route-map for ISP C, match the as-path and access-list above, then make it look
! less appealing than going through ISP A/B

route-map ISP_C 10
match ip address 1
match as-path 1
set as-path prepend <Your AS> <Your AS> <Your AS>

! The more times you prepend your AS to a route, the less desirable it looks, so traffic
! will be more likely to come in via ISP A/B than C.  The last step is to add it to your
! ISP C neighbor statement in BGP

router bgp <Your AS>
neighbor <ISP C> route-map ISP_C out

If you have more than one subnet, you can even things out a little more and use router B for site B and router A for site A by selectively prepending your AS path to individual subnets. Here's an example:

Router A:

ip access-list standard 1
permit <site B subnets>

ip as-path 1 permit ^$

route-map ISP_AB 10
match ip address 1
match as-path 1
set as-path prepend <Your AS> <Your AS>
route-map ISP AB 20
match as-path 1

router bgp <Your AS>
neighbor <ISP A> route-map ISP_AB out
neighbor <ISP B> route-map ISP_AB out

Router B:

ip access-list standard 1
permit <site A subnets>

ip as-path 1 permit ^$

route-map ISP_C 10
match ip address 1
match as-path 1
set as-path prepend <Your AS> <Your AS>
route-map ISP C 20
match as-path 1

router bgp <Your AS>
neighbor <ISP C> route-map ISP_C out

What you've effectively done is make site B's subnets look less appealing coming from Router A than they do from ISP C and make site A's subnets look less appealing coming from Router B than they do from ISP A/B. You might have to play around with your AS path prepending some to get the right amount of prepending in.

I hope this helps! BGP is a bit of a monster, but once you understand the parts, it's fun to play with. I highly recommend the CBT Nuggets series on BGP if you feel a bit shaky on the subject and I always use GNS3 as a testbed to verify big network changes before I implement them.