We are a managed service provider running a small sized network in a single datacentre in Sydney. We've recently deployed a new POP inter-state in Melbourne (both are on the east coast of Australia), and for the first time I am having to face real-world challenges in terms of traffic engineering. My hope is that I can get some guidance here on how to get some level of control over my iBGP paths.
I will likely be posting a few inter-related questions, but in this case I am specifically concerned about internal traffic engineering. I'm finding it surprisingly difficult to figure out how to get iBGP to make optimal routing decisions.
The main goal to me is that need to find a way of giving iBGP some concept of a boundary and a distance per POP. So I can distinguish between a POP that is in the same city, vs one that is inter-state, vs one that is east vs west coast. Then optimise inbound/outbound routing based on this.
I know that there will be a lot of case-by-case scenarios, but I'm hoping I can develop an iBGP routing strategy that works maybe 80% of the time and the rest I would have to deal with special edge cases in the config.
- We have just purchased 4x ASR 1001-Xs to act as our edge devices at each POP (2x per POP but due to switching hardware limitations I am only focusing on deploying 1 edge device in Melbourne for now)
- We also utilise Juniper for switching hardware. EX4500 as our "core switches" and EX4200s at the access layer.
- We now have 2x transit providers. We only interconnect to each provider in one state each.
- AS 1000 is an aggregator and uses AS 4000 as one of it's primary upstreams in Sydney.
- This poses a bit of a challenge as all the paths received by via AS 1000 are typically longer by 1 than those we get from AS 4000.
- I am using Ansible to generate IOS configs using Jinja2 templates. So it's not a problem to generate lots of per iBGP peer route-map logic to get things done.
Essentially, I want to be able to achieve optimal routing between POPs as we deploy them. But right now I am not able to achieve any level of control over how iBGP is choosing it's paths.
My current design
- I currently have 2x ASR1Ks acting as edge routers with full tables in Sydney, and one in Melbourne.
- Both POPs use different transit providers.
- We have a point to point circuit between the two POPs which is terminated on both sides by the edge devices on dot1q sub-interfaces.
- We run OSPF over this link between all the edge devices, and the cost of the link is increased so this is the lowest preference OSPF path.
- We have a single OSPF area 0 across both POPs.
- The edge devices are more of a converged core/edge - our core switches don't do much L3 as they can't handle a full table.
- In each POP the ASR1Ks act as route-reflectors for the other BGP devices in that POP - firewalls, core switches, LNSes and so on.
- Each has their own cluster-ID - not per POP. Looking a changing this to per-POP.
- Each ASR1K originates a default route to route-reflector clients over BGP.
- All the ASR1Ks are in an iBGP mesh.
- All transits have the same local pref at all sites.
Example of sub-optimal routing
- If I have my Melbourne and Sydney transits all online, outbound routing works fine. Sydney traffic exits via Sydney and Melbourne exits via Melbourne.
- The problem is that just by admin-disabling my primary Sydney transit, my Melbourne transit is automatically now preferred. Instead of my secondary Sydney transit via the BDR02 router in Sydney.
- So I end up with a scenario often where the traffic will then bounce over to Melbourne over our backhaul, exit in Melbourne, then route back to Sydney. The path that was incurring < 1ms is now about 30 ms.
To make matters worse, in this particular scenario I can't figure out why Melbourne is being preferred.
- Weight is identical
- Local pref is identical
- AS Path is same length
- Neither path is self originated.
- Both have IGP as the origin.
- Both have metric (MED?) of 0.
- Both are iBGP paths from the perspective of this router.
- IGP metric I would have thought correlates to OSPF link cost as we're using OSPF as our IGP.
- I've confirmed the 100G reference bandwidth is set across all OSPF devices.
EDIT: 30/01: I think that I am wrong about how IGP cost is calculated and perhaps they are currently the same? All my OSPF routes are type E2. If the IGP costs are the same then I suppose it makes sense best path selection happens based on RID, which in this case the RID of the MEL BDR would be lower than SYD.
I have set the OSPF link cost between Sydney to 15,000 much higher than default. I've calculated this to work reliably with our 100 Gbps reference bandwidth.
In terms of OSPF link costs - this is OSPFs preferences of each next-hop of the BGP routes:
bdr-01-syd#sh ip route x.x.201.73 (AS 4000 next hop) Routing entry for x.x.201.72/30 Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 15000 Last update from x.x.13.51 on Port-channel1.1125, 14:57:17 ago Routing Descriptor Blocks: * x.x.13.51, from x.x.13.66, 14:57:17 ago, via Port-channel1.1125 Route metric is 20, traffic share count is 1 bdr-01-syd# bdr-01-syd#sh ip route x.x.31.5 (AS 1000 next hop) Routing entry for x.x.31.4/30 Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 5 Last update from x.x.216.67 on Port-channel1.36, 1d00h ago Routing Descriptor Blocks: * x.x.216.67, from x.x.216.163, 1d12h ago, via Port-channel1.36 Route metric is 20, traffic share count is 1 bdr-01-syd# x.x.201.73 is the next hop to 126.96.36.199 via the Melbourne path. x.x.13.51 is the other end of the inter-state Point to Point. x.x.13.66 is BDR-01-MEL. x.x.31.5 is the next hop to 188.8.131.52 via the Secondary Sydney transit in the same POP as the primary transit - via BDR-02-SYD. x.x.216.67 is the local OSPF VLAN for the Sydney POP that both BDR01 and BDR02 are in. x.x.216.163 is the BDR-02-SYD router.
In terms of these OSPF choices, I can see the shorter OSPF "forward metric" is picked up. I would have thought that BGP should pick the Sydney path based on this.
You can see from this trace that we immediately hop to Melbourne via Backhaul because the first hop is 13ms: (184.108.40.206 is anycasted and has paths in both states).
bdr-01-syd#traceroute 220.127.116.11 Type escape sequence to abort. Tracing the route to 18.104.22.168 VRF info: (vrf in name/id, vrf out name/id) 1 x.x.13.51 13 msec 13 msec 13 msec 2 x.x.201.73 14 msec 14 msec 14 msec 3 x.x.196.54 [AS 4000] [MPLS: Label 25049 Exp 0] 14 msec 14 msec 14 msec 4 x.x.196.51 [AS 4000] 14 msec 14 msec 14 msec 5 22.214.171.124 [AS 1221] 14 msec 15 msec 14 msec 6 126.96.36.199 [AS 1221] 16 msec 14 msec 16 msec 7 188.8.131.52 [AS 1221] 13 msec 14 msec 14 msec bdr-01-syd# bdr-01-syd#sh ip route 184.108.40.206 Routing entry for 220.127.116.11/16 Known via "bgp 5000", distance 200, metric 0 Tag 4000, type internal Last update from x.x.201.73 06:06:14 ago Routing Descriptor Blocks: * x.x.201.73, from x.x.13.66, 06:06:14 ago Route metric is 0, traffic share count is 1 AS Hops 2 Route tag 4000 MPLS label: none bdr-01-syd# bdr-01-syd#sh ip bgp regexp ^1000 1221$ BGP table version is 11307146, local router ID is x.x.216.161 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed, t secondary path, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path ... * i 18.104.22.168 x.x.31.5 0 100 0 1000 1221 i ... Versus the path via AS 4000: bdr-01-syd#sh ip bgp regexp ^4000 1221$ *>i 22.214.171.124 x.x.201.73 0 100 0 4000 1221 i bdr-01-syd#
In this output both the secondary Sydney transit is a valid path, and so is Melbourne transit. Melbourne is picked as best.
bdr-01-syd#sh ip bgp 126.96.36.199 BGP routing table entry for 188.8.131.52/16, version 10794227 Paths: (2 available, best #2, table default) Advertised to update-groups: 66 Refresh Epoch 1 1000 1221, (received & used) x.x.31.5 (metric 20) from x.x.216.163 (x.x.216.163) Origin IGP, metric 0, localpref 100, valid, internal Community: 1000:65110 5000:1000 5000:1001 5000:1002 rx pathid: 0, tx pathid: 0 Refresh Epoch 2 4000 1221, (received & used) x.x.201.73 (metric 20) from x.x.13.66 (x.x.13.66) Origin IGP, metric 0, localpref 100, valid, internal, best Community: 4000:5307 4000:6100 4000:53073 5000:1000 5000:1030 5000:1031 rx pathid: 0, tx pathid: 0x0 bdr-01-syd#
What I've tried
I tried adding an OSPF link cost of 15,000 which I calculated as a safe figure based on my ref bandwidth of 100 Gbps as always being least preferred OSPF cost. I thought this would count as "IGP cost" and yet BGP is still preferring the Melbourne path for some reason.
After this didn't seem to have any impact, my main plan was to use AS PATH prepending on iBGP. The plan was that I would have peer-groups per POP. And in my templating I would designate how many prepends to do, based on how far apart the two POPs were. I had thought this would be a fairly common type of goal.
- 0 prepends if intra-POP
- 1 prepend if intra-state POP
- 2 prepends if inter-state POP
- 3 prepends if east-west coast POP
I thought this would work pretty perfectly, be a pretty elegant solution, and is exactly the type of solution I am hoping to get to. I wrote up the configs in a couple of hours and deployed. But scratched my head until I realised that iBGP does not support AS path prepending.
Even if I could get this working, it seems like it would never be a supported solution.
What I'm considering
- That last link @ ipspace.net mentions that you could use local-pref as it persists inside an AS. But I already have mapped out a local-pref policy to prefer downstream customer routes, IXes, the usual... It seems like using localpref for this would not mix well. And Ivan doesn't suggest it!
- I did consider using BGP Confederations - but this seems like a lot of extra work for our small network. And I also read that it doesn't add AS path hops between confederated ASes anyway. So I'd probably end up in the same spot.
- I would consider using MPLS (I think MPLS TE?) But I'm very green when it comes to MPLS and already have a lot of challenges ahead of me. So I would like to avoid the added complexity, unless it is a good solution to my problem.
I will add more details tomorrow. For now, here's a diagram that depicts our current setup.