8

In an one-armed SLB configuration, SNAT is used to force return-traffic to pass through the SLB. This has one negative: simply that web logs cannot capture the true client IP unless passed in XFF (X-Forwarded-For) header and the web server is able to log.

An alternative is to use PBR (policy based routing) to get the return-traffic back to the SLB, but I try to avoid PBR unless there's no other/better solution On the 6500E platform with SUP720/PFC3B -- and I know the particular IOS version can be a factor too -- does PBR add any latency over doing SNAT assuming PBR is all done in hardware? If PBR is done in hardware using only the commands supported by it today, is it possible upgrading IOS in the future could change PBR to be done in software/process-switched?

Today, our load-balancers have most of the web server VLANs directly behind them -- default g/w pointing to SLB -- and other servers like SQL in non-SLB VLANs. However, this web-sql traffic transits the SLB. Our goal would be to avoid crossing the SLB and just keeping SQL traffic separate, and still retaining the true client in web logs. I'd prefer to not introduce troubleshooting complexity with PBR and possibly having this change from hardware to software processed in the future. Short of the XFF and SNAT mentioned earlier, is PBR the only option here and what's the best way to keep PBR tightly configured?

  • I'm not 100% sure I understand the topology, does your server have two interfaces and you need SNAT so that server can have single static route to return 'production' traffic to SLB facing interface? But SNAT in 1:N configuration implies states, which are always to be avoided, PBR is stateless so it scales much better. – ytti Sep 7 '13 at 5:34
  • 3
    Normally, I advise against the one-armed load-balancer design (or you might call it load-balancer on a stick), and go with a front-side subnet for load-balanced VIPs, and a back-side subnet for the server pools. The servers default through the back-side interface on the load-balancer. This completely eliminates the need for SNAT or PBR. – Mike Pennington Sep 7 '13 at 13:38
  • Aside from my comment about load-balancer topologies... could you add a few diagrams for reference, because some of the question is unclear when you're not sure what the bigger picture looks like. – Mike Pennington Sep 7 '13 at 13:40
  • @ytti, current topology has SLB inline with servers' default g/w pointing to SLB -- single NIC/server. We have essentialy what Mike described with client-side and server-side VLANs on two interfaces of the SLB now, but this is a question about moving to a different topology like one-armed so that server to SQL traffic doesn't have to traverse the SLB and we prefer not to add any complexity to the servers such as dual NICs with server static routes, etc. Understanding the latency of SNAT vs PBR is key as well as other designs I missed (like the Direct Server Return described in an answer). – generalnetworkerror Sep 12 '13 at 7:40
  • What servers are they? In Linux (or *BSD) it's easy to setup so that packet is returned to interface where it came, which is useful in SLB setups (we use this to redundantly connect servers to two disconnected SLB boxes, VIPs are ECMPd so both are hot, but could be even from different vendor as they are unaware of each other). – ytti Sep 12 '13 at 7:57
6

does PBR add any latency over doing SNAT assuming PBR is all done in hardware?

Sup720 supports PBR in HW, the additional latency (if any) is negligible because PBR doesn't add more interface queueing. I think PBR would make things harder than they need to be (and I'm still not even sure whether it would work... the specifics of that option aren't totally clear)

Short of the XFF and SNAT mentioned earlier, is PBR the only option here and what's the best way to keep PBR tightly configured?

PBR is not the only option. Your proposed option is a bit unclear, but PBR normally boils down to nothing more than a fancier way to do static routing.

Typically this is the best topology for load-balanced services that require SQL queries...

  • Put your VIPs on a front-side subnet... 172.16.1.0/24 in the diagram
  • Put your server pools in a back-side subnet... 172.16.2.0/24 in the diagram
  • Put your SQL queries on another interface... 172.16.3.0/24 in the diagram. Add a second interface to all servers that require SQL queries. Make all your SQL queries to addresses on this subnet. This topology works for both Unix and Windows, since you're only relying on ARP or host-routes (depending on your preference) for SQL connectivity.

Diagram:

LB w SQL query network

This topology has additional benefits:

  • It separates SQL interfaces from web traffic, since SQL queries are bursty and may wind up congesting web traffic.
  • It provides different MTUs for your load-balanced services (which usually need to stay at 1500 or lower, if they transit the internet) and your SQL services (which favor jumbo frames).

Any one-armed load-balancer topology is a less desirable option since you wind up cutting your max throughput in half because of the one-armed topology.

EDIT for question about HW vs SW switching on Sup720

This is a deep topic, but I will give the summary version... Sup720 applies an ACL in each direction (ingress / egress) and the ACL must fit into TCAM based on whatever merge algorithm the platform has chosen. Sup720's Feature Manager (i.e. fm) is responsible for mediating the features into TCAM and reporting whether you have a punt adjacency (meaning SW switching), or whether the combination of protocol and direction is switched in HW. To isolate whether

  1. First, identify all ingress and egress Layer3 interfaces that the PBR traffic could transit
  2. Next, examine the output of show fm fie int <L3_intf_name> | i ^Interf|Result|Flow|Config (you must look at both ingress and egress directions for all interfaces in Step 1). Your traffic will be HW switched if the values in CAPS match the values you see below... note that the output of the command I'm using is very similar to what you see in show fm fie summary...

Tx.Core01#sh fm fie int Vl220 | i ^Interf|Result|Flow|Config
Interface Vl220:
 Flowmask conflict status for protocol IP : FIE_FLOWMASK_STATUS_SUCCESS      <--- in HW
 Flowmask conflict status for protocol OTHER : FIE_FLOWMASK_STATUS_SUCCESS   <--- in HW
 Flowmask conflict status for protocol IPV6 : FIE_FLOWMASK_STATUS_SUCCESS    <--- in HW
Interface Vl220 [Ingress]:
 FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT                        <--- in HW
 Features Configured : V4_DEF   - Protocol : IP
 FIE Result for protocol OTHER : FIE_SUCCESS_NO_CONFLICT                     <--- in HW
 Features Configured : OTH_DEF   - Protocol : OTHER
 FIE Result for protocol IPV6 : FIE_SUCCESS_NO_CONFLICT                      <--- in HW
 Features Configured : V6_DEF   - Protocol : IPV6
Interface Vl220 [Egress]:
 No Features Configured
No IP Guardian Feature Configured
No IPv6 Guardian Feature Configured
No QoS Feature Configured
Tx.Core01#

The interface above doesn't show egress output, but that's irrelevant... the output is similar to the Ingress direction. Ricky Micky wrote up an outstanding explanation of 'sh fm fie interface' if you would like more details on the dynamics of TCAM banks / merge results.

  • I had already eliminated this design option as it requires L2 adjacency between the front-end tier and the back-end tier as well as not scaling well for our multiple back-end VLANs and because of the potential need to place a firewall (not transparent) between tiers. However, it's still an excellent option for those who don't have these concerns. Good point about the MTUs. – generalnetworkerror Sep 14 '13 at 7:01
  • Short of consulting documentation of PBR for specific IOS versions to know whether a given PBR feature triggers the need for it to be done in software, can this be determined within IOS? This is what I meant by "tightly configured" for PBR to keep it operating within hardware. – generalnetworkerror Sep 14 '13 at 7:08
  • @generalnetworkerror, what hardware do you want to do PBR on? If Sup720 / Sup2T, it's not so hard to identify whether you're switching in HW vs SW... we need more specifics to help. In fact, if you wouldn't mind a diagram of how this PBR concept works would be really helpful – Mike Pennington Sep 14 '13 at 9:04
  • SUP720/PFC3B... looking to see how you can determine from CLI if a given PBR feature forced this into s/w switching – generalnetworkerror Sep 17 '13 at 1:39
  • @generalnetworkerror, sh fm fie summary... or read my answer for more info... – Mike Pennington Sep 17 '13 at 2:29
1

If your load balancer supports it Direct Server Return would also do what you want. It needs to be supported by your load balancer and there are some operating system concerns. It involves putting 'loopback' interfaces in each server that all have the IP address of the VIP, the load balancer while having the real server addresses only use the MAC address of the real server to forward the packet, since server has the loopback interface with the VIP in it, the server accepts the packet.

You need to consult the doc of the specific LB vendor and your server teams have to be able to manage the virtual adapter (we do not use this feature because of we did not think our automated server provisioning could manage an MS loopback adapter.

But this does not use NAT in the LB and you don't have to do PBR.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.