Overview
I am trying to setup BGP based ECMP to two end servers that sit on separate subnets/VLANs and advertise out the same IP address to a SRX firewall. The end nodes are linux boxes running BIRD and have a simple NGINX web server running on them.
I would like to load balance traffic based on source IP and source port equally over the two end nodes.
Current Status
- Node 1 is configured on VLAN 56 and uses the private IP address of 192.168.70.2/30 (with .1 being used by the SRX). It has a loopback address of 10.240.0.0/32 (the VIP) and 10.240.0.1/32 which are both advertised out via BGP.
- Node 2 is configured on VLAN 556 and uses the private IP address of 192.168.70.6/30 (with .5 being used by the SRX). It has a loopback address of 10.240.0.0/32 (the VIP) and 10.240.0.2/32 which are both advertised out via BGP.
- The SRX is a SRX240 running in a chassis cluster. I had tried with a single SRX100 and a vSRX with the same results. The SRX has 2 interfaces which are both in the trust zone (which has all protocols and services on and a trust to trust policy that permits everything). It has 2 BGP sessions configured in the same group to each of the end nodes.
- reth6.56 - 192.168.70.1/30
- reth6.556 - 192.168.70.5/30
show protocols bgp
group www-test {
type external;
multipath;
neighbor 192.168.70.2 {
peer-as 65001;
local-as 65000;
}
neighbor 192.168.70.6 {
peer-as 65001;
local-as 65000;
}
}
show bgp summary
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.70.2 65001 16493 15988 0 0 5d 1:28:19 Establ
inet.0: 2/2/2/0
192.168.70.6 65001 16169 15668 0 1 4d 23:03:17 Establ
inet.0: 2/2/2/0
I have added the required policy to the fowarding-table to ensure multiple routes are added
policy-options {
policy-statement backup-routes {
then {
load-balance per-packet;
}
}
}
routing-options {
forwarding-table {
export backup-routes;
}
}
which shows the correct result in the routing table
show route 10.240.0.0
inet.0: 735 destinations, 771 routes (735 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.240.0.0/32 *[BGP/170] 5d 01:32:57, localpref 100
AS path: 65001 I
to 192.168.70.6 via reth6.556
> to 192.168.70.2 via reth6.56
[BGP/170] 4d 23:07:55, localpref 100
AS path: 65001 I
> to 192.168.70.6 via reth6.556
and also the forwarding table
show route forwarding-table matching 10.240.0.0
Routing table: default.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
10.240.0.0/32 user 0 ulst 262150 3
192.168.70.6 ucst 2239 6 reth6.556
192.168.70.2 ucst 2283 6 reth6.56
On top of that I also have the layer-3 and layer-4 hash-key forward-options set.
forwarding-options {
hash-key {
family inet {
layer-3;
layer-4;
}
}
}
Which according to Juniper https://www.jnpr.net/documentation/en_US/junos/topics/reference/configuration-statement/family-edit-forwarding-options-hash-key-inet.html is supported on the SRX/vSRX and should add layer 3 and layer 4 information to the hash algorithm.
Relavent bits quoted:
family inet—Incorporate port data into the hash key for flow determination. By default, port data is ignored when determining flows.
layer-3—Incorporate Layer 3 (IP) data into the hash key. You must include the layer-3 statement. If you omit the layer-3 statement, the management process removes the hash-key statement from the configuration and the router behaves as if you specified layer-3. By default, or if you specify only the layer-3 statement, the router uses the following Layer 3 information in the packet header for per-flow load balancing:
- Source IP address
- Destination IP address
- Protocol
- Incoming interface index
layer-4—Incorporate Layer 4 Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) data into the hash key. If you include the layer-4 statement, the router uses the following Layer 4 information to load-balance:
- Source port number
- Destination port number
- IP type of service
Tests
I ran a series of tests. Firstly with the hash-key options off. I used 120 hosts split over 5 different subnets and my results showed at 51/49% split over the routes. Great. This is equal enough for me. I repeated this 3 times recording my results. I could see that every time every remote host ALWAYS went the same way. Even after making sure the current sessions were cleared and repeating the tests, they went the same way every time.
Next I enabled the layer-3 and layer-4 settings. This should now load balance traffic taking into account the source port the connection was coming from.
I ran a script from my local computer that generated 43 concurrent connections, each one from a different source port and I could see they all went to the same host. (The same host as they went to with the options switch off)
Session ID: 2357, Policy name: DEFAULT/22, State: Active, Timeout: 294, Valid
In: 192.168.3.253/53745 --> 10.240.0.0/80;tcp, If: reth6.1199, Pkts: 3, Bytes: 206
Out: 10.240.0.0/80 --> 192.168.3.253/53745;tcp, If: reth6.56, Pkts: 2, Bytes: 92
Session ID: 10588, Policy name: DEFAULT/22, State: Active, Timeout: 294, Valid
In: 192.168.3.253/53752 --> 10.240.0.0/80;tcp, If: reth6.1199, Pkts: 3, Bytes: 206
Out: 10.240.0.0/80 --> 192.168.3.253/53752;tcp, If: reth6.56, Pkts: 2, Bytes: 92
My Questions
- Has anyone done this before and did they get it working on a SRX?
- Have I missed something? According to the examples there isn't much config needed to make this work.
