3

Overview

I am trying to setup BGP based ECMP to two end servers that sit on separate subnets/VLANs and advertise out the same IP address to a SRX firewall. The end nodes are linux boxes running BIRD and have a simple NGINX web server running on them.

I would like to load balance traffic based on source IP and source port equally over the two end nodes.

Current Status

  • Node 1 is configured on VLAN 56 and uses the private IP address of 192.168.70.2/30 (with .1 being used by the SRX). It has a loopback address of 10.240.0.0/32 (the VIP) and 10.240.0.1/32 which are both advertised out via BGP.
  • Node 2 is configured on VLAN 556 and uses the private IP address of 192.168.70.6/30 (with .5 being used by the SRX). It has a loopback address of 10.240.0.0/32 (the VIP) and 10.240.0.2/32 which are both advertised out via BGP.
  • The SRX is a SRX240 running in a chassis cluster. I had tried with a single SRX100 and a vSRX with the same results. The SRX has 2 interfaces which are both in the trust zone (which has all protocols and services on and a trust to trust policy that permits everything). It has 2 BGP sessions configured in the same group to each of the end nodes.
    • reth6.56 - 192.168.70.1/30
    • reth6.556 - 192.168.70.5/30

show protocols bgp

group www-test {
    type external;
    multipath;
    neighbor 192.168.70.2 {
        peer-as 65001;
        local-as 65000;
    }
    neighbor 192.168.70.6 {
        peer-as 65001;
        local-as 65000;
    }
}

show bgp summary

Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.70.2          65001      16493      15988       0       0  5d 1:28:19 Establ
  inet.0: 2/2/2/0
192.168.70.6          65001      16169      15668       0       1 4d 23:03:17 Establ
  inet.0: 2/2/2/0

I have added the required policy to the fowarding-table to ensure multiple routes are added

policy-options {
    policy-statement backup-routes {
        then {
            load-balance per-packet;
        }
    }
}

routing-options {
    forwarding-table {
        export backup-routes;
    }
}

which shows the correct result in the routing table

show route 10.240.0.0

inet.0: 735 destinations, 771 routes (735 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.240.0.0/32      *[BGP/170] 5d 01:32:57, localpref 100
                      AS path: 65001 I
                      to 192.168.70.6 via reth6.556
                    > to 192.168.70.2 via reth6.56
                    [BGP/170] 4d 23:07:55, localpref 100
                      AS path: 65001 I
                    > to 192.168.70.6 via reth6.556

and also the forwarding table

show route forwarding-table matching 10.240.0.0

Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index NhRef Netif
10.240.0.0/32      user     0                    ulst 262150     3
                              192.168.70.6       ucst  2239     6 reth6.556
                              192.168.70.2       ucst  2283     6 reth6.56

On top of that I also have the layer-3 and layer-4 hash-key forward-options set.

forwarding-options {
    hash-key {
        family inet {
            layer-3;
            layer-4;
        }
    }
}

Which according to Juniper https://www.jnpr.net/documentation/en_US/junos/topics/reference/configuration-statement/family-edit-forwarding-options-hash-key-inet.html is supported on the SRX/vSRX and should add layer 3 and layer 4 information to the hash algorithm.

Relavent bits quoted:

family inet—Incorporate port data into the hash key for flow determination. By default, port data is ignored when determining flows.

layer-3—Incorporate Layer 3 (IP) data into the hash key. You must include the layer-3 statement. If you omit the layer-3 statement, the management process removes the hash-key statement from the configuration and the router behaves as if you specified layer-3. By default, or if you specify only the layer-3 statement, the router uses the following Layer 3 information in the packet header for per-flow load balancing:

  • Source IP address
  • Destination IP address
  • Protocol
  • Incoming interface index

layer-4—Incorporate Layer 4 Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) data into the hash key. If you include the layer-4 statement, the router uses the following Layer 4 information to load-balance:

  • Source port number
  • Destination port number
  • IP type of service

Tests

I ran a series of tests. Firstly with the hash-key options off. I used 120 hosts split over 5 different subnets and my results showed at 51/49% split over the routes. Great. This is equal enough for me. I repeated this 3 times recording my results. I could see that every time every remote host ALWAYS went the same way. Even after making sure the current sessions were cleared and repeating the tests, they went the same way every time.

Next I enabled the layer-3 and layer-4 settings. This should now load balance traffic taking into account the source port the connection was coming from.

I ran a script from my local computer that generated 43 concurrent connections, each one from a different source port and I could see they all went to the same host. (The same host as they went to with the options switch off)

Session ID: 2357, Policy name: DEFAULT/22, State: Active, Timeout: 294, Valid
  In: 192.168.3.253/53745 --> 10.240.0.0/80;tcp, If: reth6.1199, Pkts: 3, Bytes: 206
  Out: 10.240.0.0/80 --> 192.168.3.253/53745;tcp, If: reth6.56, Pkts: 2, Bytes: 92

Session ID: 10588, Policy name: DEFAULT/22, State: Active, Timeout: 294, Valid
  In: 192.168.3.253/53752 --> 10.240.0.0/80;tcp, If: reth6.1199, Pkts: 3, Bytes: 206
  Out: 10.240.0.0/80 --> 192.168.3.253/53752;tcp, If: reth6.56, Pkts: 2, Bytes: 92

My Questions

  1. Has anyone done this before and did they get it working on a SRX?
  2. Have I missed something? According to the examples there isn't much config needed to make this work.
  • I raised this with JTAC who have been investigating. They currently say this is by design (regardless of the layer-4 setting). I will update this with more information when they get back to me – Gareth Hastings Nov 17 '17 at 15:15
1

The answer is:

The setting

forwarding-options {
    hash-key {
        family inet {
            layer-4;
        }
    }
}

only works in PACKET MODE

The default mode for an SRX is flow mode. You can set configure packet mode by deleting ALL of the configuration under the security stanza and then running

set security forwarding-options family mpls mode packet-based

once you have committed that you will need to reboot your device.

You can check and see which mode it's running in by running the operational mode command

show security flow status

This gives the following output

 Flow forwarding mode:
    Inet forwarding mode: packet based
    Inet6 forwarding mode: flow based
    MPLS forwarding mode: packet based
    ISO forwarding mode: drop
  Flow trace status
    Flow tracing status: off
  Flow session distribution
    Distribution mode: RR-based
    GTP-U distribution: Disabled
  Flow ipsec performance acceleration: off
  Flow packet ordering
    Ordering mode: Hardware

You are interested in lines 2 & 3. Below, mine show that the device is in packet mode for IPV4 traffic and flow mode for IPV6.

Inet forwarding mode: packet based
Inet6 forwarding mode: flow based

For more details see here: https://kb.juniper.net/InfoCenter/index?page=content&id=KB30461

Note: It took a JTAC engineer a few weeks to work this out after building it in a lab and testing it. He is going to try and get the documentation updated to note it only works in packet mode!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.