0

I have a 10Gbps network containing 2 servers (DellPowerEdge 4700r) and a hardware switch ( ARUBA 2930F) in between. The two servers are connected via 10G SFP+ ports to the switch. I send TCP traffic at 10Gbps from one server to another. In the switch configuration, I can either choose to use the OpenFlow pipeline by configuring the OpenFlow instance on the switch or use default L2 switch behavior. When I am using default switch behavior (no OpenFlow), I get a throughput of ~9.4Gbps from one server to another. Also, I don’t see any Rx discards on the ingress port. However, when I use OpenFlow instead (enable OpenFlow instance), and send traffic again, I see thousands of Rx discards on the switch. As a result, the throughput fluctuates between 2Gbps to 8Gbps (standard deviation of 3 Gbps)(RTO hits many times).

Is it the issue with slower packet processing in the OpenFlow pipeline? Has someone else observed it? Does this seem like an issue with the configuration? I have made sure that the controller code and OpenFlow switch configuration is correct.

There are standard and custom pipelines for OpenFlow. I have been using the standard pipeline of OpenFlow. I have read in the documents of the switch here (under OpenFlow custom pipeline -> Performance in custom pipeline model; picture reference added as well) about this behavior regarding the OpenFlow custom pipeline. It says that in OpenFlow custom pipeline: “packets are dropped at the ingress if the pipeline is busy processing too many packets. You can check Rx discards on the switch to obtain such data”. Maybe a similar issue can arise in the standard pipeline as well. It is known that if packets are processed in software, they won't be processed at the wire-rate, i.e. it would take longer to process them and Rx discards might occur. Now, In the OpenFlow pipeline (both standard and custom) we have an option of choosing where we place our flow rules, either in the software table (SRAM) or hardware table (TCAM). All my flow rules of OpenFlow are in the hardware table (TCAM). Shouldn't they be processed at the line rate now and not show any Rx discards?

The server sending traffic is connected to the switch at port 25, the server receiving the traffic is connected at port 27. The configuration on the switch with the OpenFlow standard pipeline setup is as follows:

To show running-configuration on the switch:

leaf1(config)# sh running-config

    Running configuration:

    ; JL253A Configuration Editor; Created on release #WC.16.05.0007
    ; Ver #12:08.1d.9b.3f.bf.bb.ef.7c.59.fc.6b.fb.9f.fc.ff.ff.37.ef:ba
    hostname "leaf1"
    module 1 type jl253a
    max-vlans 2048
    snmp-server community "public" unrestricted
    openflow
       controller-id 1 ip 192.168.0.1 controller-interface vlan 2
       instance "l1"
          listen-port
          member vlan 3
          controller-id 1
          version 1.3 only
          limit software-rate 2000
          max-backoff-interval 10
          pipeline-model standard-match
          packet-in vlan-tagging input-form
          enable
          exit
       enable
       exit
    vlan 1
       name "DEFAULT_VLAN"
       no untagged 1-3,11-13,19-28
       untagged 4-10,14-18
       no ip address
       exit
    vlan 2
       name "c-vlan"
       untagged 19-24
       ip address 192.168.1.1 255.255.0.0
       exit
    vlan 3
       name "s-vlan"
       untagged 1-3,11-13,25-28
       ip address 10.1.0.1 255.0.0.0
       exit

There are 3 tables in the pipleine: Start table, policy table and sw-table-1. Start table and policy table are in hardware (TCAM), while sw-table-1 is in software. Start table with 1 flow rule, policy table with 5 flow rules and then sw-table-1 with 1 flow rule. All packets go to start table directly. The one flow in start table has a go-to-table rule which forwards all packets to the policy table. I have all my rules in the policy table. Then, the sw-table-1 has a rule which drops the packets. This sw-table-1 is never called in the pipeline.

To show flow tables in the standard pipeline:

leaf1(config)# sh openflow instance l1 flow-table

 OpenFlow Instance Flow Table Information

 Table                       Flow     Available Free Miss
 ID    Table Name            Count    Flow Count     Count         Goto Table
 ----- --------------------- -------- -------------- ------------- -------------
 0     Start                 1        NA             0             100
 100   Policy Table          5        NA             722           200
 200   SW Table 1            1        NA             0             *

 * Denotes that the pipeline could end here.

To show flow rules in start table:

leaf1(of-inst-l1)# sh openflow instance l1 flows flow-table 0

 OpenFlow Flow Table

 Flow 1
 Match
  Incoming Port : Any                   Ethernet Type    : Any
  Source MAC    : Any                   Destination MAC  : Any
  Source MAC Mask      : 000000-000000
  Destination MAC Mask : 000000-000000
  VLAN ID       : Any                   VLAN Priority    : Any
  Source IP Address      : Any
  Destination IP Address : Any
  IP Protocol   : Any
  IP ECN        : Any                   IP DSCP          : Any
  Source Port   : Any                   Destination Port : Any
  Source Port Range      : NA
  Destination Port Range : NA
  TCP Flags     : NA
  Custom Match One         : Any
  Custom Match One Mask    : Any
  Custom Match Two         : Any
  Custom Match Two Mask    : Any
  Custom Match Three       : Any
  Custom Match Three Mask  : Any
  Custom Match Four        : Any
  Custom Match Four Mask   : Any
 Attributes
  Priority      : 0                     Duration         : 16 seconds
  Hard Timeout  : 0 seconds             Idle Timeout     : 0 seconds
  Byte Count    : NA                    Packet Count     : NA
  Flow Table ID : 0                     Controller ID    : 1
  Cookie        : 0x0
  Hardware Index: NA
 Instructions
   Goto Table ID                 : 100

To show flow rules in Policy Table:

leaf1(config)# sh openflow instance l1 flows flow-table 100

 OpenFlow Flow Table

 Flow 1
 Match
  Incoming Port : 27                    Ethernet Type    : ARP
  Source MAC    : Any                   Destination MAC  : Any
  Source MAC Mask      : 000000-000000
  Destination MAC Mask : 000000-000000
  VLAN ID       : Any                   VLAN Priority    : Any
  Source Protocol Address   : Any
  Target Protocol Address   : Any
  Source Hardware Address      : Any
  Source Hardware Address Mask : 000000-000000
  Target Hardware Address      : Any
  Target Hardware Address Mask : 000000-000000
  ARP Opcode    : Any
  IP ECN        : Any                   IP DSCP          : Any
  Source Port   : Any                   Destination Port : Any
  Source Port Range      : NA
  Destination Port Range : NA
  TCP Flags     : NA
  TCP Mask      : NA
  Custom Match One         : Any
  Custom Match One Mask    : Any
  Custom Match Two         : Any
  Custom Match Two Mask    : Any
  Custom Match Three       : Any
  Custom Match Three Mask  : Any
  Custom Match Four        : Any
  Custom Match Four Mask   : Any
 Attributes
  Priority      : 1                     Duration         : 5239 seconds
  Hard Timeout  : 0 seconds             Idle Timeout     : 0 seconds
  Byte Count    : NA                    Packet Count     : 0
  Flow Table ID : 100                   Controller ID    : 1
  Cookie        : 0x0
  Hardware Index: 0
 Instructions
   Apply Actions
     Output                      : 25

 Flow 2
 Match
  Incoming Port : 25                    Ethernet Type    : ARP
  Source MAC    : Any                   Destination MAC  : Any
  Source MAC Mask      : 000000-000000
  Destination MAC Mask : 000000-000000
  VLAN ID       : Any                   VLAN Priority    : Any
  Source Protocol Address   : Any
  Target Protocol Address   : Any
  Source Hardware Address      : Any
  Source Hardware Address Mask : 000000-000000
  Target Hardware Address      : Any
  Target Hardware Address Mask : 000000-000000
  ARP Opcode    : Any
  IP ECN        : Any                   IP DSCP          : Any
  Source Port   : Any                   Destination Port : Any
  Source Port Range      : NA
  Destination Port Range : NA
  TCP Flags     : NA
  TCP Mask      : NA
  Custom Match One         : Any
  Custom Match One Mask    : Any
  Custom Match Two         : Any
  Custom Match Two Mask    : Any
  Custom Match Three       : Any
  Custom Match Three Mask  : Any
  Custom Match Four        : Any
  Custom Match Four Mask   : Any
 Attributes
  Priority      : 1                     Duration         : 5239 seconds
  Hard Timeout  : 0 seconds             Idle Timeout     : 0 seconds
  Byte Count    : NA                    Packet Count     : 0
  Flow Table ID : 100                   Controller ID    : 1
  Cookie        : 0x0
  Hardware Index: 0
 Instructions
   Apply Actions
     Output                      : 27

 Flow 3
 Match
  Incoming Port : 27                    Ethernet Type    : Any
  Source MAC    : Any                   Destination MAC  : Any
  Source MAC Mask      : 000000-000000
  Destination MAC Mask : 000000-000000
  VLAN ID       : Any                   VLAN Priority    : Any
  Source IP Address      : Any
  Destination IP Address : Any
  IP Protocol   : Any
  IP ECN        : Any                   IP DSCP          : Any
  Source Port   : Any                   Destination Port : Any
  Source Port Range      : NA
  Destination Port Range : NA
  TCP Flags     : NA
  TCP Mask      : NA
  Custom Match One         : Any
  Custom Match One Mask    : Any
  Custom Match Two         : Any
  Custom Match Two Mask    : Any
  Custom Match Three       : Any
  Custom Match Three Mask  : Any
  Custom Match Four        : Any
  Custom Match Four Mask   : Any
 Attributes
  Priority      : 65500                 Duration         : 5239 seconds
  Hard Timeout  : 0 seconds             Idle Timeout     : 0 seconds
  Byte Count    : NA                    Packet Count     : 17202875
  Flow Table ID : 100                   Controller ID    : 1
  Cookie        : 0x0
  Hardware Index: 0
 Instructions
   Apply Actions
     Output                      : 25

 Flow 4
 Match
  Incoming Port : 25                    Ethernet Type    : Any
  Source MAC    : Any                   Destination MAC  : Any
  Source MAC Mask      : 000000-000000
  Destination MAC Mask : 000000-000000
  VLAN ID       : Any                   VLAN Priority    : Any
  Source IP Address      : Any
  Destination IP Address : Any
  IP Protocol   : Any
  IP ECN        : Any                   IP DSCP          : Any
  Source Port   : Any                   Destination Port : Any
  Source Port Range      : NA
  Destination Port Range : NA
  TCP Flags     : NA
  TCP Mask      : NA
  Custom Match One         : Any
  Custom Match One Mask    : Any
  Custom Match Two         : Any
  Custom Match Two Mask    : Any
  Custom Match Three       : Any
  Custom Match Three Mask  : Any
  Custom Match Four        : Any
  Custom Match Four Mask   : Any
 Attributes
  Priority      : 65500                 Duration         : 5239 seconds
  Hard Timeout  : 0 seconds             Idle Timeout     : 0 seconds
  Byte Count    : NA                    Packet Count     : 498479217
  Flow Table ID : 100                   Controller ID    : 1
  Cookie        : 0x0
  Hardware Index: 0
 Instructions
   Apply Actions
     Output                      : 27

 Flow 5
 Match
  Incoming Port : Any                   Ethernet Type    : Any
  Source MAC    : Any                   Destination MAC  : Any
  Source MAC Mask      : 000000-000000
  Destination MAC Mask : 000000-000000
  VLAN ID       : Any                   VLAN Priority    : Any
  Source IP Address      : Any
  Destination IP Address : Any
  IP Protocol   : Any
  IP ECN        : Any                   IP DSCP          : Any
  Source Port   : Any                   Destination Port : Any
  Source Port Range      : NA
  Destination Port Range : NA
  TCP Flags     : NA
  TCP Mask      : NA
  Custom Match One         : Any
  Custom Match One Mask    : Any
  Custom Match Two         : Any
  Custom Match Two Mask    : Any
  Custom Match Three       : Any
  Custom Match Three Mask  : Any
  Custom Match Four        : Any
  Custom Match Four Mask   : Any
 Attributes
  Priority      : 0                     Duration         : 5239 seconds
  Hard Timeout  : 0 seconds             Idle Timeout     : 0 seconds
  Byte Count    : NA                    Packet Count     : 726
  Flow Table ID : 100                   Controller ID    : 1
  Cookie        : 0x0
  Hardware Index: NA
 Instructions
   Apply Actions
    Controller Port

To show flow rules in sw-table-1:

leaf1(of-inst-l1)# sh openflow instance l1 flows flow-table 200

 OpenFlow Flow Table

 Flow 1
 Match
  Incoming Port : Any                   Ethernet Type    : Any
  Source MAC    : Any                   Destination MAC  : Any
  Source MAC Mask      : 000000-000000
  Destination MAC Mask : 000000-000000
  VLAN ID       : Any                   VLAN Priority    : Any
  Source IP Address      : Any
  Destination IP Address : Any
  IP Protocol   : Any
  IP ECN        : Any                   IP DSCP          : Any
  Source Port   : Any                   Destination Port : Any
  Source Port Range      : NA
  Destination Port Range : NA
  TCP Flags     : NA
  TCP Mask      : NA
  Custom Match One         : Any
  Custom Match One Mask    : Any
  Custom Match Two         : Any
  Custom Match Two Mask    : Any
  Custom Match Three       : Any
  Custom Match Three Mask  : Any
  Custom Match Four        : Any
  Custom Match Four Mask   : Any
 Attributes
  Priority      : 0                     Duration         : 56 seconds
  Hard Timeout  : 0 seconds             Idle Timeout     : 0 seconds
  Byte Count    : 0                     Packet Count     : 0
  Flow Table ID : 200                   Controller ID    : NA
  Cookie        : 0x0
  Hardware Index: NA
 Instructions
    Drop

Now showing packet statistics at each port 25 and 27. You will see thousands of Rx Discards at port 25 when I ran TCP flow for 40 seconds:

leaf1(of-inst-l1)# sh int 25

 Status and Counters - Port Counters for port 25

  Name  :
  MAC Address      : 548028-5c79a7
  Link Status      : Up
  Port Enabled     : Yes
  Totals (Since boot or last clear) :
   Bytes Rx        : 2,260,064,239        Bytes Tx        : 64,489,408
   Unicast Rx      : 26,948,680           Unicast Tx      : 918,509
   Bcast/Mcast Rx  : 46                   Bcast/Mcast Tx  : 2
  Errors (Since boot or last clear) :
   FCS Rx          : 0                    Drops Tx        : 0
   Alignment Rx    : 0                    Collisions Tx   : 0
   Runts Rx        : 0                    Late Colln Tx   : 0
   Giants Rx       : 0                    Excessive Colln : 0
   Total Rx Errors : 0                    Deferred Tx     : 0
  Others (Since boot or last clear) :
   Discard Rx      : 5,704                Out Queue Len   : 0
   Unknown Protos  : 0
  Rates (5 minute weighted average) :
   Total Rx(Kbps) : 2,149,080              Total Tx(Kbps) : 4,320
   Unicast Rx (Pkts/sec) : 174,967        Unicast Tx (Pkts/sec) : 6,433
   B/Mcast Rx (Pkts/sec) : 0              B/Mcast Tx (Pkts/sec) : 0
   Utilization Rx  : 21.49 %              Utilization Tx  : 00.04 %


leaf1(of-inst-l1)# sh int 27

 Status and Counters - Port Counters for port 27

  Name  :
  MAC Address      : 548028-5c79a5
  Link Status      : Up
  Port Enabled     : Yes
  Totals (Since boot or last clear) :
   Bytes Rx        : 64,489,042           Bytes Tx        : 2,251,406,709
   Unicast Rx      : 918,509              Unicast Tx      : 26,948,680
   Bcast/Mcast Rx  : 2                    Bcast/Mcast Tx  : 47
  Errors (Since boot or last clear) :
   FCS Rx          : 0                    Drops Tx        : 0
   Alignment Rx    : 0                    Collisions Tx   : 0
   Runts Rx        : 0                    Late Colln Tx   : 0
   Giants Rx       : 0                    Excessive Colln : 0
   Total Rx Errors : 0                    Deferred Tx     : 0
  Others (Since boot or last clear) :
   Discard Rx      : 0                    Out Queue Len   : 0
   Unknown Protos  : 0
  Rates (5 minute weighted average) :
   Total Rx(Kbps) : 4,256                  Total Tx(Kbps) : 2,118,904
   Unicast Rx (Pkts/sec) : 6,342          Unicast Tx (Pkts/sec) : 172,523
   B/Mcast Rx (Pkts/sec) : 0              B/Mcast Tx (Pkts/sec) : 0
   Utilization Rx  : 00.04 %              Utilization Tx  : 21.18 %

My flow rules are all in hardware, the switch should be able to process packets at the wire-rate, i.e. 10Gbps, and should not show any Rx discards. Any help will be appreciated!

Performance in custom pipeline model

8
  • Regular switching is almost always done in hardware, so it can be wirespeed switching. When you introduce a software layer, you slow down from wirespeed, and you will get discards because the switch cannot process the frames fast enough in software. – Ron Maupin Feb 19 '20 at 17:37
  • Thanks for the reply. I have updated the question. In OpenFlow, we have the option of choosing where we place our flow rules, either in the software table (SRAM) or hardware table (TCAM). All my flow rules of OpenFlow are in the hardware table (TCAM). Shouldn't they be processed at the line rate now and not show any Rx discards? – Ramzah Rehman Feb 20 '20 at 9:16
  • That is not what I meant. I'm referring to the actual switching and decision making are built into the hardware of switches. That is possible because switching is actually pretty simple, and it is based on the destination MAC address. When you add something like your software to change the rules, then you are using software and CPU cycles to do switching. That will never be as fast as hardware switching. – Ron Maupin Feb 20 '20 at 14:11
  • I get your point. In my case, I have all the rules from the controller (software) in the switch's flow table already. The packets would not need to go to the controller. All required flow rules are in the policy table (which is stored in TCAM memory and is a hardware flow table). So, essentially all packets would directly go to the TCAM table. So, if all the rules are in the hardware table, and packets don't need to go to the controller, line-rate performance should be expected (but is not seen). What do you say? – Ramzah Rehman Feb 26 '20 at 12:55
  • No, software is not going to give you line-rate performance, which is why modern switches have gone to hardware switching, even for cheap consumer-grade switches. Routers require software to forward packets, and (unless you have a very expensive router with a high-powered CPU and lots of RAM and hardware assist) that is why routers do not route at wire speed. SDN is basically moving routing to the switch and controller, and you will not get the performance of hardware layer-2 switching. – Ron Maupin Feb 26 '20 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.