1

I have encountered a rather unusual problem that I'm having difficulty getting to the bottom of... Here is the configuration:

SMB server ->(2x10Gb SFP+ LAG) Cisco SG500X-24 switch ->(1x10Gb SFP+) Netgear XS716T

The problem

When the SMB client (Mac OS 10.13.6, 1Gbase-T interface, short cable run (<10m Cat-6A) is connected to the Cisco switch, write transfer rate is good ~100MB/s, read is variable going down to 5-6MB/s up to 50-60MB/s - it is unstable. But transferring the cable so that the client is on the downstream Netgear switch, without changing any other parameters, I now have a stable 100MB/s both ways on the same system - this is the performance I would expect. Also 10GBase-T clients on the Netgear have lower than the expected performance (250MB/s read+write, with one 10G client active).

This is completely reproducible and occurs on several clients. Clients on Mac OS 10.14.6 (also 1Gbase-T) have better transfer rates on the Cisco (100GB/s write, 70GB/s rate) but still 30% lower than the available bandwidth.

What I have tried

  • Changing cables from the SG500X switch to the client. No change
  • Activating and deactivating flow control. Seems too make some difference, but not much
  • Removing all other clients from the Cisco switch for test. The problem remains with just the server, the Cisco switch and the client as the only devices present.
  • Updating the SG500X switch's firmware to the latest version
  • Factory reset and reconfiguration of the SG500X switch
  • Exchanging the SG500X switch for identical model to exclude hardware failure

Other configuration details

The SG500X is configured with factory defaults, no QOS configuration, the default 4 queues, L2/L3 (routing activated but not presently used), default VLAN, standalone mode. The only changes made from factory default are : configuration of the subnet, EEE disabled, the switch is put into standalone mode (no stacking), and the LAG to the upstream SMB server configured. MTUs are 1500 throughout. Activating jumbo frame capability on the switch makes no difference as expected.

I can't understand why a placing the client connection downstream drastically improves the performance, which leads me to question some configuration issue on the SG500X. I have not yet had a chance to get in with Wireshark to see if I can see anything.

Does anyone have any clue as to what is going on or suggestions of paths to explore? Any ideas welcome!!

Cisco SG500X-24 configuration file

config-file-header
switchd1395b
v1.4.7.6 / R800_NIK_1_4_194_194
CLI v1.0
set system queues-mode 4 

file SSD indicator excluded
@
port jumbo-frame
voice vlan oui-table add 0001e3 Siemens_AG_phone________
voice vlan oui-table add 00036b Cisco_phone_____________
voice vlan oui-table add 00096e Avaya___________________
voice vlan oui-table add 000fe2 H3C_Aolynk______________
voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone
voice vlan oui-table add 00d01e Pingtel_phone___________
voice vlan oui-table add 00e075 Polycom/Veritel_phone___
voice vlan oui-table add 00e0bb 3Com_phone______________
no eee enable 
bonjour interface range vlan 1
no qos 
hostname switch
passwords aging 0 
username cisco password encrypted XXX privilege 15 
!
interface vlan 1
 ip address 192.168.99.200 255.255.255.0 
 no ip address dhcp 
!
interface gigabitethernet1/1
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/2
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/3
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
 no macro auto smartport 
!
interface gigabitethernet1/4
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/5
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/6
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/7
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/8
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/9
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/10
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/11
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/12
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/13
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/14
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/15
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/16
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/17
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/18
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/19
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/20
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/21
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/22
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/23
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface gigabitethernet1/24
 no eee enable 
 flowcontrol on 
 no eee lldp enable 
!
interface tengigabitethernet1/1
 flowcontrol on 
 lacp timeout short 
 channel-group 1 mode auto 
 no macro auto smartport 
!
interface tengigabitethernet1/2
 flowcontrol on 
 lacp timeout short 
 no macro auto smartport 
!
interface tengigabitethernet1/3
 flowcontrol on 
 lacp timeout short 
 channel-group 1 mode auto 
!
interface tengigabitethernet1/4
 lacp timeout short 
!
interface Port-channel1
 flowcontrol auto 
 description XXX
 no snmp trap link-status 
 no macro auto smartport 
!
interface Port-channel2
 flowcontrol auto 
!
exit

LAG 1 stats

Bytes Received: 1288915754
Drop Events:    5960612
Packets Received:   2504690512
Broadcast Packets Received: 11768
Multicast Packets Received: 176663
CRC & Align Errors: 0
Undersize Packets:  0
Oversize Packets:   0
Fragments:  0
Jabbers:    0
Collisions: 0
Frames of 64 Bytes: 9823411
Frames of 65 to 127 Bytes:  1072963532
Frames of 128 to 255 Bytes: 118984791
Frames of 256 to 511 Bytes: 18631305
Frames of 512 to 1023 Bytes:    15774615
Frames of 1024 Bytes or More:   1268512858

GE15 stats
Bytes Received: 1609303336
Drop Events:    0
Packets Received:   49604716
Broadcast Packets Received: 2408
Multicast Packets Received: 877
CRC & Align Errors: 0
Undersize Packets:  0
Oversize Packets:   0
Fragments:  0
Jabbers:    0
Collisions: 0
Frames of 64 Bytes: 15092
Frames of 65 to 127 Bytes:  24547933
Frames of 128 to 255 Bytes: 1910884
Frames of 256 to 511 Bytes: 410342
Frames of 512 to 1023 Bytes:    784807
Frames of 1024 Bytes or More:   21935658

Interface statistics (with the all ports active (switch is in use, but has same issue when in a minimal configuration) )

Interface RX Total Bytes (Octets) Unicast Packets Multicast Packets Broadcast Packets Packets with Errors TX Total Bytes (Octets) Unicast Packets Multicast Packets Broadcast Packets
GE1 1914771463 8881813 967 33616 0 5003196556 4388558 373018 283384
GE2 2924295671 2158000 146 2131 0 1376014518 1188455 4636 6253
GE3 18314582 19259 1507 35 0 3317050 23741 1328 3413
GE4 1610193506 24779700 831 291 0 91587132392 66022893 216809 134416
GE5 1785931115 18347068 270 257 0 96968345968 66588299 181578 109479
GE6 3366871042 2673219 1247 585 0 1758705961 2053922 183192 113628
GE7 776696078 573832217 598 513 0 410038276096 435640528 192392 125422
GE8 3281916167 20044634 1114 683 0 49840870438 33899587 68320 52629
GE9 3147242197 93149866 449 782 0 2316161273 25946416 226173 140373
GE10 0 0 0 0 0 0 0 0 0
GE11 11668814 34093 90 127 0 24586872 30558 8734 10890
GE12 1433807334 16248509 158149 62811 0 47201704133 31642035 237196 258701
GE13 0 0 0 0 0 0 0 0 0
GE14 2560485641 13071733 510 1737 0 9018751222 7543145 205019 124718
GE15 1609927785 49605076 883 2511 0 109001196462 79900959 240081 156721
GE16 0 0 0 0 0 0 0 0 0
GE17 3641387874 16345400 745 513 0 54112776988 38374193 194917 120167
GE18 410520499 2672004316 9140 4513 0 106570486881 869744932 371227 318656
GE19 4096587109 49615757 486 479 0 193265149894 136865784 264575 183179
GE20 0 0 0 0 0 0 0 0 0
GE21 1837729908 2173122 5243 725 0 501919494 1268902 225662 147760
GE22 254816121 629899 302 1104 0 1623851943 1236679 379628 320703
GE23 2018388782 6819040 834 708 0 16700032706 11892710 50744 26999
GE24 850703938 141402143 182473 152039 0 314779880945 239441934 247770 170694
XG1 1963305513 1762211013 91082 12192 0 1252066021017 1340462183 354024 25595
XG2 823218861 650990080 50191 5129 0 918646564630 787696151 333303 318809
XG3 3625591452 742303743 90499 0 0 3701977843871 2749796466 211438 287424
XG4 3157857128 93010284 26478 44712 0 17513685286 19412242 353458 278958

UPDATE (6/4/2021):

Further investigations, including packet traces with Wireshark, reveal packet loss (and TCP retransmissions) when transfer rates fall. The ingress dropped frames counters increase on XG1/1, XG1/3 and Po1 (the LACP LAG) when the throughput issues occur. Although the server's LACP hashing policy was misconfigured (L4 instead of L2), changing this did not remedy the problem, which was expected as I believe there is no requirement for hashing policies to be homogeneous with LACP. I will do one further capture with port mirroring and packet capture on one of the offending ingress ports to take a look, but as all other error counters are zero, I am suspecting that the switch's buffers are not handling the burst rates when the destination port is a 1Gbit local port, which is odd as the system is not under particular strain (the problem remains with 1 server and 1 client) - but a recent server upgrade could have changed something with the burst rates. Although Flow control is activated on the two ports XG1/1 and XG1/3, it is not effective (probably as it dependant on it being activated on the LAG).

#show interfaces status xg1/1
                                            Flow Link          Back   Mdix
Port     Type         Duplex  Speed Neg      ctrl State       Pressure Mode
-------- ------------ ------  ----- -------- ---- ----------- -------- -------
te1/1    10G-Fiber    Full    10000 Disabled Off  Up          Disabled Off   

#show interfaces configuration xg1/1
                                               Flow    Admin     Back   Mdix
Port     Type         Duplex  Speed  Neg      control  State   Pressure Mode
-------- ------------ ------  -----  -------- -------  -----   -------- ----
te1/1    10G-Fiber    Full    10000  Disabled On       Up      Disabled Off

Activating flow-control on the LAG causes complete loss of connectivity on the last attempt by the engineer who has responsibility for the server. I don't quite know why.

As for the improvement when connected to a downstream switch - what I think might be happening is that when the destination port for incoming frames is the 10Gb SFP+ port (that leads to the downstream switch), the frames are being offloaded more quickly to the downstream switch, so the Cisco's buffers fill less and the throughput issues reduce. Does this seem possible?

Improve this question
9
  • Unfortunately, your question is simply too broad to do anything but speculate, which is off-topic here. We have no configurations, debugs, logs, etc. You can refer to the Network Engineering Question Checklist for guidance, then edit you question.
    – Ron Maupin ♦
    Apr 2, 2021 at 18:18
  • It would also help to show the interface statistics for the switch. Also, is the server configured for LACP because the channel group on the switch is set to PAgP, not LACP, so that would not form a channel on the switch?
    – Ron Maupin ♦
    Apr 2, 2021 at 18:51
  • You should configure the switch to use LACP : channel-group 1 mode active. I would also disable flow control.
    – Ron Maupin ♦
    Apr 2, 2021 at 20:59
  • @RonMaupin Thank you for the suggestion! I certainly want LACP and not PAgP, didn't notice this. I will try that tomorrow when I am on site and post back here. I might have difficulty disabling flow control as I have no direct control over the server which has bidirectional flow control activated. This was previously deactivated (with the same issue present) and was only activated during recent troubleshooting
    – januszb
    Apr 2, 2021 at 21:05
  • At lest disable flow control on the switch. It is often problematic and poorly supported. It can be used end-to-end between hosts, and no reason to involve the switch.
    – Ron Maupin ♦
    Apr 2, 2021 at 21:07

2 Answers 2

Reset to default
1

Problem solved and cause found.

The LAG was working fine, and my inkling on burst packet rates was close to the mark. The problem also occurred with only a single link - no LAG. The problem was reproducible on another switch with similar buffer characteristics (Netgear S3300 - which has a 12MB buffer like the Cisco SG500X). Flow control between the server and the switch (specifically IEEE 802.3x pause frames) avoided the problem but at the expense of 15-20% lower throughput and potential head-of-line blocking which was undesirable.

It turned out that a recent server upgrade (for the record OmniOS r151036) activated large segment offload (LSO) on outbound traffic on the Intel NIC by default. Intel specifically advises activating flow control when using LSO, I suspect because of potential high burst packet rates. Deactivating both outbound LSO and flow-control restored normal operation and throughput. The variation between client systems seemed to due to variations in efficiency in handling the dropped packets and ensuing retransmissions, and was secondary to the root of the problem, namely overrun on of the ingress buffers on the switch due to high burst packet rates with outbound LSO activated.

Improve this answer
0

Wired as I had exactly the same issue: slow read speed from clients (12 Mbps), but write speed is normal (+750 Mbps) It turned out a connection issue from the SG350X Cisco switch SFP+ 10 Gb port trunked to the SG350 1Gb Lan port. I tried every possible config with no luck (flow control, manually setting the port speed to 1Gb...) The only fix was using the regular RJ45 10 Gb ports or 1 Gb ports to trunk the SG350X to the SG350

I also so reports of changing SMB server Sockets to No Delay, but not in my case.

Just posting as I found this thread while I had the same issue on a Cisco Switch !

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.