I have encountered a rather unusual problem that I'm having difficulty getting to the bottom of... Here is the configuration:
SMB server ->(2x10Gb SFP+ LAG) Cisco SG500X-24 switch ->(1x10Gb SFP+) Netgear XS716T
The problem
When the SMB client (Mac OS 10.13.6, 1Gbase-T interface, short cable run (<10m Cat-6A) is connected to the Cisco switch, write transfer rate is good ~100MB/s, read is variable going down to 5-6MB/s up to 50-60MB/s - it is unstable. But transferring the cable so that the client is on the downstream Netgear switch, without changing any other parameters, I now have a stable 100MB/s both ways on the same system - this is the performance I would expect. Also 10GBase-T clients on the Netgear have lower than the expected performance (250MB/s read+write, with one 10G client active).
This is completely reproducible and occurs on several clients. Clients on Mac OS 10.14.6 (also 1Gbase-T) have better transfer rates on the Cisco (100GB/s write, 70GB/s rate) but still 30% lower than the available bandwidth.
What I have tried
- Changing cables from the SG500X switch to the client. No change
- Activating and deactivating flow control. Seems too make some difference, but not much
- Removing all other clients from the Cisco switch for test. The problem remains with just the server, the Cisco switch and the client as the only devices present.
- Updating the SG500X switch's firmware to the latest version
- Factory reset and reconfiguration of the SG500X switch
- Exchanging the SG500X switch for identical model to exclude hardware failure
Other configuration details
The SG500X is configured with factory defaults, no QOS configuration, the default 4 queues, L2/L3 (routing activated but not presently used), default VLAN, standalone mode. The only changes made from factory default are : configuration of the subnet, EEE disabled, the switch is put into standalone mode (no stacking), and the LAG to the upstream SMB server configured. MTUs are 1500 throughout. Activating jumbo frame capability on the switch makes no difference as expected.
I can't understand why a placing the client connection downstream drastically improves the performance, which leads me to question some configuration issue on the SG500X. I have not yet had a chance to get in with Wireshark to see if I can see anything.
Does anyone have any clue as to what is going on or suggestions of paths to explore? Any ideas welcome!!
Cisco SG500X-24 configuration file
config-file-header
switchd1395b
v1.4.7.6 / R800_NIK_1_4_194_194
CLI v1.0
set system queues-mode 4
file SSD indicator excluded
@
port jumbo-frame
voice vlan oui-table add 0001e3 Siemens_AG_phone________
voice vlan oui-table add 00036b Cisco_phone_____________
voice vlan oui-table add 00096e Avaya___________________
voice vlan oui-table add 000fe2 H3C_Aolynk______________
voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone
voice vlan oui-table add 00d01e Pingtel_phone___________
voice vlan oui-table add 00e075 Polycom/Veritel_phone___
voice vlan oui-table add 00e0bb 3Com_phone______________
no eee enable
bonjour interface range vlan 1
no qos
hostname switch
passwords aging 0
username cisco password encrypted XXX privilege 15
!
interface vlan 1
ip address 192.168.99.200 255.255.255.0
no ip address dhcp
!
interface gigabitethernet1/1
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/2
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/3
no eee enable
flowcontrol on
no eee lldp enable
no macro auto smartport
!
interface gigabitethernet1/4
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/5
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/6
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/7
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/8
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/9
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/10
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/11
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/12
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/13
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/14
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/15
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/16
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/17
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/18
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/19
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/20
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/21
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/22
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/23
no eee enable
flowcontrol on
no eee lldp enable
!
interface gigabitethernet1/24
no eee enable
flowcontrol on
no eee lldp enable
!
interface tengigabitethernet1/1
flowcontrol on
lacp timeout short
channel-group 1 mode auto
no macro auto smartport
!
interface tengigabitethernet1/2
flowcontrol on
lacp timeout short
no macro auto smartport
!
interface tengigabitethernet1/3
flowcontrol on
lacp timeout short
channel-group 1 mode auto
!
interface tengigabitethernet1/4
lacp timeout short
!
interface Port-channel1
flowcontrol auto
description XXX
no snmp trap link-status
no macro auto smartport
!
interface Port-channel2
flowcontrol auto
!
exit
LAG 1 stats
Bytes Received: 1288915754
Drop Events: 5960612
Packets Received: 2504690512
Broadcast Packets Received: 11768
Multicast Packets Received: 176663
CRC & Align Errors: 0
Undersize Packets: 0
Oversize Packets: 0
Fragments: 0
Jabbers: 0
Collisions: 0
Frames of 64 Bytes: 9823411
Frames of 65 to 127 Bytes: 1072963532
Frames of 128 to 255 Bytes: 118984791
Frames of 256 to 511 Bytes: 18631305
Frames of 512 to 1023 Bytes: 15774615
Frames of 1024 Bytes or More: 1268512858
GE15 stats
Bytes Received: 1609303336
Drop Events: 0
Packets Received: 49604716
Broadcast Packets Received: 2408
Multicast Packets Received: 877
CRC & Align Errors: 0
Undersize Packets: 0
Oversize Packets: 0
Fragments: 0
Jabbers: 0
Collisions: 0
Frames of 64 Bytes: 15092
Frames of 65 to 127 Bytes: 24547933
Frames of 128 to 255 Bytes: 1910884
Frames of 256 to 511 Bytes: 410342
Frames of 512 to 1023 Bytes: 784807
Frames of 1024 Bytes or More: 21935658
Interface statistics (with the all ports active (switch is in use, but has same issue when in a minimal configuration) )
Interface | RX Total Bytes (Octets) | Unicast Packets | Multicast Packets | Broadcast Packets | Packets with Errors | TX Total Bytes (Octets) | Unicast Packets | Multicast Packets | Broadcast Packets |
---|---|---|---|---|---|---|---|---|---|
GE1 | 1914771463 | 8881813 | 967 | 33616 | 0 | 5003196556 | 4388558 | 373018 | 283384 |
GE2 | 2924295671 | 2158000 | 146 | 2131 | 0 | 1376014518 | 1188455 | 4636 | 6253 |
GE3 | 18314582 | 19259 | 1507 | 35 | 0 | 3317050 | 23741 | 1328 | 3413 |
GE4 | 1610193506 | 24779700 | 831 | 291 | 0 | 91587132392 | 66022893 | 216809 | 134416 |
GE5 | 1785931115 | 18347068 | 270 | 257 | 0 | 96968345968 | 66588299 | 181578 | 109479 |
GE6 | 3366871042 | 2673219 | 1247 | 585 | 0 | 1758705961 | 2053922 | 183192 | 113628 |
GE7 | 776696078 | 573832217 | 598 | 513 | 0 | 410038276096 | 435640528 | 192392 | 125422 |
GE8 | 3281916167 | 20044634 | 1114 | 683 | 0 | 49840870438 | 33899587 | 68320 | 52629 |
GE9 | 3147242197 | 93149866 | 449 | 782 | 0 | 2316161273 | 25946416 | 226173 | 140373 |
GE10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
GE11 | 11668814 | 34093 | 90 | 127 | 0 | 24586872 | 30558 | 8734 | 10890 |
GE12 | 1433807334 | 16248509 | 158149 | 62811 | 0 | 47201704133 | 31642035 | 237196 | 258701 |
GE13 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
GE14 | 2560485641 | 13071733 | 510 | 1737 | 0 | 9018751222 | 7543145 | 205019 | 124718 |
GE15 | 1609927785 | 49605076 | 883 | 2511 | 0 | 109001196462 | 79900959 | 240081 | 156721 |
GE16 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
GE17 | 3641387874 | 16345400 | 745 | 513 | 0 | 54112776988 | 38374193 | 194917 | 120167 |
GE18 | 410520499 | 2672004316 | 9140 | 4513 | 0 | 106570486881 | 869744932 | 371227 | 318656 |
GE19 | 4096587109 | 49615757 | 486 | 479 | 0 | 193265149894 | 136865784 | 264575 | 183179 |
GE20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
GE21 | 1837729908 | 2173122 | 5243 | 725 | 0 | 501919494 | 1268902 | 225662 | 147760 |
GE22 | 254816121 | 629899 | 302 | 1104 | 0 | 1623851943 | 1236679 | 379628 | 320703 |
GE23 | 2018388782 | 6819040 | 834 | 708 | 0 | 16700032706 | 11892710 | 50744 | 26999 |
GE24 | 850703938 | 141402143 | 182473 | 152039 | 0 | 314779880945 | 239441934 | 247770 | 170694 |
XG1 | 1963305513 | 1762211013 | 91082 | 12192 | 0 | 1252066021017 | 1340462183 | 354024 | 25595 |
XG2 | 823218861 | 650990080 | 50191 | 5129 | 0 | 918646564630 | 787696151 | 333303 | 318809 |
XG3 | 3625591452 | 742303743 | 90499 | 0 | 0 | 3701977843871 | 2749796466 | 211438 | 287424 |
XG4 | 3157857128 | 93010284 | 26478 | 44712 | 0 | 17513685286 | 19412242 | 353458 | 278958 |
UPDATE (6/4/2021):
Further investigations, including packet traces with Wireshark, reveal packet loss (and TCP retransmissions) when transfer rates fall. The ingress dropped frames counters increase on XG1/1, XG1/3 and Po1 (the LACP LAG) when the throughput issues occur. Although the server's LACP hashing policy was misconfigured (L4 instead of L2), changing this did not remedy the problem, which was expected as I believe there is no requirement for hashing policies to be homogeneous with LACP. I will do one further capture with port mirroring and packet capture on one of the offending ingress ports to take a look, but as all other error counters are zero, I am suspecting that the switch's buffers are not handling the burst rates when the destination port is a 1Gbit local port, which is odd as the system is not under particular strain (the problem remains with 1 server and 1 client) - but a recent server upgrade could have changed something with the burst rates. Although Flow control is activated on the two ports XG1/1 and XG1/3, it is not effective (probably as it dependant on it being activated on the LAG).
#show interfaces status xg1/1
Flow Link Back Mdix
Port Type Duplex Speed Neg ctrl State Pressure Mode
-------- ------------ ------ ----- -------- ---- ----------- -------- -------
te1/1 10G-Fiber Full 10000 Disabled Off Up Disabled Off
#show interfaces configuration xg1/1
Flow Admin Back Mdix
Port Type Duplex Speed Neg control State Pressure Mode
-------- ------------ ------ ----- -------- ------- ----- -------- ----
te1/1 10G-Fiber Full 10000 Disabled On Up Disabled Off
Activating flow-control on the LAG causes complete loss of connectivity on the last attempt by the engineer who has responsibility for the server. I don't quite know why.
As for the improvement when connected to a downstream switch - what I think might be happening is that when the destination port for incoming frames is the 10Gb SFP+ port (that leads to the downstream switch), the frames are being offloaded more quickly to the downstream switch, so the Cisco's buffers fill less and the throughput issues reduce. Does this seem possible?
channel-group 1 mode active
. I would also disable flow control.