I'm building a test lab to support a networking monitoring application I'm developing.
I have written a simple UDP packet generator and receiver, and I use iperf3 to create some bulk background traffic. My monitoring application looks for the UDP packets generated by my packet generate amidst the noise being generated by iperf3.
This system is using 10G ethernet over fiber with SFP+ transceivers.
Lets call my UDP packet generator machine A, my UDP receiver B, the iperf3 server C and my monitoring machine D.
I have two switches, call them S1 and S2.
My UDP generator/receiver both have APIs so I can clear and request counters and measure that the packets sent by A are received by B, and my monitoring app also has an API so I can count how many packets were received by D and these numbers should all be equal in a good test.
If I run my test where all of my systems are connected to the same switch (S1), such that A is generating my UDP packets and sending about 800mb/s of traffic, AND running the iperf3 client, which is sending 8gbps of traffic and I mirror port A, then the UDP receiver on B and the monitor on D see the same numbers 100% of the time over hundreds of millions of packets.
Now lets say I have a dedicated iperf3 client machine E, and I connect A and E to a second switch (S2)
and generate my packets on A as before, but run the iperf3 client on E, and mirror the trunk port connecting the two switches on S1, B(S1) will reliably receive 100% of the packets from A(S2), while D(S1) will drop anywhere from 1 to a few hundred packets.
I don't understand this. The total throughput on the mirror source is exactly the same in both cases so why would the mirror port be dropping packets when the switches are trunked together, when 100% of the packets from A(S2) are being received by B(S1)? Its not bursty traffic so it seems the queues in the S1 would be facing the same pressure in both cases.
I initially tried this on two cheap 4 port Mikrotik SFP+ switches, then tried on Ubiquiti Aggregation switches and today trying on a Cisco SX550X thinking that some proper Cisco gear would solve it but the performance is roughly the same as the Ubiquiti Aggregation switch.
Can anyone shed light on why the mirror port would work well when all the traffic is coming from a NIC attached source port, but not when the mirror source is connected to the other switch?