Calculate max transfer speed in LAN network

Question

We have a LAN network between consumers and producers servers. Producers' output path has been mounted in consumers. There is a 10 Gig switch between producers and consumers and the link between each servers (consumer or producer) and the switch is 1 Gig. Each consumers read data from random server with 2 threads.

The question is what is the maximum speed of transfer rate between a consumer and a producer? Now we scored average speed of 3 MB/s. Is this maximum speed? or even is it normal?

And if there is flaw in network design, what is it and how can we make it better?

Answer 1

The maximum throughput on the application level depends slightly on the used transport protocol - UDP adds 8 byte header overhead, TCP adds 20 byte. The IPv4 packet uses 20 bytes and the Ethernet frame 18 byte. On the Ethernet wire, there's an additional overhead of 20 byte.

A maximum sized Ethernet frame can transport 1500 byte, including IP and TCP overhead. This is also the maximum size for the IP packet (MTU). Plus the frame and physical layer overhead, 1538 byte are going over the wire.

All in all, you get 1460 byte user data (TCP) with an overall size of 1538 byte. At gigabit speed, that's 81.274 frames/s and a total TCP throughput of 118.6 MB/s or 119.6 MB/s with UDP (1472 byte user data).

While these are the maximum speeds you can physically achieve, a well-designed infrastructure with proper application support can actually reach them.

To your question: no, 3 MB/s as maximum throughput is not normal for a gigabit speed network. From the calculation above you should easily be able to run 30 times that.

You should try to diagnose the problem:

• sink data as close to the producer as possible
• feed dummy data directly into the consumer
• run a performance test over the entire data path or parts of it (e.g. with iperf)
• check switches and routers on the way for signs of errors (packet drops, FCS error) or congestion (buffer load, CPU load)
• rule out duplex mismatches (increasing FCS errors on the FDX side, (late) collisions on the HDX side)

Answer 2

1Gbs is one Gigabit per second, while 3MB/s is 3 Megabyte per second. (Byte is noted with a capital B, while bit is noted with a lowercase b)

Since 1 byte = 8 bit, you achieve a speed of 24Mb/s over a 1000Mb/s link. This is poor.

When an application transfer data between 2 machines, this data is chunked into small blocks and each block receive additional information needed to reliably transport the block to the intended destination. (think about the envelope in which you put a letter so the actual mail weigh more than the letter).

Due to this overhead, from an application perspective, the actual bandwidth on a 1Gbs link is closer to 800 / 850 Mbs.

24Mbs is much slower than what you should get.

It is likely that the culprit is either on the consumer or producer machine. You should check CPU usage, disk subsystem I/O rate etc...

But you could also check the logs and the interfaces counters on the switch for any abnormal behavior.

To check whether the servers or the network is at fault, you can try running a tool like iperf on both machines (one as server the other as client).

Answer 3

I'd worked in networks for a very long time before I learned that the ethernet "Mbit/sec" speeds are 1,000,000 bits; while obviously we normally think of 2^20 for amounts of bytes. Threw my calculations out by ~5% for many years!

Also: the ratio of data bytes to TCP header varies greatly; previous answer omits mention of segment size and fragmented packets. Though of course I don't disagree with the general conclusion.

I'd start with network sniffer to see the lags between packets: do the servers respond promptly? Are your packets full of 1-byte segments/datagrams?

And while previous answer says B is byte and b is bit, many people are not as meticulous, and many are confused: personally I always write bit/sec and byte/sec to be unambiguous.

Let us know how you got on.

Jonathan.

Answer 4

If you are still not using at least CAT5e (if not CAT6) or higher then you are stuck with physical limitations of old Ethernet cable design; also could be old/damaged cables. Heaven forbid you're using sub-CAT5 which would account for the terrible throughput speeds. I have CAT5e and max out at 100Mbps up&down each way. That's at least double what you claim to have measured.

There are also other factors that can effect the speed such as if there is data being read off of a hard drive (rather than an SSD), for example. Especially if there is file fragmentation on an HDD it will slow things down a lot. If multiple machines are a making HDD requests of a server that can add to the bottleneck. A possible [yet costly] solution: NAS using only SSD wired to the 10G switch with fiber lines 1Gb+ (Bonus points for FibreChannel). If using Cisco switches connected via Ethernet cables you might have luck with an EtherChannel setup between switches which requires multiple parallel Ethernet cables, that is assuming you aren't connecting your switches via fiber optics (SFP/SFP+) already. Slow speeds can also depend on the network topology and how well it can balance network traffic load all around.

A certified networking professional should be able to figure that out pretty easily, IMHO. I'm just an expired CCENT so this is all off of memory from almost a decade ago.