tcpdump not capturing all TCP packets when capturing whole packets

Question

I have a server and a client on different machines and on different locations. I'm trying to capture TCP packets with tcpdump on an interface of the server machine. I've injected an identifier in the payload of the packets to I can identify those packets when the output from tcpdump is displayed.

The full command I'm using is:

tcpdump -l  -AnXSs 0 -i eth0 -tt tcp port 7778 | tee tcpdump_test 

If I give tcpdump the option -s 0 (display the whole packet) together with the other options shown above, tcpdump seem to miss packets arriving at the interface.

However, if I give tcpdump the option -s 100 (display the first 100 bytes of the packet) together with the other options above, tcpdump seem to give me all packets that I expect.

The larger packets I send over TCP, the higher ratio of missing packets I experience. An import note is that I receive all tcp packets on the client and the server as I'm expecting, I just can't see all of them arriving on the interface if I'm specifying the flag -s 0 to tcpdump

In the situation when I'm losing packets and turning off tcpdump I get this output from tcpdump:

126639 packets captured                                                                    
1544770 packets received by filter                                                           
1416694 packets dropped by kernel 

However when I turn off tcpdump in the situation when I don't lose packets I get this output from tcpdump:

2006 packets captured
2006 packets received by filter
0 packets dropped by kernel

Why is it not capturing all TCP packets, and what can I do to make it capture all TCP packets?

I'm running it on Linux SUSU with tcpdump version 4.5.1 and libpcap version 1.5.3

Answer 1

Found the solution to my own question with the help and hint from @Guy Harris. The kernel was dropping packets due to that the buffer tcpdump uses got overfull when capturing whole TCP packets.

From tcpdump man page:

packets dropped by kernel (this is the number of packets that were dropped, due to a lack of buffer space, by the packet capture mechanism in the OS on which tcpdump is running, if the OS reports that information to applications; if not, it will be reported as 0).

The kernel puts captured packets in a fixed-size capture buffer. If tcpdump doesn't empty that buffer quickly enough, the kernel will begin overwriting old packets in the buffer and correspondingly incrementing the dropped counter. The value of that counter is what you see as "dropped by kernel".

The capture buffer can be re sized by giving tcpdump the -B option to specify new buffer size in KiB