Packet sizes in a TCP stream

Question

I'm network traffic and wish to divide each TCP session into a series of requests and responses (the protocols I'm working with all work that way, like HTTP or SSL).

I had a simple assumption (ignoring out of order and resent packets) - given a chunk of data that needs to be sent, it will be sent using the largest possible packets and the last packet will be either smaller than the maximum size or be followed by a packet from the other side (ignoring ACK empty packets). So in an HTTP session I expect to see something like (again, disregarding acks) -

Packet 1 - Request "Get..."

Packet 2 - Response, size 1434

Packet 3 - Response, size 1434

Packet 4 - Response, size 1434

Packet 5 - Response, size 500

Which is what I get on most of the sessions, however there's at least one occasion I saw which looked like

Packet 1 - Request "Get..."

Packet 2 - Response, size 1434

Packet 3 - Response, size 1080

Packet 4 - Response, size 1434

Packet 5 - Response, size 500

No retransmissions, out of order packets here or no exceptional delays on the server.

I want to know - what can cause this and when will it occur? How wrong is my assumption?

UPDATE

I put an example pcap file here

UPDATE 2

Including a tshark dump with relevant fields...

$ tshark -r http_1082.pcap -T fields -e frame.number -e frame.len \
    -e ip.src -e ip.dst -e tcp.flags.push -e http.request.method \
    -e http.request.uri -e http.response.code | head -n 47
1     66      192.168.1.103    206.33.49.126    0            
2     62      206.33.49.126    192.168.1.103    0            
3     64      192.168.1.103    206.33.49.126    0            
4     411     192.168.1.103    206.33.49.126    1    GET    /money/.element/script/3.0/video/xmp/xmp_playlistapi.js    
5     54      206.33.49.126    192.168.1.103    0            
6     1434    206.33.49.126    192.168.1.103    0            
7     1434    206.33.49.126    192.168.1.103    0            
8     64      192.168.1.103    206.33.49.126    0            
9     1434    206.33.49.126    192.168.1.103    0            
10    1434    206.33.49.126    192.168.1.103    0            
11    1434    206.33.49.126    192.168.1.103    0            
12    64      192.168.1.103    206.33.49.126    0            
13    1434    206.33.49.126    192.168.1.103    0            
14    1434    206.33.49.126    192.168.1.103    0            
15    1434    206.33.49.126    192.168.1.103    0            
16    1434    206.33.49.126    192.168.1.103    0            
17    64      192.168.1.103    206.33.49.126    0            
18    1434    206.33.49.126    192.168.1.103    0            
19    1434    206.33.49.126    192.168.1.103    0            
20    1434    206.33.49.126    192.168.1.103    0            
21    1434    206.33.49.126    192.168.1.103    0            
22    1434    206.33.49.126    192.168.1.103    0            
23    64      192.168.1.103    206.33.49.126    0            
24    1434    206.33.49.126    192.168.1.103    0            
25    1434    206.33.49.126    192.168.1.103    0            
26    1434    206.33.49.126    192.168.1.103    0            
27    1434    206.33.49.126    192.168.1.103    0            
28    1434    206.33.49.126    192.168.1.103    0            
29    1434    206.33.49.126    192.168.1.103    0            
30    64      192.168.1.103    206.33.49.126    0            
31    1434    206.33.49.126    192.168.1.103    0            
32    1434    206.33.49.126    192.168.1.103    0            
33    1434    206.33.49.126    192.168.1.103    0            
34    1082    206.33.49.126    192.168.1.103    1     <------ Packet in question        
35    1434    206.33.49.126    192.168.1.103    0            
36    1434    206.33.49.126    192.168.1.103    0            
37    1434    206.33.49.126    192.168.1.103    0            
38    64      192.168.1.103    206.33.49.126    0            
39    1434    206.33.49.126    192.168.1.103    0            
40    1434    206.33.49.126    192.168.1.103    0            
41    1434    206.33.49.126    192.168.1.103    0            
42    1434    206.33.49.126    192.168.1.103    0            
43    1434    206.33.49.126    192.168.1.103    0            
44    1434    206.33.49.126    192.168.1.103    0            
45    1434    206.33.49.126    192.168.1.103    0            
46    626     206.33.49.126    192.168.1.103    1            200
47    64      192.168.1.103    206.33.49.126    0 

Answer 1

The TCP layer uses the Nagle algorithm to buffer traffic (it sends fewer large packets, instead of more small packets... making it more efficent); there is a way for the application to say 'send it now'. You see that in the TCP header with a flag called the PSH (push) bit. While the bit is set by the stack, the push is done at the request of the application.

So this is intended and normal behavior.

Answer 2

The packet size depends on how the application and/or the OS buffers and sends network data. If the application and/or the OS decides to send the data after 1080 bytes are in the buffer then the packet will be 1080 bytes (plus headers). There could be many reasons for it to do that. In your case you would have to look in the webserver source code and/or the OS network stack.

Answer 3

Packet size is defined by the OS (in general), and is related to buffers, amount of data provided by the application, etc. Many strategies can be used to achieve maximum performance, and sometimes send smaller packets can be faster than waiting to create a bigger packet.

Sometimes the amount of apps running can demand the OS to be faster (send whatever it has in the buffer so far) instead of saturating the buffer.

Perhaps, you could give us more detail about the scenario you were working with (ex.: server OS, apps running on it).

Answer 4

Fundamentally the problem is the TCP implementation doesn't know what the application is going to do next. When the server application makes a sequence of writes the stack doesn't know whether the writes it has received so-far are the whole sequence or only part of it.

Most of the time the server application makes writes to the buffer quicker than the network stack is able to empty it. So the buffer is full and full-sized packets come out.

But sometimes something else slows the server application down. Maybe waiting for a disk read on an overloaded disk array or something. So the buffer empties and the network stack has to choose between sending out a smaller packet (more overhead) or waiting for data that may never come (adding delay).

Answer 5

If you look at frame 34, you will see that the server has transmitted a 32kB buffer and that the PSH bit is set. If you look at 82 you will see the same, 32 kB from the previous PSH bit. Packet 52 has a PSH bit even though there has been less than 2kB of the response.

The PSH bit is typically set by a TCP stack for the last segment of an application PDU written to the network. So the application uses a 32kB buffer and when there is a lot of data, writes it to the TCP socket 32kB at a time. When there is less data as in frames 51-52, it is because the application wrote out that record first in the response and it was only 1820 bytes.

Notice that the application that I refer to may in fact not be the server application itself but some intermediate software such as a Java Virtual Machine (JVM) or whatever. It's not clear from the data content why that 1820 byte PDU was sent, perhaps a 32kB buffer wasn't available at the time?

The point is that it shouldn't matter, there's no substantive performance penalty.