Situation is like this:
http client ----> corporate firewall ----> http server
Due to a keepalive, server and client would keep TCP connections open and the client would use a connection pool for HTTP requests.
The firewall has a rule to "kill" long-standing TCP connections after 1 hour. The problem is that our HTTP client would not detect that TCP connection was destroyed and it tried to reuse essentially dead connections which on our side looked like the client "hanged" after a period of time. A request would hang, then the next one would work, presumably because a new connection was established.
The question here is what is the mechanism with which the firewall is killing TCP connections in a way that our HTTP client was unable to detect them. I tried to reproduce this behavior locally in a few ways:
- Kill TCP connections on our vyos router, Wireshark on client side captured TCP FIN-ACK. OK
- Kill TCP connection client side in TCPView on Windows, Wireshark detected TCP RST on the client side. OK
- Block port after established connection to the client-side firewall, resulted in socket reset exception. OK
I have a Wireshark dump on server side and I tried to find if firewall sends a FIN or RST with
ip.dst==serverip && (tcp.flags.reset==1 || tcp.flags.fin==1) but nothing showed up.
Additionally, Wireshark capture on client side shows the problem as HTTP request going out, followed by a dozen of TCP retransmissions, ultimately not going anywhere.
HTTP client is a Java native and/or Jetty HTTP client (tried both), both failed to detect a dead TCP connection. I'd like to reproduce the behavior locally but I am unable to figure out in what dodgy way the firewall is killing the connections, therefore looking for possible answers.