How would a firewall kill a TCP connection without RST or FIN? [closed]

Question

Situation is like this:

http client ----> corporate firewall ----> http server

Due to a keepalive, server and client would keep TCP connections open and the client would use a connection pool for HTTP requests.

The firewall has a rule to "kill" long-standing TCP connections after 1 hour. The problem is that our HTTP client would not detect that TCP connection was destroyed and it tried to reuse essentially dead connections which on our side looked like the client "hanged" after a period of time. A request would hang, then the next one would work, presumably because a new connection was established.

The question here is what is the mechanism with which the firewall is killing TCP connections in a way that our HTTP client was unable to detect them. I tried to reproduce this behavior locally in a few ways:

1. Kill TCP connections on our vyos router, Wireshark on client side captured TCP FIN-ACK. OK
2. Kill TCP connection client side in TCPView on Windows, Wireshark detected TCP RST on the client side. OK
3. Block port after established connection to the client-side firewall, resulted in socket reset exception. OK

I have a Wireshark dump on server side and I tried to find if firewall sends a FIN or RST with ip.dst==serverip && (tcp.flags.reset==1 || tcp.flags.fin==1) but nothing showed up.

Additionally, Wireshark capture on client side shows the problem as HTTP request going out, followed by a dozen of TCP retransmissions, ultimately not going anywhere.

HTTP client is a Java native and/or Jetty HTTP client (tried both), both failed to detect a dead TCP connection. I'd like to reproduce the behavior locally but I am unable to figure out in what dodgy way the firewall is killing the connections, therefore looking for possible answers.

Answer 1

You don't mention the kind of firewall, but I suspect most simply drop the packets.

I have a Wireshark dump on server side and I tried to find if firewall sends a FIN or RST with ip.dst==serverip && (tcp.flags.reset==1 || tcp.flags.fin==1) but nothing showed up.

Which would tend to confirm this.

Answer 2

Most likely the firewall just dropped the packet without sending an RST packet, probably after hitting a session timeout value of some sort. This is typically configurable behaviour.

I personally prefer having that RST packet sent precisely because it helps clients behave normally, but I have heard arguments to the effect that this should not be done on externally-facing firewalls to avoid providing any kind of feedback to potential attackers.

I have seen this cause quite a few issues because clients typically don't handle this kind of scenario very elegantly. Essentially, they keep retrying through the original TCP session (which is now dead) and never try to re-establish a new one. Eventually a client-side timeout triggers and the user gets a nasty error message. Setting up HTTP keepalive appropriately for the app can help to fix this.

Answer 3

@Ron Trunk is exactly correct, almost certainly the open connection is being dropped either actively (deny rule inserted) or passively (removed from known connections and not allowed to be recreated without a syn). One of the comments suggested trying it out yourself. Here is a recipe for doing so using linux network namespaces. It assumes that ip forwarding is enabled in your host's kernel, you are root, and probably other things.

# Create network namespacs
ip netns add one; ip netns add two; ip netns add three
# Create interfaces between namespaces
ip link add dev i12 type veth peer name i21
ip link add dev i32 type veth peer name i23
# Bring interfaces up and assign them to respective namespaces
ip link set dev i12 netns one up
ip link set dev i21 netns two up
ip link set dev i32 netns three up
ip link set dev i23 netns two up
# Assign IP addresses
ip netns exec one ip addr add 1.1.1.1/24 dev i12
ip netns exec two ip addr add 1.1.1.2/24 dev i21
ip netns exec three ip addr add 3.3.3.3/24 dev i32
ip netns exec two ip addr add 3.3.3.2/24 dev i23
# Add routes when necessary
ip netns exec one ip route add default via 1.1.1.2
ip netns exec three ip route add default via 3.3.3.2

You then need three windows/shells/screens/terminals. Run each command below in a distinct terminal:

• Start listening on server: ip netns exec three socat TCP-LISTEN:5001 STDIO
• Start transmitting on client: ip netns exec one socat STDIO TCP:3.3.3.3:5001

Note that after running these commands, everything you type in one window will be reflected in the other, and vise-versa (after hitting return). If that isn't true, you might need to enable ip forwarding.

• Instantiate the deny rule: ip netns exec two iptables -I FORWARD -j DROP

Then nothing you type will be allowed through.

You can simulate a less active drop method with (untested) forward rules like:

ip netns exec two iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
ip netns exec two iptables -A FORWARD -m tcp -p tcp -dport 5001 --tcp-flags ALL SYN -j ACCEPT
ip netns exec two iptables -A FORWARD -j DROP

See https://unix.stackexchange.com/questions/127081/conntrack-tcp-timeout-for-state-stablished-not-working and https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt for information on how to adjust the timeouts--though it isn't clear to me that iptables natively supports a maximum connection lifetime; I believe all timeouts are idle timeouts.

Clean up with ip netns del one; ip netns del two; ip netns del three

Answer 4

The firewall can send an ICMP packet indicating that the target was unreachable. For anything but TCP, that is the only possible error indication, for example sending a packet to a closed UDP port will generate a "destination unreachable" message with the reason code set to "port unreachable".

It is also possible to send "port unreachable" messages as a response to TCP packets, this also terminates the connection, but anyone analyzing packet dumps will notice that this is unusual, because TCP convention is to indicate closed ports with a RST.

The sender is expected to map any ICMP error packets received back to the originating connection and handle them appropriately, so a firewall generated error packet can also be used to terminate a TCP connection. The ICMP packet contains a copy of the headers of the offending packet to allow this mapping.