What happens to this packet? (routed to machine on local network)

Question

Hi Network engineering,

I'm having an issue with my network design. My goal is to have two networks behind a gateway device. "Circle1" would contain my internet facing proxy servers and "Circle2" would contain any number of app servers. "Cirle1" should connect to "Circle2" directly. This works great in most cases. I am however running into an issue where any packets originating from "Circle2" with a destination of the PUBLIC interface of a machine on "Circle1" seem to fall off somewhere.

The frusterating thing is that my packet captures of all points show the packet at the very least leaving the gateway server bound for 172.16.0.100 but do not show any packets coming into that server they didn't show up in stdout when running tcpdump but they do show up when looking at the dump after using wireshark.

Question

Is this the result of my network architecture or configuration issue? Are packets not permitted to enter an interface for a machine for which the machine already has a different interface listening on that source network?

Some quick notes:

1. all hosts are running CentOS 7
2. SELinux is disabled for troubleshooting purposes
3. iptables have been flushed with the exception of the gateway which contains 1 rule (iptables -t nat -I POSTROUTING -s 10.0.0.0/8 -o enp0s3 -j MASQUERADE)
4. I'm running on virtualbox and have not attempted to confirm this configuration on physical hosts :-/ sorry not enough machines.

Note: the red addresses should would be replaced with public facing ip addresses in a non-lab environment

Problem: Traffic from 10.1.0.0/24 to 172.16.0.100 is not recieved.

Debugging steps:

1. Setup a network similar to the one presented in the diagram above.
2. Execute ping 172.16.0.100 from the app server at 10.1.0.101.
3. Traffic too and from all networks to ensure there is no routing errors.

Expected results:

1. The app server will send the ICMP packet to the web proxy server (172.16.0.100) via the default gateway (10.1.0.1)
2. The default gateway will send the packet to the web proxy server (172.16.0.100)
3. The web proxy server will accept the ICMP packet [source-ip 10.1.0.101; destination-ip 172.16.0.100]
4. The web proxy server will generate an ICMP reply packet [source-ip 172.16.0.100; destination-ip 10.1.0.101]
5. The web proxy server will send the generated packet directly to the app server via enp0s8

Actual results:

1. The app server sends the ICMP packet to the web proxy server (172.16.0.100) via the default gateway (10.1.0.1)
2. The default gateway sends the packet to the web proxy server (172.16.0.100)
3. The web proxy doesn't seem to recieve the packet (tcpdump -i enp0s3 does not show any packets entering the system)
4. EDIT FOR STEP 3 It looks like packet does make it to the proxy server but the tcpdump command doesn't show it in stdout, it does show it when using the -w option. I'm baffled.

Debugging tool outputs:

1. Script ip addr && ip route output app-server, gateway-server, proxy-server

2. Wireshark compatible packet captures proxy-enp0s3, proxy-enp0s8 (empty), gateway-enp0s8, gateway-enp0s9

Thanks to those of you who've taken the time to try and help me. I do apologize about the rather verbose format.

more testing due to confusion caused by tcpdump

tcpdump -w cap does show all of the packets I'm expecting while tcpdump does not show any via stdout. It does however show a large number of packets dropped by kernel. This occurs regardless of run time or use of -B 4096 to expand the tcpdump buffer. It seems pretty obvious that the OS is rejecting the packets but I'm not sure why. (tcpdump hangs briefly when closing after piping to stdout but I'm not sure if that is related)

Answer 1

I spent a few more hours googling and found the appropriate support document on Red Hat's site: https://access.redhat.com/solutions/53031

It turns out Red Hat 6+ will filter asymmetrically routed packets by default.

The solution is to To make this behaviour persistent across reboots, modify /etc/sysctl.conf and make the following change prior to reboot:

net.ipv4.conf.default.rp_filter = 2