Hi Network engineering,
I'm having an issue with my network design. My goal is to have two networks behind a gateway device. "Circle1" would contain my internet facing proxy servers and "Circle2" would contain any number of app servers. "Cirle1" should connect to "Circle2" directly. This works great in most cases. I am however running into an issue where any packets originating from "Circle2" with a destination of the PUBLIC interface of a machine on "Circle1" seem to fall off somewhere.
The frusterating thing is that my packet captures of all points show the packet at the very least leaving the gateway server bound for 172.16.0.100 but
do not show any packets coming into that server they didn't show up in
stdout when running
tcpdump but they do show up when looking at the dump after using wireshark.
Is this the result of my network architecture or configuration issue? Are packets not permitted to enter an interface for a machine for which the machine already has a different interface listening on that source network?
Some quick notes:
- all hosts are running CentOS 7
- SELinux is disabled for troubleshooting purposes
- iptables have been flushed with the exception of the gateway which contains 1 rule (
iptables -t nat -I POSTROUTING -s 10.0.0.0/8 -o enp0s3 -j MASQUERADE)
- I'm running on virtualbox and have not attempted to confirm this configuration on physical hosts :-/ sorry not enough machines.
Note: the red addresses should would be replaced with public facing ip addresses in a non-lab environment
Problem: Traffic from 10.1.0.0/24 to 172.16.0.100 is not recieved.
- Setup a network similar to the one presented in the diagram above.
ping 172.16.0.100from the app server at 10.1.0.101.
- Traffic too and from all networks to ensure there is no routing errors.
- The app server will send the ICMP packet to the web proxy server (172.16.0.100) via the default gateway (10.1.0.1)
- The default gateway will send the packet to the web proxy server (172.16.0.100)
- The web proxy server will accept the ICMP packet [source-ip 10.1.0.101; destination-ip 172.16.0.100]
- The web proxy server will generate an ICMP reply packet [source-ip 172.16.0.100; destination-ip 10.1.0.101]
- The web proxy server will send the generated packet directly to the app server via enp0s8
- The app server sends the ICMP packet to the web proxy server (172.16.0.100) via the default gateway (10.1.0.1)
- The default gateway sends the packet to the web proxy server (172.16.0.100)
The web proxy doesn't seem to recieve the packet (
tcpdump -i enp0s3does not show any packets entering the system)
- EDIT FOR STEP 3 It looks like packet does make it to the proxy server but the
tcpdumpcommand doesn't show it in stdout, it does show it when using the
-woption. I'm baffled.
Debugging tool outputs:
Thanks to those of you who've taken the time to try and help me. I do apologize about the rather verbose format.
more testing due to confusion caused by tcpdump
tcpdump -w cap does show all of the packets I'm expecting while
tcpdump does not show any via stdout. It does however show a large number of packets dropped by kernel. This occurs regardless of run time or use of
-B 4096 to expand the tcpdump buffer. It seems pretty obvious that the OS is rejecting the packets but I'm not sure why. (tcpdump hangs briefly when closing after piping to stdout but I'm not sure if that is related)