2

Hi Network engineering,

I'm having an issue with my network design. My goal is to have two networks behind a gateway device. "Circle1" would contain my internet facing proxy servers and "Circle2" would contain any number of app servers. "Cirle1" should connect to "Circle2" directly. This works great in most cases. I am however running into an issue where any packets originating from "Circle2" with a destination of the PUBLIC interface of a machine on "Circle1" seem to fall off somewhere.

The frusterating thing is that my packet captures of all points show the packet at the very least leaving the gateway server bound for 172.16.0.100 but do not show any packets coming into that server they didn't show up in stdout when running tcpdump but they do show up when looking at the dump after using wireshark.

Question

Is this the result of my network architecture or configuration issue? Are packets not permitted to enter an interface for a machine for which the machine already has a different interface listening on that source network?

Some quick notes:

  1. all hosts are running CentOS 7
  2. SELinux is disabled for troubleshooting purposes
  3. iptables have been flushed with the exception of the gateway which contains 1 rule (iptables -t nat -I POSTROUTING -s 10.0.0.0/8 -o enp0s3 -j MASQUERADE)
  4. I'm running on virtualbox and have not attempted to confirm this configuration on physical hosts :-/ sorry not enough machines.

quick network diagram Note: the red addresses should would be replaced with public facing ip addresses in a non-lab environment

Problem: Traffic from 10.1.0.0/24 to 172.16.0.100 is not recieved.

Debugging steps:

  1. Setup a network similar to the one presented in the diagram above.
  2. Execute ping 172.16.0.100 from the app server at 10.1.0.101.
  3. Traffic too and from all networks to ensure there is no routing errors.

Expected results:

  1. The app server will send the ICMP packet to the web proxy server (172.16.0.100) via the default gateway (10.1.0.1)
  2. The default gateway will send the packet to the web proxy server (172.16.0.100)
  3. The web proxy server will accept the ICMP packet [source-ip 10.1.0.101; destination-ip 172.16.0.100]
  4. The web proxy server will generate an ICMP reply packet [source-ip 172.16.0.100; destination-ip 10.1.0.101]
  5. The web proxy server will send the generated packet directly to the app server via enp0s8

Actual results:

  1. The app server sends the ICMP packet to the web proxy server (172.16.0.100) via the default gateway (10.1.0.1)
  2. The default gateway sends the packet to the web proxy server (172.16.0.100)
  3. The web proxy doesn't seem to recieve the packet (tcpdump -i enp0s3 does not show any packets entering the system)
  4. EDIT FOR STEP 3 It looks like packet does make it to the proxy server but the tcpdump command doesn't show it in stdout, it does show it when using the -w option. I'm baffled.

Debugging tool outputs:

  1. Script ip addr && ip route output app-server, gateway-server, proxy-server

  2. Wireshark compatible packet captures proxy-enp0s3, proxy-enp0s8 (empty), gateway-enp0s8, gateway-enp0s9

Thanks to those of you who've taken the time to try and help me. I do apologize about the rather verbose format.

more testing due to confusion caused by tcpdump

tcpdump -w cap does show all of the packets I'm expecting while tcpdump does not show any via stdout. It does however show a large number of packets dropped by kernel. This occurs regardless of run time or use of -B 4096 to expand the tcpdump buffer. It seems pretty obvious that the OS is rejecting the packets but I'm not sure why. (tcpdump hangs briefly when closing after piping to stdout but I'm not sure if that is related)

  • Can you capture off the circle 1 switch? This would allow you to validate if the problem is on the network or on the proxy server. If it is on the network, what is the vendor/model of the circle 1 switch and what is it's configuration? If it is a problem on the proxy server, then we should migrate this to another more appropriate stack. – YLearn Apr 3 '15 at 23:30
  • The switch is more for easy representation. The entire setup is in a hypervisor. The packet capture from the gateway-server is running in promiscuous mode and should have everything but I can spin up another VM to monitor the virtual switch passively. Also I've updated part of it. Regarding the actual results. The packet does make it the proxy server but for some reason doesn't seem to be treated like it's destine for that server (can see in packet capture file but not when viewing live.) – Colton Apr 3 '15 at 23:37
  • At this point I'm somewhat tempted to just add a bunch of static routes via the DHCP and call it a day but that doesn't seem overly manageable. – Colton Apr 3 '15 at 23:38
3

I spent a few more hours googling and found the appropriate support document on Red Hat's site: https://access.redhat.com/solutions/53031

It turns out Red Hat 6+ will filter asymmetrically routed packets by default.

The solution is to To make this behaviour persistent across reboots, modify /etc/sysctl.conf and make the following change prior to reboot:

net.ipv4.conf.default.rp_filter = 2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.