5

I currently have a set of four virtual hosts on a private network, with a fifth host running pfSense to act as the gateway.

When communicating with hosts outside of the private network, all traffic is routed via the pfSense gateway (10.0.0.1). However, when two hosts on the same subnet communicate, they do not do so via the gateway. I have gathered that these hosts realise they are on the same subnet and switch and opt to communicate via MAC address w/ARP rather than using the IP gateway I have set up.

I would like these hosts to communicate via the gateway so that I can monitor the traffic between the hosts on the private network. I have attempted to set static routes, for example: ip route add 10.0.0.4 via 10.0.0.1 dev eth0

The first traceroute subsequent to this will correctly route via the gateway:

root@file-server ~ # traceroute 10.0.0.4
traceroute to 10.0.0.4 (10.0.0.4), 30 hops max, 60 byte packets
 1  10.0.0.1 (10.0.0.1) 0.241ms 0.287ms 0.274ms
 2  10.0.0.4 (10.0.0.4) 0.559ms 0.567ms 0.574ms

If I then perform the traceroute again however, the route stops using the gateway.

root@file-server ~ # traceroute 10.0.0.4
 1  10.0.0.4 (10.0.0.4) 0.345ms 0.376ms 0.362ms

My guess would be that perhaps my host has populated its ARP table with the MAC address of the traceroute target after the initial traceroute, and opts to use that for communication in future.

If that is the case, I am not sure how to counter it. I believed that my static route would take precedent over any other decisions taken by the system. I would like to research further, but I am not actually sure where to look to solve this issue as I am not sure why my static route is not adhered to.

Can someone please provide insight as to why my static route is only adhered to with my initial traceroute, and how I would go about forcing a static route to the gateway for same-subnet communications?

**Edit: ** I should add that this set-up is for experimentation, so I am not restricted to solely things that would be a good idea in the real world.

4

You would have to capture packets to be sure, but I suspect what is going on is an ICMP Redirect.

If a Router receives a packet for which a better next-hop exists, it can send an ICMP Redirect to the original sender to inform them to use the better next-hop when trying to speak to that same destination.

The initial sender will update its routing table temporarily with the new information. I believe for 10 minutes, but this is probably different with each OS/vendor/implementation.

There is generally a way to disable ICMP redirects at the Router. And there is more than likely a way to disable accepting ICMP redirects on the Host. If not specifically, then indirectly by blocking it with an ACL or iptables entry.

You can read more about ICMP Redirects here: http://www.cymru.com/gillsr/documents/icmp-redirects-are-bad.htm

The are now-a-days considered a security vulnerability, but when they were created they were based on good intentions. As the Internet got bigger, and less trust-able, however, their use has been significantly reduced -- and discouraged, for that matter.

  • 1
    That was exactly the problem, thanks! I hadn't heard of ICMP redirects before. I'll read into them a bit. – Joe Feb 20 '16 at 21:08
  • 1
    The problem with relying on disabling ICMP redirects is that your hosts are the ones who make the decision whether to use the gateway or not (with routes). I know you are in a lab so it does not matter, but if you are doing that for security your goal is not achieved. :) See my answer for a way to force your hosts to use the gateway. – Benjamin A. Feb 23 '16 at 8:22
4

You want to look into two features called private VLAN and ARP proxy.

When your host's VLAN is configured as a private VLAN, the switch will forbid your host (let's assume it has IP 10.0.0.3) from communicating with other hosts in the same VLAN, as for instance 10.0.0.4. When I say forbid, I mean that it will not forward ethernet frames incoming from a host in the VLAN to another host in the same VLAN. That means no ARP, no ICMP, nothing.

That's a bit against everything we learned in networking class but that's how it works. Bear with me because that is not all there is to know with private VLAN.

The VLAN where our two hosts (host .3 and .4) are configured is known as the secondary VLAN. You can configure a port (typically the gateway's) to be a little special : this port will be in another VLAN, the primary VLAN. Both VLAN are linked somehow (configuration varies with vendors) in the switch's config. A host in the primary VLAN will be able to communicate (i.e. the switch will allow frames to be forwarded) with any hosts, whether they are in the primary or the secondary VLAN, and vice versa.

So there we are. Our hosts (10.0.0.3 and .4) cannot communicate between them but can communicate with the gateway (10.0.0.1), and, per extension, the rest of the internet.

That can be very useful for hosting providers that do not want different clients to communicate and do not want to waste IP addresses in a lot of /30 networks. But that is not what you want. Yet.

The twin feature of private VLAN is ARP proxy. That is just a config on the gateway to instruct it to respond on ARP requests originated from host .3, and asking for 10.0.0.4's MAC, with its own MAC address. The gateway then initiate its own ARP request to 10.0.0.4. This time host .4 sees it (the gateway is in the primary VLAN, remember?) and responds.

This way, the gateway is performing some sort of legitimate man-in-the-middle between your two host, and can control the traffic. It may perform an ICMP redirect, you might want to disable that.

That's it. You can now control traffic between hosts in the same subnet. Well, you need a switch with the correct features, of course ! ;)

(or you can use two different VLAN, as Ron outlined.)

EDIT : some more information about how private VLANs work.

The primary (say VLAN ID 10) and secondary VLAN (ID 20) have different 802.1q numbers.

A frame from the gateway will travel your switched network tagged with ID 10. Switches will then forward your frame to trunk ports, and access ports in VLAN 10 and 20, because of the relation between primary and secondary VLANs that is configured on the device.

A frame from an isolated host will travel your network tagged as VLAN 20. It will be forwarded on trunk ports, and access ports in VLAN 10 but not access ports in VLAN 20.

There is no communication involved between switches because none is required : it is only a matter of local treatment. But it is advised to have the same primaries/secondaries config on all switches, or expect headaches.

If you send frames to your hosts tagged, you will have to play with allowed vlan and some VLAN mapping magic. Refer to your vendor's doc.

Also, I only talked about isolated secondary VLANs. There are also community secondary VLANs, but that is for another time/another edit.

0

That is how networking works. Hosts on the same layer-2 network will use layer-2 to communicate; doing otherwise is inefficient and unnecessary. Gateways (routers) connect networks. A host will inspect the layer-3 destination address and compare it to its own address and mask. If the network is different, it will send the traffic to its configured gateway, which is really just another host on the LAN. Even if you did succeed in sending the traffic to the gateway, the gateway would drop it because the destination is on the same LAN (routers route between networks, not on the same network).

You can configure different networks (VLANs) so that traffic from one host to another goes through the gateway to get from one network to the other.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.