Can computer device with multiple NIC be configured with same subnets for isolated networks?

Question

Two Networks

I have an embedded device using Xilinx ARM processor, with eth0 and eth1. I also have wlan0 and I am running Linux 3.17.

PC0 <---left--->[eth0  ARM   eth1]<---right:some devices--->PC1

Left: point to point, only PC0 is connected to eth0. This is a local area network. We use this side to do ftp. We also configures the IP address of eth1 from our PC software. We plan to specify eth0 to be 192.168.1.100, and PC0 to be 192.168.1.99.

Right: some devices are connected to be a local area network. Our customers configure their right side, and don't us to participate on this side. Our customer can use our same PC software to read data or ftp from eth1 side.

Between Left and Right: nothing should be forwarded, no bridging, completely isolated.

Question

The following configurations have potential problem from my opinion and I would like to have peer opinions:

1. If customer already configured their networks to be 192.168.1.x, and configures eth1 to be 1.100 which == eth0, this device will have two NIC with same IP on different network. Should we avoid this?

   PC0(99) <---left--->[eth0(100) ARM eth1(100)]<---right:some devices--->PC1

2. If customer configured their networks to be 192.168.1.x, and configures eth1 to be 1.99 == PC0, now will this confuses the Linux on ARM device? Now the stack knows that both PC0 and eth1 are same IP. Should we avoid this?

   PC0(99) <---left--->[eth0(100) ARM eth1(99)]<---right:some devices--->PC1

3. If customer configured their networks to be 192.168.1.x, and configure eth1 to be 1.98, and PC1 to be 100, will there be a problem? Should we avoid this?

   PC0(99) <---left--->[eth0(100) ARM eth1(98)]<---right:some devices--->PC1(100)

4. If customer configured their networks to be 192.168.1.x, and configure eth1 to be 1.98, and PC1 to be 99, will there be a problem? Should we avoid this?

   PC0(99) <---left--->[eth0(100) ARM eth1(98)]<---right:some devices--->PC1(99)

5. If customer configured their networks to be 192.168.1.x, and configure eth1 to be 1.98, and PC1 to be 101, and other devices on the right have IP different from 98, or 100. I hope there is a way to configure the routing table so requests to eth0 are replied through eth0, and requests to eth1 are replied through eth1. But will there be a problem? Is it recommended to avoid this?

   PC0(99) <---left--->[eth0(100) ARM eth1(98)]<---right:some devices--->PC1(101)

Should we tells customer what is our subnet on the left side and not to use this same subnet for the right side at all? Imagine if we run two OS on this ARM device and each controls one NIC then there is no problem.

Answer 1

The host in the middle of two networks, both using 192.168.1.0/24, would have no way of knowing that 192.168.1.99 is on one of the networks by itself unless you have a specific route for that. You should also make sure that no hosts on the other side have that same address, or the address of 192.168.1.100. The isolated host (192.168.1.99) would not be able to send or receive anything except to 192.168.1.100, and nothing on the other side would be able to communicate with the isolated PC.

The obvious solution would be to select addressing for the left side which doesn't exist anywhere else on the right side. You could use a point-to-point link with a /31 mask length since it will be only need two addresses. There are a lot of address blocks which probably won't be used, such as TEST-NET-1, TEST-NET-2, TEST-NET-3, etc. Look at RFC 5735, Special Use IPv4 Addresses, Section 4, Summary Table, for candidate address blocks. In fact, addresses from the Link-local bock (169.254.0.0/16) may be ideal for this since those addresses are forbidden from being routed. You also aren't supposed to change the mask size from /16, but that doesn't really matter for your application.