How to identify a crossover connected and a switch connected NICs between two servers

Question

I have two openbsd servers with 6 NIC cards each.

>lspci | grep -i ether
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

Out of these, one nic on the first one, and two on the second one are unplugged, and the remaining nics except one is connected via a switch. The last nic on server1 is connected to the second server directly via a crossover cable.

How do I identify which interface is the one that is connected directly to the other server, and not via switch? I tried arping and ping and analyzing the tcpdump, hoping the ones connected to switch would show the switch's mac-id, but everything showed the machine's nic mac-id.

Answer 1

Short answer: you can't.

Switches do not modify Ethernet frames, so there's no difference between one that was sent directly from the server and one that passed through the switch.

Answer 2

The buzzword that you're looking for is MDI-X.

On Linux systems, you can use ethtool to retrieve such information (it might not be supported with every NIC):

# ethtool eth0
Settings for eth0:
  Speed: 100Mb/s
  Duplex: Full
  ...
  MDI-X: off

I am not sure, what an equivalent of ethtool for OpenBSD is or if it available there. Sorry, if not. But theoretically, you can retrieve such information from the interface state, but not from the traffic.

Answer 3

You've got several options:

Option 1. Record the IP and MAC address of the directly connected server's NIC (the "other" server, as you call it). From your original server ping that IP address and use the arp command to check on which interface does the MAC address of the "other" server appear.

Option 2. Generate traffic via the switches, e.g. ping multiple servers on your network that you know are connected via the switch ports. Run arp, and if there's more than one MAC address to a port, then this interface is connected to a switch port (multiple MAC addresses are a give-away).

Option 3. Check you server's routing table. You should be able to see which route is for the directly connected server, and where it goes, e.g.

netstat -rn

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.1.2        0.0.0.0         255.255.255.255 UH        0 0          0 eth0
10.10.2.0        10.8.0.2        255.255.255.0   UG        0 0          0 eth1
60.60.60.0      0.0.0.0         255.255.255.0   U         0 0          0 eth2
10.254.254.0     0.0.0.0         255.255.0.0     U         0 0          0 eth3
0.0.0.0         60.60.60.1      0.0.0.0         UG        0 0          0 eth0

All of this depends on some details, e.g. whether any of these interfaces uses teaming/bonding, or some level of aggregation, whether the server has bridged interfaces, etc.

Ethernet switches typically work using something called transparent bridging - emphasis on transparent, meaning they're usually somewhat invisible to the end hosts (bar things like STP/CDP). So never expect to see a switch MAC address in a host's ARP table!

I hope you find these useful :-)

Answer 4

While you can't directly tell which interfaces connect to a switch and which connect the two servers directly, there may be ways to do this indirectly. Namely, look for the type of traffic that is generally produced by a switch and not by a server.

Unless you have the servers configured to participate in STP (i.e. generally only a concern if you are bridging or running virtualization with a switch/bridge), they don't tend to send out BPDUs like a switch will if running STP.

You can use tcpdump to capture these frames if present (substituting the appropriate interface):

tcpdump -nn -v -i eth0 -c 1 'stp'

If your servers are using STP or your switches are not, then you can look at other options.

If they are Cisco switches, by default they will be using a process called CDP (Cisco Discovery Protocol). This is often disabled in some environments so you may need to consider enabling it to test. Some other vendors also support CDP as well, but it may be disabled by default. You can capture these with tcpdump as follows:

tcpdump -nn -v -i eth0 -c 1 'ether[20:2] == 0x2000'

The standard based version of CDP is LLDP (link layer discovery protocol). If the vendor's switches support it, they may have this on by default while others have it off. Here is the tcpdump command to capture this traffic:

tcpdump -nn -v -i eth0 -c 1 'ether proto 0x88cc'

There may be other options, but these are the ones I would expect a good chance of seeing from a switch in a business environment.