Another Possible Reason - dhcpcd incorrect settings
I add this answer, although it is (probably) not directly related to the specific question above, it is relevant in some cases
If a systems static ip / dhcp settings are incorrect, this can show up as an indirect side effect.
In my specific case I had a Linux machine which I moved locations and changed the network settings for.
I forgot to update them in
/etc/dhcpcd.conf, when I moved the machine back again.
This is my
When it was "broken", I had the line
static routers=192.168.1.254 commented out, and the line below
#static routers=192.168.1.1 was uncommented.
This then caused
arp to display an invalid entry ("incomplete") for the address
Even though I only moved this system a week ago, then moved it back a few days ago, I had absolutely no clue what was wrong with it. As far as I could tell, I could connect it to my local network, ssh into it fine, and there were no issues showing up with
traceroute also wasn't showing any useful info, it just suggested that all queries were going via a single hop to "machine host name". Obviously that made no sense and was wrong, but it didn't give me any indication as to what the issue was - it implied that all queries were being resolved (somehow) on the same machine. This machine does DNS by the way.
Not directly related to the question of OP, but may be very useful to others who have this issue in future, and this is the only way you have found thus far to diagnose it, as not much else appears to be wrong!
Just some notes on how I diagnosed this
- When using this machine as the DNS server for other machines on the network, those machines were unable to resolve external DNS requests
- I could ssh into the DNS server (that would be expected as its IP is set static, and the switch/router also expects to see a machine with this static IP connected on the local ethernet via the switch).
- The first thing that indicated some issue was
sudo apt update failed. (Since this is a DNS server, it has no graphical interface and therefore although running something like a web browser would have indicated this problem a lot sooner, or from the Network desktop tray icon, it doesn't have any of those things.)
ping 22.214.171.124 was also failing, however ssh and pinging local machines was fine
- I checked
/etc/network/interfaces[.d] however all files in these dirs were blank / set to defaults, as the DNS software manages these
- There were no errors/warnings in the DNS software when connecting to the DNS server via the web interface supplied
- I started checking things like
ip addr (no issues) and
arp. I already said
traceroute didn't indicate what the problem might be, but did indicate something weird was going on.
- Searching for reasons why
arp might have an incomplete entry guided me along the right lines
- I found this question (https://serverfault.com/questions/765380/when-do-stale-arp-entries-become-failed-when-never-used) and started looking into how to remove arp entries
- I remove the entry with
arp -d 192.168.1.1
- However it kept coming back
- I tried to connect to
192.168.1.1 on my local network from another machine, however I was not surprised I couldn't do this or ping it because there are no machines on my network with this address
- Usually, this address would be the address of your gateway / an ISP router in home applications
- This reminded me that I had set a network adapter in the other location to have the address
192.168.1.1, and that the computer that was connected to this address was being used as a router to connect the networks
- The original location is on network
192.168.1.X, the location I moved to was on
- Since most
192.168.1.X/24 networks use
192.168.1.1 as the default gateway, seeing this IP didn't initially arouse any suspicions, and it didn't seem "weird" for any reason
- However I changed the default gateway in
dhcpcd.conf as described above and the issue was resolved
- Lesson learned, when creating a weird computer-to-computer network off site temporarily, give the router adapter/NIC a weird static IP like
192.168.1.50 - that might help to remind ones self or indicate something out of the ordinary is set in a settings file somewhere