Another Possible Reason - dhcpcd incorrect settings
I add this answer, although it is (probably) not directly related to the specific question above, it is relevant in some cases
If a systems static ip / dhcp settings are incorrect, this can show up as an indirect side effect.
In my specific case I had a Linux machine which I moved locations and changed the network settings for.
I forgot to update them in /etc/dhcpcd.conf
, when I moved the machine back again.
This is my dhcpcd.conf
interface eth0
static ip_address=192.168.1.8/24
static routers=192.168.1.254
#static routers=192.168.1.1
static domain_name_servers=127.0.0.1
When it was "broken", I had the line static routers=192.168.1.254
commented out, and the line below #static routers=192.168.1.1
was uncommented.
This then caused arp
to display an invalid entry ("incomplete") for the address 192.168.1.1
.
Even though I only moved this system a week ago, then moved it back a few days ago, I had absolutely no clue what was wrong with it. As far as I could tell, I could connect it to my local network, ssh into it fine, and there were no issues showing up with ip addr
. traceroute
also wasn't showing any useful info, it just suggested that all queries were going via a single hop to "machine host name". Obviously that made no sense and was wrong, but it didn't give me any indication as to what the issue was - it implied that all queries were being resolved (somehow) on the same machine. This machine does DNS by the way.
Not directly related to the question of OP, but may be very useful to others who have this issue in future, and this is the only way you have found thus far to diagnose it, as not much else appears to be wrong!
Just some notes on how I diagnosed this
- When using this machine as the DNS server for other machines on the network, those machines were unable to resolve external DNS requests
- I could ssh into the DNS server (that would be expected as its IP is set static, and the switch/router also expects to see a machine with this static IP connected on the local ethernet via the switch).
- The first thing that indicated some issue was
sudo apt update
failed. (Since this is a DNS server, it has no graphical interface and therefore although running something like a web browser would have indicated this problem a lot sooner, or from the Network desktop tray icon, it doesn't have any of those things.)
ping 8.8.8.8
was also failing, however ssh and pinging local machines was fine
- I checked
/etc/network/interfaces[.d]
however all files in these dirs were blank / set to defaults, as the DNS software manages these
- There were no errors/warnings in the DNS software when connecting to the DNS server via the web interface supplied
- I started checking things like
ip addr
(no issues) and arp
. I already said traceroute
didn't indicate what the problem might be, but did indicate something weird was going on.
- Searching for reasons why
arp
might have an incomplete entry guided me along the right lines
- I found this question (https://serverfault.com/questions/765380/when-do-stale-arp-entries-become-failed-when-never-used) and started looking into how to remove arp entries
- I remove the entry with
arp -d 192.168.1.1
- However it kept coming back
- I tried to connect to
192.168.1.1
on my local network from another machine, however I was not surprised I couldn't do this or ping it because there are no machines on my network with this address
- Usually, this address would be the address of your gateway / an ISP router in home applications
- This reminded me that I had set a network adapter in the other location to have the address
192.168.1.1
, and that the computer that was connected to this address was being used as a router to connect the networks 192.168.0.X
and 192.168.1.X
- The original location is on network
192.168.1.X
, the location I moved to was on 192.168.0.X
- Since most
192.168.1.X/24
networks use 192.168.1.1
as the default gateway, seeing this IP didn't initially arouse any suspicions, and it didn't seem "weird" for any reason
- However I changed the default gateway in
dhcpcd.conf
as described above and the issue was resolved
- Lesson learned, when creating a weird computer-to-computer network off site temporarily, give the router adapter/NIC a weird static IP like
192.168.1.50
- that might help to remind ones self or indicate something out of the ordinary is set in a settings file somewhere