Weird internet disconnects and suspicious stuff in the log
-
Today I started getting some random looking disconnects I could only solve by rebooting pfSense. I rebooted everything including the antenna (we have wireless connection to our ISP), called the ISP and asked whether they were doing any maintenance, but no.
Basically the gateway seemed to be disconnecting. Not much in the logs, the gateway section at least.
But then I went into the general tab, and found this:Mar 21 18:52:33 php-fpm 401 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' Mar 21 18:52:33 php-fpm 401 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WANGW' Mar 21 18:52:32 check_reload_status 432 Reloading filter Mar 21 18:52:32 check_reload_status 432 Restarting OpenVPN tunnels/interfaces Mar 21 18:52:32 check_reload_status 432 Restarting IPsec tunnels Mar 21 18:52:32 check_reload_status 432 updating dyndns WANGW Mar 21 18:52:32 rc.gateway_alarm 2980 >>> Gateway alarm: WANGW (Addr:xxx Alarm:1 RTT:4.404ms RTTsd:20.155ms Loss:21%) Mar 21 18:52:17 kernel arp: 44:d9:e7:b2:31:97 is using my IP address xxx on igc0! Mar 21 18:49:17 kernel arp: short packet received on igc0 Mar 21 18:48:09 kernel arp: short packet received on igc0 Mar 21 18:46:47 nginx 2025/03/21 18:46:47 [error] 56459#100278: send() failed (54: Connection reset by peer) while logging to syslog, server: unix:/var/run/log Mar 21 18:46:47 syslogd kernel boot file is /boot/kernel/kernel Mar 21 18:45:40 kernel arp: short packet received on igc0 Mar 21 18:43:54 kernel arp: packet with invalid ethernet address length 0 received on igc0 Mar 21 18:42:05 kernel arp: packet with unknown hardware format 0x17 received on igc0 Mar 21 18:37:58 kernel arp: packet with unknown hardware format 0x129 received on igc0 Mar 21 18:37:16 kernel arp: packet with unknown hardware format 0x07 received on igc0 Mar 21 18:35:32 kernel arp: packet with unknown hardware format 0x129 received on igc0 Mar 21 18:35:20 syslogd kernel boot file is /boot/kernel/kernelarp: 44:d9:e7:b2:31:97 is using my IP address xxx on igc0!
What the hell? What's that?
igc0 is the wan port on the router, it's only connected to the antenna on the roof.
That MAC address is not that of the LAN interface of the antenna, because I saw a screenshot from the admin screen previously.Does anyone have any idea what could be going on?
-
It's a WiFi ISP?
Some other device on the WAN is using the IP as it says. A Ubiquity device by the looks of it.
Is the WAN DHCP or static?
-
@stephenw10 What does wifi ISP mean? We're connected via wireless to the closest AP.
There are no other devices, there cannot be. The path is literally ISP's AP-air-antenna on the roof-cable-our router. This is a family house, noone else lives here.
And the WAN has static IP, yes. -
Also, why is the link to the antenna dropping all the time? The cable is perfectly fine.
Mar 21 20:39:59 php-fpm 52310 /rc.start_packages: Restarting/Starting all packages. Mar 21 20:39:58 check_reload_status 432 Reloading filter Mar 21 20:39:58 check_reload_status 432 Starting packages Mar 21 20:39:58 php-fpm 76406 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - xxx -> xxx - Restarting packages. Mar 21 20:39:56 php-fpm 76406 /rc.newwanip: Creating rrd update script Mar 21 20:39:56 php-fpm 76406 /rc.newwanip: Resyncing OpenVPN instances for interface WAN. Mar 21 20:39:54 dhcpleases 71966 Could not deliver signal HUP to process 76214: No such process. Mar 21 20:39:52 php-fpm 76406 /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' Mar 21 20:39:52 php-fpm 76406 /rc.newwanip: Gateway, NONE AVAILABLE Mar 21 20:39:51 php-fpm 76406 /rc.newwanip: rc.newwanip: on (IP address: xxx) (interface: WAN[wan]) (real interface: igc0). Mar 21 20:39:51 php-fpm 76406 /rc.newwanip: rc.newwanip: Info: starting on igc0. Mar 21 20:39:50 check_reload_status 432 Reloading filter Mar 21 20:39:50 check_reload_status 432 rc.newwanip starting igc0 Mar 21 20:39:50 php-fpm 401 /rc.linkup: HOTPLUG: Triggering address refresh on wan (igc0) Mar 21 20:39:50 php-fpm 401 /rc.linkup: DEVD Ethernet attached event for wan Mar 21 20:39:50 php-fpm 401 /rc.linkup: Hotplug event detected for WAN(wan) static IP address (4: xxx) Mar 21 20:39:50 check_reload_status 432 Reloading filter Mar 21 20:39:46 dhcpleases 28937 Could not deliver signal HUP to process 55789: No such process. Mar 21 20:39:45 kernel igc0: link state changed to UP Mar 21 20:39:45 check_reload_status 432 Linkup starting igc0 Mar 21 20:39:44 php-fpm 400 /rc.linkup: DEVD Ethernet detached event for wan Mar 21 20:39:44 php-fpm 400 /rc.linkup: Hotplug event detected for WAN(wan) static IP address (4: xxx) Mar 21 20:39:43 kernel igc0: link state changed to DOWN Mar 21 20:39:43 check_reload_status 432 Linkup starting igc0 Mar 21 20:35:07 php-fpm 52310 /rc.start_packages: Restarting/Starting all packages. Mar 21 20:35:06 check_reload_status 432 Reloading filter Mar 21 20:35:06 check_reload_status 432 Starting packages Mar 21 20:35:06 php-fpm 27731 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - xxx -> xxx - Restarting packages. Mar 21 20:35:04 php-fpm 27731 /rc.newwanip: Creating rrd update script Mar 21 20:35:04 php-fpm 27731 /rc.newwanip: Resyncing OpenVPN instances for interface WAN. Mar 21 20:35:02 dhcpleases 97821 Could not deliver signal HUP to process 89944: No such process. Mar 21 20:35:00 php-fpm 27731 /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' Mar 21 20:35:00 php-fpm 27731 /rc.newwanip: Gateway, NONE AVAILABLE Mar 21 20:34:59 php-fpm 27731 /rc.newwanip: rc.newwanip: on (IP address: xxx) (interface: WAN[wan]) (real interface: igc0). Mar 21 20:34:59 php-fpm 27731 /rc.newwanip: rc.newwanip: Info: starting on igc0. Mar 21 20:34:58 check_reload_status 432 Reloading filter Mar 21 20:34:58 check_reload_status 432 rc.newwanip starting igc0 Mar 21 20:34:58 php-fpm 400 /rc.linkup: HOTPLUG: Triggering address refresh on wan (igc0) Mar 21 20:34:58 php-fpm 400 /rc.linkup: DEVD Ethernet attached event for wan Mar 21 20:34:58 php-fpm 400 /rc.linkup: Hotplug event detected for WAN(wan) static IP address (4: xxx) Mar 21 20:34:58 check_reload_status 432 Reloading filter Mar 21 20:34:54 dhcpleases 38262 Could not deliver signal HUP to process 70529: No such process. Mar 21 20:34:53 kernel igc0: link state changed to UP Mar 21 20:34:53 check_reload_status 432 Linkup starting igc0 Mar 21 20:34:52 php-fpm 88422 /rc.linkup: DEVD Ethernet detached event for wan Mar 21 20:34:52 php-fpm 88422 /rc.linkup: Hotplug event detected for WAN(wan) static IP address (4: xxx) Mar 21 20:34:51 kernel igc0: link state changed to DOWN Mar 21 20:34:51 check_reload_status 432 Linkup starting igc0 Mar 21 19:18:29 check_reload_status 432 Reloading filter Mar 21 19:18:29 check_reload_status 432 Syncing firewall Mar 21 19:18:29 php-fpm 76406 /system_advanced_network.php: Configuration Change: admin@192.168.0.90 (Local Database): Networking Advanced Settings saved Mar 21 19:10:32 nginx 2025/03/21 19:10:32 [error] 56421#100316: send() failed (54: Connection reset by peer) while logging to syslog, server: unix:/var/run/log Mar 21 19:04:27 syslogd kernel boot file is /boot/kernel/kernel -
@Octopuss I have noticed another thing after I had a complete internet disconnect,
First of all, I have no idea what "ARP table" is, but at the top of the list, there were two entries, the top one being the IP of the ISP's AP few hundred metres away, and the 2nd is actually our public IP.
When I manually removed the top one, the connection was instantly restored.
-
The top entry there is the upstream ISP gateway address. That is somewhere in the ISP network, the next logical hop, but almost certainly not the actual wireless device.
@Octopuss said in Weird internet disconnects and suspicious stuff in the log:
There are no other devices, there cannot be. The path is literally ISP's AP-air-antenna on the roof-cable-our router.
Why do you think there could not be some other client in the WAN subnet? If you're lucky the ISP might have some sort of client isolation enabled in their wireless device but they might not. It would only take some other customer to typo the IP they are supposed to be using.
@Octopuss said in Weird internet disconnects and suspicious stuff in the log:
Mar 21 20:39:43 kernel igc0: link state changed to DOWN
Mar 21 20:39:43 check_reload_status 432 Linkup starting igc0The NIC there actually lost link to the local wireless device. Did the upstream device reboot? You might have a bad cable or a bad port.
-
@stephenw10 said in Weird internet disconnects and suspicious stuff in the log:
The top entry there is the upstream ISP gateway address. That is somewhere in the ISP network, the next logical hop, but almost certainly not the actual wireless device.
What do you mean by "the actual wireless device"? It's the ISP's AP we're connecting to. It's some 260m away on a roof somewhere in the neighborhood.
@stephenw10 said in Weird internet disconnects and suspicious stuff in the log:
Why do you think there could not be some other client in the WAN subnet? If you're lucky the ISP might have some sort of client isolation enabled in their wireless device but they might not. It would only take some other customer to typo the IP they are supposed to be using.
I don't know? I don't really understand networking. I simply assume it's impossible, because the IP (our public IP) is not shared between others customers (who have their own IPs) or anything?
@stephenw10 said in Weird internet disconnects and suspicious stuff in the log:
The NIC there actually lost link to the local wireless device. Did the upstream device reboot? You might have a bad cable or a bad port.
I am not really sure how to read it. I always thought the disconnect is between the pfSense box and the AP/antenna on our roof, and the error meant pfSense lost connection to the "roof", meaning the problem was always on my end.
When I chatted with the guy from the ISP, he said the uptime on our device was 48 days, the signal was perfect, but the LAN connection was dropping pretty often. -
@Octopuss said in Weird internet disconnects and suspicious stuff in the log:
What do you mean by "the actual wireless device"? It's the ISP's AP we're connecting to. It's some 260m away on a roof somewhere in the neighborhood.
The entry in the ARP table is the ISPs gateway. But that is almost certainly not the ISPs AP. It will be some router in the ISPs network.
@Octopuss said in Weird internet disconnects and suspicious stuff in the log:
I always thought the disconnect is between the pfSense box and the AP/antenna on our roof,
Yes, it is. It's losing link on the NIC which means a disconnect on the local Ethernet cable. That would happen if the local AP rebooted. But if the ISP can see the uptime there and it didn't then I would check the cable
-
@stephenw10 I woke up, the internet is down again. The system log says that particular MAC address is using our public IP (I have never seen that before, so something must have started happening yesterday).
The gateways log says this since I went to bed:Mar 22 08:55:51 dpinger 4486 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 217.168.xxx.xxx bind_addr 217.168.xxx.xxx identifier "WANGW " Mar 22 08:55:51 dpinger 8004 exiting on signal 15 Mar 22 08:54:06 dpinger 8004 WANGW 217.168.xxx.xxx: Clear latency 2398us stddev 2031us loss 5% Mar 22 08:52:28 dpinger 8004 WANGW 217.168.xxx.xxx: Alarm latency 0us stddev 0us loss 100% Mar 22 08:52:26 dpinger 8004 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 217.168.xxx.xxx bind_addr 217.168.xxx.xxx identifier "WANGW " Mar 22 08:52:26 dpinger 79965 exiting on signal 15 Mar 22 08:47:30 dpinger 79965 WANGW 217.168.xxx.xxx: Alarm latency 2444us stddev 1863us loss 22% Mar 22 05:07:52 dpinger 79965 WANGW 217.168.xxx.xxx: Clear latency 3401us stddev 2626us loss 0% Mar 22 05:07:00 dpinger 79965 WANGW 217.168.xxx.xxx: Alarm latency 18656861us stddev 16181793us loss 11% Mar 22 05:06:50 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:50 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:50 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:50 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:50 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:49 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:48 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:48 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:48 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:48 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:48 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:48 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:48 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:48 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:48 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 05:06:47 dpinger 79965 WANGW 217.168.xxx.xxx: Alarm latency 28029335us stddev 16895331us loss 90% Mar 22 04:44:52 dpinger 79965 WANGW 217.168.xxx.xxx: Alarm latency 2139us stddev 1306us loss 22% Mar 22 02:55:11 dpinger 79965 WANGW 217.168.xxx.xxx: Clear latency 4230us stddev 9071us loss 0% Mar 22 02:54:18 dpinger 79965 WANGW 217.168.xxx.xxx: Alarm latency 19691566us stddev 16695811us loss 10% Mar 22 02:54:08 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:08 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:08 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:08 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:08 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:08 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:08 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:08 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:08 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:07 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:06 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:06 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:06 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:06 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:06 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:06 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:06 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:06 dpinger 79965 WANGW 217.168.xxx.xxx: duplicate echo reply received Mar 22 02:54:05 dpinger 79965 WANGW 217.168.xxx.xxx: Alarm latency 14475616us stddev 5342208us loss 97% Mar 22 02:22:13 dpinger 79965 WANGW 217.168.xxx.xxx: Alarm latency 2558us stddev 2894us loss 22%And from the other logs I can see the unbound service seems to be stopping and starting all the time, but that's probably irrelevant. I don't understand any of this.
-
Something else in the WAN subnet using your IP address is a massive problem! That's going to break connectivity for you (and them). There's really no point investigating anything else whilst that is still present IMO.
If you can speak to the ISP I would tell them this is happening and see what they say. Most likely some other customer has typo'd the IP address. Or tried to just use another IP.
-
@stephenw10 Later yesterday I heard back from one of the guys from the ISP. He told me they found and blocked the MAC address and were investigating WTF was going on. I hope they will let me know.
It clearly was something inside the local wireless clients segment, but I can't imagine any possible cause aside from someone setting our IP address by mistake, but that makes little sense as well because that would likely be a permanent conflict and not something that happens repeatedly in semi-random interval of 30mins to 2 hours. It could have been something malicious but I cannot imagine any scenario there. -
Hanlon's Razor applies here.
It was probably just a mistake somewhere. Or perhaps some client thought they could just add more IPs to use and it wouldn't matter. If they didn't use them all the time that might explain it.
Anyway let us know if you still see any issues now that can't happen.
