Router Locking Up (maybe due to excessive lan traffic?)

stephenw10

Did you decide what 192.0.0.1 is?

One interesting thing is that it's detected as an APU1 but that's an APU2. What does the output of kenv | grep smbios show?

Ximulate

@stephenw10
RE 192.0..0.1
https://wirelessjoint.com/viewtopic.php?t=4191

My gut here is that the gateway pinger under pfsense is wreaking havoc if/when your IG is going between 192.168.225.x and 192.0.0.x. due to its default IPPT configuration. There are changes you can make to improve the situation with IPPT or simply disable it if you don't mind the additional NAT layer. All of this explained in detail if you read on :)
In addressing some of your issues it will first be good to shed some light on why you are seeing a 192.0.0.x address on your WAN interface even though the default UI access IP is 192.168.225.1. The InvisaGig provides IP Passthrough (IPPT) mode to the device connected to its Ethernet port by default and it uses ARP to find the connected device's MAC address which is to be assigned the IPPT address. This means that when the modem successfully connects to the carrier it bridges the IPv4 assigned by the carrier to the Ethernet interface of the InvisaGig.
If your cellular carrier network carrier uses IPv6 for the connection and you do not have a plan which specifically provides a statically assigned IPv4 address, the modem will connect to the carrier over IPv6 but perform IPv4 translation. This translation is referred to as dual stack lite, customer-side translation (CLAT) and will utilize an IPv4 address in the range defined by RFC 6333 which is typically 192.0.0.1 at the modem gateway and 192.0.0.2 assigned to the IG-connected IPPT device. In the USA, this scenario is most typical of T-Mobile consumer cellular plans and/or business plans without the static IP add-on so I am guessing this is your carrier. This is especially the case when connecting over 5G Standalone (5G SA) as that is IPv6 only. Under 5G Non-standalone (5G NSA) and LTE it is possible you may still receive a native, legacy IPv4 address from the carrier grade NAT (CGNAT) DHCP pool instead of a 192.0.0.x address.
Most devices and/or routers connected to the InvisaGig with the default IPPT functionality enabled will not have an issue with 192.0.0.x addressing on their WAN but from time to time we do see some enterprise equipment which has an issue with this address space and/or issues with IPPT bridge creation / MAC autodetection of its WAN port by the IG when reconnecting to the carrier. Depending on the root cause in the case of your PFSense router/firewall there may be a couple of settings which need changing. The first I would suggest would be to set the IPPT MAC address of the PFSense WAN port manually.

Ximulate

@stephenw10
RE kenv | grep smbios

smbios.bios.reldate="02/28/2017"
smbios.bios.revision="4.0"
smbios.bios.vendor="coreboot"
smbios.bios.version="4.0.7"
smbios.chassis.maker="PC Engines"
smbios.chassis.type="Desktop"
smbios.planar.location=""
smbios.planar.maker="PC Engines"
smbios.planar.product="APU2"
smbios.planar.serial="1131876"
smbios.planar.tag=""
smbios.planar.version="1.0"
smbios.system.maker="PC Engines"
smbios.system.product="APU2"
smbios.system.serial="1131876"
smbios.system.sku="4 GB"
smbios.system.version="1.0"
smbios.version="2.7"

JonathanLee

Is it overheating? I had a pcie mini card reboot randomly until I added heatsinks. My logs however just showed <Boot> nothing else before it. Weird one

Ximulate

@JonathanLee

I've wondered about that, but I don't see any evidince of it. The box get good air flow, its in a cool basement. I've even placed old heat sinks on the case (not attached) to help a little. Not sure that the screenshot below is going to post, but it shows fairly steady temps of about 55°C +/- 1° (the dip is when I power cycled)

Screenshot of Monitoring Graph

stephenw10

Ah yes DS-Lite. I see that so infrequently I always forget about it. So 192.0.0.2 is a legitimate IP address in that case.

I agree the switch from local DHCP to bridged is what I always suspected is the issue. The problem happens is the modem reboots or it;s link is lost and it starts handing out IPs in it's own subnet. When the link is re-established pfSense will not necessarily try to pull a new lease so fails until it renews.

This is quite common in cable modems and the usu workaround is to deny leases from the cable modem IP so it just keeps trying until the upstream link comes back up.

If you are seeing that it will be logged. The dhclient logs in the dhcp log will show the local modem address that's serving leases.

Ximulate

@stephenw10

Mar 13 19:48:49 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:48:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:48:54 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:48:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:04 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:10 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:22 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:39 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:50 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:50:13 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:50:29 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:50:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:51:47 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:52:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:52:54 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:04 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:24 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:45 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:54:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:54:29 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:54:39 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:54:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:55:13 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:55:38 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:55:45 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:55:56 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:56:11 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:56:22 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:56:32 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:56:47 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:57:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:57:53 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:58:34 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:58:56 router dhcpd[74227]: DHCPREQUEST for 10.11.11.64 from 00:15:88:6b:bb:4d via igb2.11
Mar 13 19:58:56 router dhcpd[74227]: DHCPACK on 10.11.11.64 to 00:15:88:6b:bb:4d via igb2.11
Mar 13 19:59:28 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 20:00:36 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 20:01:32 router dhcpd[74227]: DHCPREQUEST for 10.111.11.138 from 82:58:13:8d:90:9a (Pixel-8) via igb2.111
Mar 13 20:01:32 router dhcpd[74227]: DHCPACK on 10.111.11.138 to 82:58:13:8d:90:9a (Pixel-8) via igb2.111
Mar 13 20:01:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 20:03:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 20:05:14 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 20:06:08 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 20:08:35 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 20:08:44 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:08:54 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:09:04 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:09:18 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:09:43 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:10:05 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:10:28 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:10:41 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:10:55 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:11:30 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:13:01 router dhcpd[74227]: DHCPREQUEST for 10.111.11.134 from 00:15:cd:5a:43:44 (0005CD5A4344) via igb2.111
Mar 13 20:13:01 router dhcpd[74227]: DHCPACK on 10.111.11.134 to 00:15:cd:5a:43:44 (0005CD5A4344) via igb2.111
Mar 13 20:13:05 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:13:53 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:14:45 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:16:01 router dhcpd[74227]: DHCPREQUEST for 10.251.11.126 from dc:h6:32:9a:88:95 (svrRPI364b) via igb2.251
Mar 13 20:16:01 router dhcpd[74227]: Wrote 0 class decls to leases file.
Mar 13 20:16:01 router dhcpd[74227]: Wrote 0 deleted host decls to leases file.
Mar 13 20:16:01 router dhcpd[74227]: Wrote 0 new dynamic host decls to leases file.
Mar 13 20:16:01 router dhcpd[74227]: Wrote 129 leases to leases file.
Mar 13 20:16:01 router dhcpd[74227]: DHCPACK on 10.251.11.126 to dc:h6:32:9a:88:95 (svrRPI364b) via igb2.251
Mar 13 20:16:14 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:19:42 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:21:45 router dhcpd[74227]: DHCPREQUEST for 10.111.11.130 from e6:34:b1:e6:73:8a (Pixel-7a) via igb2.111
Mar 13 20:21:45 router dhcpd[74227]: DHCPACK on 10.111.11.130 to e6:34:b1:e6:73:8a (Pixel-7a) via igb2.111
Mar 13 20:21:48 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Mar 13 20:22:20 router dhclient[6155]: EXPIRE
Mar 13 20:22:20 router dhclient[7266]: Deleting old routes
Mar 13 20:22:20 router dhclient[8836]: PREINIT

Mar 13 20:54:36 router dhcpd[50201]: Server starting service.
Mar 13 20:54:36 router dhclient[51958]: PREINIT
Mar 13 20:54:36 router dhclient[53206]: EXPIRE
Mar 13 20:54:36 router dhclient[54111]: Deleting old routes
Mar 13 20:54:36 router dhclient[55441]: PREINIT
Mar 13 20:54:37 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 20:54:39 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 3
Mar 13 20:54:42 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 5
Mar 13 20:54:47 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 11
Mar 13 20:54:58 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12
Mar 13 20:55:10 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 15
Mar 13 20:55:25 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12
Mar 13 20:55:37 router dhclient[50960]: No DHCPOFFERS received.
Mar 13 20:55:37 router dhclient[50960]: No working leases in persistent database - sleeping.
Mar 13 20:55:37 router dhclient[92864]: FAIL
Mar 13 20:55:51 router dhcpd[82618]: Internet Systems Consortium DHCP Server 4.4.3-P1
Mar 13 20:55:51 router dhcpd[82618]: Copyright 2004-2022 Internet Systems Consortium.
Mar 13 20:55:51 router dhcpd[82618]: All rights reserved.
Mar 13 20:55:51 router dhcpd[82618]: For info, please visit https://www.isc.org/software/dhcp/
Mar 13 20:55:51 router dhcpd[82618]: Config file: /etc/dhcpd.conf
Mar 13 20:55:51 router dhcpd[82618]: Database file: /var/db/dhcpd.leases
Mar 13 20:55:51 router dhcpd[82618]: PID file: /var/run/dhcpd.pid
Mar 13 20:55:51 router dhcpd[82618]: Internet Systems Consortium DHCP Server 4.4.3-P1
Mar 13 20:55:51 router dhcpd[82618]: Copyright 2004-2022 Internet Systems Consortium.
Mar 13 20:55:51 router dhcpd[82618]: All rights reserved.
Mar 13 20:55:51 router dhcpd[82618]: For info, please visit https://www.isc.org/software/dhcp/
Mar 13 20:55:51 router dhcpd[82618]: Wrote 0 class decls to leases file.
Mar 13 20:55:51 router dhcpd[82618]: Wrote 0 deleted host decls to leases file.
Mar 13 20:55:51 router dhcpd[82618]: Wrote 0 new dynamic host decls to leases file.
Mar 13 20:55:51 router dhcpd[82618]: Wrote 129 leases to leases file.
Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.101/00:0d:b9:a9:16:92/10.101.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Sending on   BPF/igb2.101/00:0d:b9:a9:16:92/10.101.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.251/00:0d:b9:a9:16:92/10.251.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Sending on   BPF/igb2.251/00:0d:b9:a9:16:92/10.251.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.111/00:0d:b9:a9:16:92/10.111.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Sending on   BPF/igb2.111/00:0d:b9:a9:16:92/10.111.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.31/00:0d:b9:a9:16:92/10.31.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Sending on   BPF/igb2.31/00:0d:b9:a9:16:92/10.31.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.11/00:0d:b9:a9:16:92/10.11.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Sending on   BPF/igb2.11/00:0d:b9:a9:16:92/10.11.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2/00:0d:b9:a9:16:92/10.0.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Sending on   BPF/igb2/00:0d:b9:a9:16:92/10.0.11.0/24
Mar 13 20:55:51 router dhcpd[82618]: Sending on   Socket/fallback/fallback-net
Mar 13 20:55:51 router dhcpd[82618]: Server starting service.
Mar 13 20:55:52 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 20:55:54 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 4
Mar 13 20:55:58 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 6
Mar 13 20:56:04 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 10
Mar 13 20:56:14 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 17
Mar 13 20:56:31 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 20
Mar 13 20:56:51 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 20:56:53 router dhclient[50960]: No DHCPOFFERS received.
Mar 13 20:56:53 router dhclient[50960]: No working leases in persistent database - sleeping.
Mar 13 20:56:53 router dhclient[34633]: FAIL

stephenw10

Hmm, nothing shown there at all. No responses from any DHCP server. That was during some outage I assume?

Ximulate

@stephenw10
Right, that was about the time the network failure occured yesterday. I don't know exactly when it started failing, but I was troubleshooting prior to having to power cycle at 21:47 (see log in yesterday's post).

Mar 13 19:44:30 router dhcpd[74227]: DHCPDISCOVER from 30:e9:50:8e:e2:91 (Sweeper_p100) via igb2.31
Mar 13 19:44:30 router dhcpd[74227]: DHCPOFFER on 10.31.11.235 to 30:e9:50:8e:e2:91 (Sweeper_p100) via igb2.31
Mar 13 19:44:31 router dhcpd[74227]: reuse_lease: lease age 1510 (secs) under 25% threshold, reply with unaltered, existing lease for 10.31.11.235
Mar 13 19:44:31 router dhcpd[74227]: DHCPREQUEST for 10.31.11.235 (10.31.11.1) from 30:e9:50:8e:e2:91 (Sweeper_p100) via igb2.31
Mar 13 19:48:49 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:48:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:48:54 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:48:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:04 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:10 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:22 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:39 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:49:50 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:50:13 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:50:29 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:50:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:51:47 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:52:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:52:54 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:04 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:24 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:53:45 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:54:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:54:29 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:54:39 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:54:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:55:13 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:55:38 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:55:45 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:55:56 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:56:11 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:56:22 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:56:32 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:56:47 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:57:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:57:53 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:58:34 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 19:58:56 router dhcpd[74227]: DHCPREQUEST for 10.11.11.64 from 00:15:88:6b:bb:4d via igb2.11
Mar 13 19:58:56 router dhcpd[74227]: DHCPACK on 10.11.11.64 to 00:15:88:6b:bb:4d via igb2.11
Mar 13 19:59:28 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 20:00:36 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67
Mar 13 20:01:32 router dhcpd[74227]: DHCPREQUEST for 10.111.11.138 from 82:58:13:8d:90:9a (Pixel-8) via igb2.111
Mar 13 20:01:32 router dhcpd[74227]: DHCPACK on 10.111.11.138 to 82:58:13:8d:90:9a (Pixel-8) via igb2.111

Then is a lot of the following DHCP logs just prior to power cycle:

Mar 13 21:29:49 router dhclient[9249]: FAIL
Mar 13 21:30:04 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 1
Mar 13 21:30:05 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 21:30:07 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 5
Mar 13 21:30:12 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 6
Mar 13 21:30:18 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 15
Mar 13 21:30:33 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 18
Mar 13 21:30:51 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12
Mar 13 21:31:03 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 21:31:05 router dhclient[50960]: No DHCPOFFERS received.
Mar 13 21:31:05 router dhclient[50960]: No working leases in persistent database - sleeping.
Mar 13 21:31:05 router dhclient[53331]: FAIL
Mar 13 21:31:20 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 21:31:22 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 4
Mar 13 21:31:26 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 4
Mar 13 21:31:30 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 10
Mar 13 21:31:40 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 13
Mar 13 21:31:53 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 16
Mar 13 21:32:06 router dhcpd[82618]: uid lease 10.111.11.126 for client 48:a9:44:91:ec:4f is duplicate on 10.111.11.0/24
Mar 13 21:32:06 router dhcpd[82618]: DHCPREQUEST for 10.111.11.51 from 48:e9:44:91:ec:4f via igb2.111
Mar 13 21:32:06 router dhcpd[82618]: DHCPACK on 10.111.11.51 to 48:a9:44:91:ec:4f via igb2.111
Mar 13 21:32:09 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12
Mar 13 21:32:13 router dhcpd[82618]: DHCPREQUEST for 10.0.11.121 from 78:a9:58:dd:69:cf (MstrCloset) via igb2
Mar 13 21:32:13 router dhcpd[82618]: DHCPACK on 10.0.11.121 to 78:a9:58:dd:69:cf (MstrCloset) via igb2
Mar 13 21:32:18 router dhcpd[82618]: DHCPREQUEST for 10.11.11.62 from dc:a6:32:9a:88:8c via igb2.11
Mar 13 21:32:18 router dhcpd[82618]: DHCPACK on 10.11.11.62 to dc:a6:32:9a:88:8c via igb2.11
Mar 13 21:32:21 router dhclient[50960]: No DHCPOFFERS received.
Mar 13 21:32:21 router dhclient[50960]: No working leases in persistent database - sleeping.
Mar 13 21:32:21 router dhclient[22]: FAIL
Mar 13 21:32:36 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 1
Mar 13 21:32:37 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 21:32:39 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 21:32:41 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 5
Mar 13 21:32:44 router dhcpd[82618]: DHCPREQUEST for 10.11.11.91 from 00:e9:9d:db:a6:54 via igb2.11
Mar 13 21:32:44 router dhcpd[82618]: DHCPACK on 10.11.11.91 to 00:e9:9d:db:a6:54 via igb2.11
Mar 13 21:32:46 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 9
Mar 13 21:32:55 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 16
Mar 13 21:33:11 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 7
Mar 13 21:33:18 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 11
Mar 13 21:33:29 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 8
Mar 13 21:33:37 router dhclient[50960]: No DHCPOFFERS received.
Mar 13 21:33:37 router dhclient[50960]: No working leases in persistent database - sleeping.
Mar 13 21:33:37 router dhclient[79182]: FAIL
Mar 13 21:33:52 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 21:33:54 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 4
Mar 13 21:33:58 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 11
Mar 13 21:34:09 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 11
Mar 13 21:34:20 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12
Mar 13 21:34:32 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 15
Mar 13 21:34:47 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 6
Mar 13 21:34:53 router dhclient[50960]: No DHCPOFFERS received.
Mar 13 21:34:53 router dhclient[50960]: No working leases in persistent database - sleeping.
Mar 13 21:34:53 router dhclient[36776]: FAIL
Mar 13 21:35:08 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 1
Mar 13 21:35:09 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2
Mar 13 21:35:11 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 5
Mar 13 21:35:16 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 10
Mar 13 21:35:18 router dhcpd[82618]: DHCPREQUEST for 10.0.11.33 from 78:a9:58:46:f9:44 via igb2
Mar 13 21:35:18 router dhcpd[82618]: DHCPACK on 10.0.11.33 to 78:a9:58:46:f9:44 via igb2
Mar 13 21:35:26 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 13
Mar 13 21:47:13 router dhclient[30088]: PREINIT

I just noticed this in above, which is our printer. However, there is no duplicate in the currect DHCP leases.

Mar 13 21:32:06 router dhcpd[82618]: uid lease 10.111.11.126 for client 48:a9:44:91:ec:4f is duplicate on 10.111.11.0/24
Mar 13 21:32:06 router dhcpd[82618]: DHCPREQUEST for 10.111.11.51 from 48:e9:44:91:ec:4f via igb2.111
Mar 13 21:32:06 router dhcpd[82618]: DHCPACK on 10.111.11.51 to 48:a9:44:91:ec:4f via igb2.111

stephenw10

Hmm, so it appears that when this goes down there are no responding DHCP servers on the WAN. So either the modem stops both bridging the ISP DS-Lite connection or handing out it's own leases. Or the NIC in pfSense is not actually sending the DHCP leases (or not seeing the responses).

Are you able to try to capture traffic on the WAN side?

Since you're already behind double NAT you might just try disabling the IP pass through in the modem. That would avoid the IP/subnet change. It might also prevent the modem failing if that's what is happening.

Ximulate

@stephenw10
It would take some time for me to set-up, but I could probably set-up wireshark on a RasPi & a small switch to log packets (I think thats what your suggesting.)

There is a way to disable IP pass though in the modem, so I'll try that next.

stephenw10

Yep an ideal test there would be capturing packets from a switch mirror port but that's clearly quite involved to get in place.

That would prove it's not an issue with the NIC though.

Ximulate

@stephenw10 So I'm back this
I had all but decided its time to buy a new router, but had to go out of town so I threw together the script descibed below executed by cron ever 12 minutes. That was about 3 weeks ago. Router hasn't had any issue until today.

The only reason it an issue was that I was testing the back-up connection by killing power the primary wan modem. Failover was fine, but when I brought the primary WAN back online the LAN comms would die. Looking at the log generated by my script, it appears the script has cycled the wan several times over the last few weeks which makes me think the script mostly works but I'm not sure why or what problem its resolving by cycling the wan.

gwstat=$(pfSsh.php playback gatewaystatus)
WAN_STATE=$(echo "$gwstat" | awk '/'$GW_ID'/ { print $7}')
if [ $WAN_STATE = "online" ]; then exit; fi
echo "WAN Cycling on $varDate" >> $log
#turn off modem using Hue Appliance Plug (doubtful this works, because lan comms are down so the Hue Hub is prob unreachable. Just now modifed script to log ouput)
/usr/local/bin/curl -X PUT -H "Content-Type: application/json" -d '{"on": false}' $url >> $log
ifconfig igb0 down
# Sleep, then bring up modem
sleep 30
/usr/local/bin/curl -X PUT -H "Content-Type: application/json" -d '{"on": true}' $url >> $log
ifconfig igb0 up

stephenw10

Hmm, well if you kill the WAN connection deliberately that script is going to continually try to cycle the WAN modem. Though I'm not sure why that would prevent it coming back up or otherwise kill the LAN side connectivity.

VioletDragon

One thing that was neglected to mention here, is Memtest. I’d suggest running memtest to see if there are any memory errors.

Second. Test another Power Supply, recently I had a client that had the same problem due to a faulty 12v rail.

Regards

Ximulate

@stephenw10

Network just went down again, kind of, and through no action of mine. This time, I was able to load internet webpages through my fail-over connection (so that was working). I could ping the router over the LAN. I really don't think the primary WAN was down, as once I power cycled it was online. I also confirmed that the power plug was on (so everything was getting power and should be online.)

However, the router GUI was reporting a 504 error (white page with black text served by ngnix.) I was able SSH into the router, so I tried to restart PHP-FAM. This allowed me to load the GUI login, but after entering credentials the browser waited, waited, waited and then back to 504 error. Ended up power cycling.

Per @VioletDragon suggestion, I just replaced the power supply. Earlier in the troubleshooting process, I think I already tried another new power supply (maybe 3rd's the charm) and I had already run memtest overnight, no errors reported.

VioletDragon

@Ximulate Just out of interest. Can you share System -> Advanced -> Network Interfaces.

Are you routing Layer 2 or 3 ? Is Jumbo Frames been left at default 1500 ?

Have you tested a fresh install ?

Regards.

Ximulate

@VioletDragon
Hardware Checksum Offloading [Unchecked - enabled]
Hardware TCP Segmentation Offloading [Checked - disabled]
Hardware Large Receive Offloading [Checked - disabled]
hn ALTQ support [checked - enabled]
ARP Handling {unchecked - do not suppress]
Reset All States [unchecked - do not reset]

Ximulate

@VioletDragon
MTU is blank on all interfaces, so I assume default / 1500
In so far as I understand OSI, its all Layer 3. Its all firewall rules, no ethernet rules.
No I haven't tried a fresh install. I guess I should do that.