Subnet collapses periodically since 24.11-RELEASE
-
I am having a very difficult problem to solve.
Occasionally (it has happened 3 times now in 1 week) my netgate will collapse my 192.168.3.xxx LAN to 192.168.0.xxx which basically disables the entire network. VNC services, printers, the whole shebang becomes inaccessible.
Reseting the netgate pfsense router basically fixes this. This means it is physically dependent on a person to press the reset button. Not ideal, especially with people dialing in at different time zones and remotely after working hours.
When I go to DHCP Service and Refresh, the various linux servers, printers, and PC still have 192.168.0.xx address. When I reboot these servers, printers, PCs, they receive a 192.168.3.xx address. This is also not good.
When I check DHCP leases under Status I don't see any 192.168.0.xx entries.
DHCP is disabled in all the wifi routers that I use strictly for wifi and nothing else so I can't assume those TP-link Archer X73s are the culprit.
Our system has been stable for over a year, which may indicate the update is to blame, but not 100% sure.
Other things I can think of
- updated to 24.11-RELEASE
- played around with HAproxy
- saw a php error one day which was a memory issue, but it has never come back either ...
I unfortunately have nothing more to go on at this point, not even to diagnose.
Would appreciate any insight 
-
@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:
which may indicate the update is to blame
While that is possible that something changed with the update.. I am on 24.11 with multiple networks and they are not collapsing or changing in any way..
There is zero reason why a static IP set on an interface in pfsense would ever just change to a different network. Is your network actually 192.168/16 or something or 192.168.0/22 or something and your dhcp pool is changing?
Is this network a vlan on pfsense?
If pfsense somehow changed its IP to 192.168.0 vs 192.168.3 - and your clients got a dhcp from 192.168.0 then they would be listed under leases on pfsense. Another theory of what might be going on is you have some other dhcp server on the network handing out these 192.168.0 addresses..
So if it happens again, before go rebooting pfsense - look on pfsense for its actual address on this interface.. simple ifconfig would work if you can not get to the gui? from console for example
But pfsense doesn't just randomly change the IP address on an interface..
Lets see your current config for your dhcp and interface.. Example here is my lan and dhcp server settings.
So lets look at what yours is currently, and then when you see the problem again if you do - lets see what pfsense shows.
-
@vf1954
To start, I would choose a pre-update version in Boot Environments and run it for a while to make sure the issue is specifically with the 24.11 update and subsequent actions, rather than, say, a hardware problem. -
@johnpoz said in Subnet collapses periodically since 24.11-RELEASE:
There is zero reason why a static IP set on an interface in pfsense would ever just change to a different network.
I agree. Hence why I am scratching my head.
Thank you for helping so quickly. Let me get you some more information:
LAN is on 192.168.3/24
I do not use vlans.
I have not ruled out a random DHCP server acting out, but currently, afaik, I have disabled all DHCP servers except the netgate. The three tp-link archers are linked together (ethernet) with easymesh and dhcp is off. What I can say is that when I reset the netgate, the leases are all 192.168.3.xx yet the PCs (that haven't been restarted) are still on 192.168.0.xx until I reboot them too.
Next time this happens, I will do as you requested: log into through serial console and do ifconfig. Last three times I had to urgently reset it for logistic reasons.
Please see attached my configs requested!
netgate-lan1.png
-
@vf1954 you don't have to use console - that is only if you can not get to the lan interface.. But I would bet a large sum of money there is no way pfsense just out of the blue changes its static IP, which you left off your lan pic is set to static right? I have to assume that from the 192.168.3.2 /24 setting..
You sure its not rebooting and booting say a different image, like a previous one.. Just up out of the blue change its IP - yeah that makes zero sense.. Validate your uptime once it happens again before you go and reboot it.. Rebooting is really the last thing that should ever be done when troubleshooting something.. Its like a hail mary pass with 2 seconds left on the clock from the 50 yard line.. ;)
Or somehow a previous config got loaded? But normally for those to take effect you have to do a reboot. Yeah If I thought my pfsense just out of the blue changed its IP - I would be for sure scratching my head going WTF ;)
-
@johnpoz For some reason (perhaps it was the lack of !) the first image did not show (but it is clickable to show, yes, I have it on static).
I have never seen "track interface" before (which is enabled for ipv6). I don't know if that was part of the new release (or the one before).
Next time I'll bring a cable from the lan port to my own laptop to see if I can go onto the GUI. Prior attempts to see what was going on failed (save bringing out the serial cable which I didn't have time for, but I'll make time for it next time).
What I don't understand still is pfsense not communicating to the other PCs/printers to update the address when I reboot the pfsense (even if I refresh the DHCP service I would expect the ip address to update on the PC, but it doesn't).
-
@vf1954 once a client gets a lease, it doesn't care if the dhcp server is on, changes its IP range.. The only time dhcp client cares for a dhcp server is when it tries to renew, which again even no dhcp server the client is happy with the IP it had.. it will start screaming faster and faster hey give me a renewal.. When the lease finally expires it will then send a out discover.
So no your clients are not going to change their ip just because you rebooted pfsense.
-
@johnpoz Okay, that is good to know. Thank you for explaining that to me. How, then, would one change the clients IP from within pfsense? (If at all...?). Seems like this is a task beyond pfsense. But if that is the case, even more an issue as one has to be physically present in front of every PC (unless the lease expires)
-
@vf1954 you can't really change a clients IP on a whim from just pfsense, you can sure give it a reservation and then if you reboot it or release renew its lease etc it would get the new IP.
Same goes if you changed your IP range, clients wouldn't get an IP from the new lease unless the client was asking for an ip, etc.
I do this pretty much any time I bring a new device online - I let it get an IP, then I change it to reservation and have the client then reboot or release/renew so it now gets the reservation.
If the device was poe, you could prob force the IP change by cycling its ethernet port on your switch.
How fast a client would move over to a reservation or would depend on the length of the lease you gave it to start with - if your lease is like the default of 2 hours... Then 2 hours later all devices would be on the new iP range or be using a reservation you set for them, it could be faster.. But for sure you would know 2 hours all devices would have a new IP.
If your lease was like 8 days - then yeah it could take up to 8 days to move to the new IP without intervention on your part at the client.
