Subnet collapses periodically since 24.11-RELEASE
-
@johnpoz For some reason (perhaps it was the lack of !) the first image did not show (but it is clickable to show, yes, I have it on static).
I have never seen "track interface" before (which is enabled for ipv6). I don't know if that was part of the new release (or the one before).
Next time I'll bring a cable from the lan port to my own laptop to see if I can go onto the GUI. Prior attempts to see what was going on failed (save bringing out the serial cable which I didn't have time for, but I'll make time for it next time).
What I don't understand still is pfsense not communicating to the other PCs/printers to update the address when I reboot the pfsense (even if I refresh the DHCP service I would expect the ip address to update on the PC, but it doesn't).
-
@vf1954 once a client gets a lease, it doesn't care if the dhcp server is on, changes its IP range.. The only time dhcp client cares for a dhcp server is when it tries to renew, which again even no dhcp server the client is happy with the IP it had.. it will start screaming faster and faster hey give me a renewal.. When the lease finally expires it will then send a out discover.
So no your clients are not going to change their ip just because you rebooted pfsense.
-
@johnpoz Okay, that is good to know. Thank you for explaining that to me. How, then, would one change the clients IP from within pfsense? (If at all...?). Seems like this is a task beyond pfsense. But if that is the case, even more an issue as one has to be physically present in front of every PC (unless the lease expires)
-
@vf1954 you can't really change a clients IP on a whim from just pfsense, you can sure give it a reservation and then if you reboot it or release renew its lease etc it would get the new IP.
Same goes if you changed your IP range, clients wouldn't get an IP from the new lease unless the client was asking for an ip, etc.
I do this pretty much any time I bring a new device online - I let it get an IP, then I change it to reservation and have the client then reboot or release/renew so it now gets the reservation.
If the device was poe, you could prob force the IP change by cycling its ethernet port on your switch.
How fast a client would move over to a reservation or would depend on the length of the lease you gave it to start with - if your lease is like the default of 2 hours... Then 2 hours later all devices would be on the new iP range or be using a reservation you set for them, it could be faster.. But for sure you would know 2 hours all devices would have a new IP.
If your lease was like 8 days - then yeah it could take up to 8 days to move to the new IP without intervention on your part at the client.
-
@johnpoz I got a bit further. It went down three times since we last talked, this being the third. Thankfully it's midnight so I can finally test around without having to immediately restart it. Here are some findings.
Setup
Internet -> netgate/pfsense -> {wifi_router1, aruba switch}
-> wifi_router1 -> wifi_router2 -> wifi_router3
(Easy-Mesh Tp-link which disables router2/router3)
-> aruba switch -> 2nd tp-link switchTesting
- plugging in eth cable directly from netgate LAN -> laptop (running linux) does not produce a connection.
- therefore, no access to online GUI
- access to serial shows uptime of 6 days.
- I can ping 1.1.1.1 in pfsense shell, but i cannot ping domain (DNS server is pi-hole that is now on the 192.168.0.x network)
- wifi access gave me gateway of 192.168.0.1
- logging into 192.168.0.1 sent me to the second switch
- second switch was set to manual DHCP, IP 192.168.0.1 with 0.0.0.0 as gateway (not 100% sure if it went to static IP automatically but when pfsense is back up I'll create a rule for it)
- changed 2nd TP-switch to automatically get IP from DHCP server (i.e., netgate pfsense) and restarted ...
- no more access to tp-link switch ... -.-" still on 192.168.0.x connected to wifi (with no internet access)
- ran ip route | grep default and found new gateway at 192.168.0.254 (which is a TP-link router). TP-link router not accessible as I use 3 of them with Easy-Mesh and disabled DHCP... so likely using wifi_router2
- physically disconnected switch2 and router2 forcing me to go to wifi_router1 only
- 192.168.0.254 still gateway (surprised me). Still not accessible (now I'm only using wifi_router1 which I should be able to access...)
(also, wifi_router1 is set to 192.168.3.3 in pfsense) - not sure what other test I can run while under serial shell for pfsense...
Will restart system and ensure 2nd switch is in DHCP rules and update wifi_router firmware.
-
Your LAN :
so 192.168.3.2/24
Why not 192.168.3.1/24 ? .2 is ok of course, any .1 to .253 is ok - but 'strange'.@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:
192.168.0.1 sent ...
Where does this network come from ? It's not a pfSense interface.
You have a router-after-router setup ? ( ! ). Why ? Again, it can be done, it can work, but why make a more complicated network like that ?
What about the god old [ISP] <=> [pfSense WAN <-> pfSense LAN] <=> switch <=> (all your PCs, APs, all other devices)
Your PCs and all other device will use the default DHCP, so they will connect.
If you use APs, set them up with static IPs like 192.168.3.3 192.168.3.4 etc - they will all have their gateway set to 192.168.3.2 (pfSense) - disable on all APs the DHCP server - set the DNS on all APs to 192.168.3.2 (pfSense) - if your APs have a labeled "WAN" port do not use it, use a LAN port. after all, you use the APs as an AP, you don't want them to use as a 'router'. pfSense your one and only router.@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:
plugging in eth cable directly from netgate LAN -> laptop (running linux) does not produce a connection.Before plugging your laptop into the pfSense LAN port : check :
Is the pfSense DHCP server up and running ?edit : on console, menu option 8, type
ps aux | grep 'kea'If you use ISC :
ps aux | grep 'kea'end edit.
Is the laptop using DHCP client (default, it is) ?
Now, console access pfSense, menu option 8 :tail -f /var/log/dhcpd.logand now connect you laptop.
What shows up ? -
@Gertjan Thank you for your wonderful reply.
I have everything up and running since I reset the pfsense.
I have router after router because I use them as an "Easy-Mesh" network so the company can traverse the entire property without dropping the signal. So the "routers" don't actually do any DHCP. If I make all 3 AP then I lose the Easy-Mesh functionality.
The only problem is whenever I update firmware I have to start the entire process over again because these TP Archers are not connected via WAN but LAN.
.2 was because .1 was problematic due to our ISP. Today I may revert back to .1 but meh.
I suspect something strange is occuring with the routers. So I completely re-programmed them and updated the firmware. I also set a few key components to static (like the DNS and that second switch)
If the network goes down again, I'll follow your advice with the shell prompts (I assume the second one was meant to say 'isc')? Thank you so much!
-
@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:
.2 was because .1 was problematic due to our ISP
Hummmmm
You took .2 because .1 was already used ? Like "192.168.3.1" is already occupied on LAN ? WAN ? Where ? On WAN ? If so, you can't use 192.168.3.x/24 on LAN. -
@Gertjan This was many years ago.
192.168.3.1 is not in use. But since so many clients have 192.168.3.2 hardcoded it's best to just use .2
Clearly updating the firmware didn't solve the problem.
It happens just randomly. Today at 3PM I suddenly lose wifi and ethernet access. And more bizarre, only a few computers, but progressively all of them.
Uptime is currently 7 days.
When I run
ps aux | grep 'isc'I get
root 1651 0.0 0.1 4672 2256 u0 S+ 15:34 0:00.01 grep iscrunning
tail -f /var/log/dhcpd.logProduces
Sending to Solicit (multiple lines)The actual time it takes to even get a connection is a good 45 seconds, and then I just get a ? on the wired connection on ubuntu laptop and when I go to properties of the wired connection ... no IP shows up.
When I wrote the 'kea' I get more dhcp6 stuff (which is turned off in the GUI)
What is happening?
:(
-
@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:
tail -f /var/log/dhcpd.log
Doing it after I reboot the netgate produces some warnings
-
@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:
192.168.0.254 (which is a TP-link router)
Where is this set on your TP-Link? How is it connected to your pfSense LAN network?
-
@SteveITS It is simply plugged in, gets assigned a lan address from pfsense at 192.168.3.3, and then that's it
-
@SteveITS sorry I see what you mean.
It is set at 192.168.3.3 in hte LAN settings in tplink
AND
it is set to 192.168.3.3 in pfsense dhcp static.
-
@vf1954 So, what is the .254 you mentioned?
Screencap the change in pfSense when this happens.
If the fields in pfSense aren’t changing I suspect what you’re seeing is another DHCP server. Windows and I’m sure other clients will show the DHCP server used for example “ipconfig /all”
-
@SteveITS said in Subnet collapses periodically since 24.11-RELEASE:
Screencap the change in pfSense when this happens.
Not sure what you mean here. Does screencap mean screenshot? Screenshot what?
The address being circulated is 192.168.0.xx but the other DHCP router is the wifi which is turned off.
-
@vf1954 yes, screenshot pfSense with the changed settings, or some evidence.
If you’re not saying anything in pfSense actually changes then it’s not pfSense. Unplug pfSense LAN, restart a client, and see what it’s IP and DHCP server are.
-
@SteveITS Fair enough. But why would pfsense just give up its DHCP authority ... randomly ... after 6-14 days?
-
@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:
DHCP authority ... randomly ... after 6-14 days?
what authority?? When a client does a discover - the first dhcp server that answers wins..
If there is more than 1 dhcp server on your network - its a coinflip who will answer first.
You can run more than 1 on the same network.. But they need to hand out the same info.. This is can be done as a failover scenario - where you split the scope between then..
Say dhcpd1 hands out 192.168.1.10-128
Where dhcpd2 hands out 192.168.1.129-244Both point say to 192.168.1.1 for dns and gateway.. Leaving you .2-9 and .245-254 as IPs you can set statically on devices.
But if your handing out different IP range and different gateway - yeah your going to have a bad day on clients that get IP from that dhcp server.
-
@johnpoz Hello John,
Yes you taught me something new again. I thought DHCP holds authority.
But regardless, even if two DHCP servers were vying for the same "authority" (to grant leases), I'd expect, statistically, that many of the clients would choose 192.168.0.x and lose network/internet access and that that would appear sporadically during the day/week. This is not the behaviour. It is perfectly stable with netgate "in charge", all the time, for all clients, until suddenly every client decides to pivot to 192.168.0.x (albeit at different times, but once one goes the rest will follow within an hour).
You would think they all magically pick up netgate after a couple hours... but they don't either. the pfsense just become inaccessible until I console into it.
my two switches are hardcoded to be on 192.168.3.x address.
my 3 tp-link archer 5400 are set as 192.168.3.3 .4 .5 on easy-mesh with the primary dhcp = off.There is no other dhcp server afaik.
-
@vf1954 clearly there is.. Here do this.. Look at your client currently.
What does it list for the dhcp server?
$ ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : i9-win Primary Dns Suffix . . . . . . . : home.arpa Node Type . . . . . . . . . . . . : Broadcast IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : home.arpa Ethernet adapter Local: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Killer E2600 Gigabit Ethernet Controller Physical Address. . . . . . . . . : B0-4F-13-0B-FD-16 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes IPv4 Address. . . . . . . . . . . : 192.168.9.100(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Lease Obtained. . . . . . . . . . : Friday, February 14, 2025 2:01:59 PM Lease Expires . . . . . . . . . . : Tuesday, February 18, 2025 2:02:00 PM Default Gateway . . . . . . . . . : 192.168.9.253 DHCP Server . . . . . . . . . . . : 192.168.9.253 DNS Servers . . . . . . . . . . . : 192.168.3.10 NetBIOS over Tcpip. . . . . . . . : Enabled192.168.9.253 is my pfsense.. now if I look at the mac address
$ arp -a Interface: 192.168.9.100 --- 0x5 Internet Address Physical Address Type 192.168.9.10 00-11-32-7b-29-7d dynamic 192.168.9.253 00-08-a2-0c-e6-24 dynamic 192.168.9.255 ff-ff-ff-ff-ff-ff static 224.0.0.22 01-00-5e-00-00-16 static 239.255.255.250 01-00-5e-7f-ff-fa static 255.255.255.255 ff-ff-ff-ff-ff-ff staticSo its mac is 00-08-a2-0c-e6-24. If pfsense was out of the blue changing its IP and dhcp scope, that that mac address would be the same.
As to why your not seeing a random distribution, maybe pfsense dhcp answers faster - but when it goes offline the only one to answer is your other dhcp server.
Pfsense is just not going to randomly change its IP address.. You either changing it, or your loading a bad/old config? Looking to what mac address your dhcp server is at will tell you for sure that its pfsense, or its some other box.
