Subnet collapses periodically since 24.11-RELEASE

[vf1954]

@johnpoz I got a bit further. It went down three times since we last talked, this being the third. Thankfully it's midnight so I can finally test around without having to immediately restart it. Here are some findings.

Setup
Internet -> netgate/pfsense -> {wifi_router1, aruba switch}
-> wifi_router1 -> wifi_router2 -> wifi_router3
(Easy-Mesh Tp-link which disables router2/router3)
-> aruba switch -> 2nd tp-link switch

Testing

1. plugging in eth cable directly from netgate LAN -> laptop (running linux) does not produce a connection.
2. therefore, no access to online GUI
3. access to serial shows uptime of 6 days.
4. I can ping 1.1.1.1 in pfsense shell, but i cannot ping domain (DNS server is pi-hole that is now on the 192.168.0.x network)
5. wifi access gave me gateway of 192.168.0.1
6. logging into 192.168.0.1 sent me to the second switch
7. second switch was set to manual DHCP, IP 192.168.0.1 with 0.0.0.0 as gateway (not 100% sure if it went to static IP automatically but when pfsense is back up I'll create a rule for it)
8. changed 2nd TP-switch to automatically get IP from DHCP server (i.e., netgate pfsense) and restarted ...
9. no more access to tp-link switch ... -.-" still on 192.168.0.x connected to wifi (with no internet access)
10. ran ip route | grep default and found new gateway at 192.168.0.254 (which is a TP-link router). TP-link router not accessible as I use 3 of them with Easy-Mesh and disabled DHCP... so likely using wifi_router2
11. physically disconnected switch2 and router2 forcing me to go to wifi_router1 only
12. 192.168.0.254 still gateway (surprised me). Still not accessible (now I'm only using wifi_router1 which I should be able to access...)
    (also, wifi_router1 is set to 192.168.3.3 in pfsense)
13. not sure what other test I can run while under serial shell for pfsense...

Will restart system and ensure 2nd switch is in DHCP rules and update wifi_router firmware.

[Gertjan]

@vf1954

Your LAN :

so 192.168.3.2/24
Why not 192.168.3.1/24 ? .2 is ok of course, any .1 to .253 is ok - but 'strange'.

@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:

192.168.0.1 sent ...

Where does this network come from ? It's not a pfSense interface.
You have a router-after-router setup ? ( ! ). Why ? Again, it can be done, it can work, but why make a more complicated network like that ?
What about the god old [ISP] <=> [pfSense WAN <-> pfSense LAN] <=> switch <=> (all your PCs, APs, all other devices)
Your PCs and all other device will use the default DHCP, so they will connect.
If you use APs, set them up with static IPs like 192.168.3.3 192.168.3.4 etc - they will all have their gateway set to 192.168.3.2 (pfSense) - disable on all APs the DHCP server - set the DNS on all APs to 192.168.3.2 (pfSense) - if your APs have a labeled "WAN" port do not use it, use a LAN port. after all, you use the APs as an AP, you don't want them to use as a 'router'. pfSense your one and only router.

@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:

plugging in eth cable directly from netgate LAN -> laptop (running linux) does not produce a connection.

Before plugging your laptop into the pfSense LAN port : check :
Is the pfSense DHCP server up and running ?

edit : on console, menu option 8, type

ps aux | grep 'kea'

If you use ISC :

ps aux | grep 'kea'

end edit.

Is the laptop using DHCP client (default, it is) ?
Now, console access pfSense, menu option 8 :

tail -f /var/log/dhcpd.log

and now connect you laptop.
What shows up ?

[vf1954]

@Gertjan Thank you for your wonderful reply.

I have everything up and running since I reset the pfsense.

I have router after router because I use them as an "Easy-Mesh" network so the company can traverse the entire property without dropping the signal. So the "routers" don't actually do any DHCP. If I make all 3 AP then I lose the Easy-Mesh functionality.

The only problem is whenever I update firmware I have to start the entire process over again because these TP Archers are not connected via WAN but LAN.

.2 was because .1 was problematic due to our ISP. Today I may revert back to .1 but meh.

I suspect something strange is occuring with the routers. So I completely re-programmed them and updated the firmware. I also set a few key components to static (like the DNS and that second switch)

If the network goes down again, I'll follow your advice with the shell prompts (I assume the second one was meant to say 'isc')? Thank you so much!

[Gertjan]

@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:

.2 was because .1 was problematic due to our ISP

Hummmmm
You took .2 because .1 was already used ? Like "192.168.3.1" is already occupied on LAN ? WAN ? Where ? On WAN ? If so, you can't use 192.168.3.x/24 on LAN.

[vf1954]

@Gertjan This was many years ago.

192.168.3.1 is not in use. But since so many clients have 192.168.3.2 hardcoded it's best to just use .2

Clearly updating the firmware didn't solve the problem.

It happens just randomly. Today at 3PM I suddenly lose wifi and ethernet access. And more bizarre, only a few computers, but progressively all of them.

Uptime is currently 7 days.

When I run

ps aux | grep 'isc'

I get

root   1651   0.0   0.1   4672   2256   u0   S+   15:34        0:00.01 grep isc

running

tail -f /var/log/dhcpd.log

Produces

Sending to Solicit (multiple lines)

The actual time it takes to even get a connection is a good 45 seconds, and then I just get a ? on the wired connection on ubuntu laptop and when I go to properties of the wired connection ... no IP shows up.

When I wrote the 'kea' I get more dhcp6 stuff (which is turned off in the GUI)

What is happening?

:(

[vf1954]

@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:

tail -f /var/log/dhcpd.log

Doing it after I reboot the netgate produces some warnings

[SteveITS]

@vf1954 said in Subnet collapses periodically since 24.11-RELEASE:

192.168.0.254 (which is a TP-link router)

Where is this set on your TP-Link? How is it connected to your pfSense LAN network?

[vf1954]

@SteveITS It is simply plugged in, gets assigned a lan address from pfsense at 192.168.3.3, and then that's it

[vf1954]

@SteveITS sorry I see what you mean.

It is set at 192.168.3.3 in hte LAN settings in tplink

AND

it is set to 192.168.3.3 in pfsense dhcp static.

[SteveITS]

@vf1954 So, what is the .254 you mentioned?

Screencap the change in pfSense when this happens.

If the fields in pfSense aren’t changing I suspect what you’re seeing is another DHCP server. Windows and I’m sure other clients will show the DHCP server used for example “ipconfig /all”

[vf1954]

@SteveITS said in Subnet collapses periodically since 24.11-RELEASE:

Screencap the change in pfSense when this happens.

Not sure what you mean here. Does screencap mean screenshot? Screenshot what?

The address being circulated is 192.168.0.xx but the other DHCP router is the wifi which is turned off.

[SteveITS]

@vf1954 yes, screenshot pfSense with the changed settings, or some evidence.

If you’re not saying anything in pfSense actually changes then it’s not pfSense. Unplug pfSense LAN, restart a client, and see what it’s IP and DHCP server are.