[Resolved] pfSense hangs when WAN is unstable or lost
-
@asv345h said in pfSense hangs when WAN is unstable or lost:
Can you think of any reason why unbound would be chewing cpu cycles like this after losing WAN access?
Unbound get's restarted when the state of the WAN interface changes. So if your WAN interface is flapping it will cause constant restarts, and if you use DNSBL with a lot of lists the unbound restarts cost additional CPU cycles.
-
@grimson said in pfSense hangs when WAN is unstable or lost:
Unbound get's restarted when the state of the WAN interface changes. So if your WAN interface is flapping it will cause constant restarts
Did not know that. Does gateway monitoring (dpinger) have any impact on the way unbound responds to WAN instability?
and if you use DNSBL with a lot of lists the unbound restarts cost additional CPU cycles.
I do use DNSBL and have 15 lists plus easylist - too many?
-
I also noticed a while back that my web gui felt very slow on the LAN when having issues with my modem on the WAN. I think in my case I had very bad latency on the WAN due to the modem. Restarting my modem solved my latency issues and the slow web GUI issue. I was a little puzzled about why my web gui was slow on the LAN, when my real issue was on the WAN. I never gave it much thought because the service provided by the ISP is usually solid and therefore not an issue.
-
Same thing happened again this morning with the exception that unbound did not look to be a cpu hog. GUI was almost totally unresponsive but was able to ssh into system just fine.
- Restarting webConfigurator had no effect.
- This time I rebooted my Virgin modem and waited for it to come back online but pfSense remained in the same 'hung' state.
- I eventually rebooted pfSense and, as usual, all was good again.
I don't spend much time looking at the log so not sure what is normal. Can you see anything? Rebooted at 8:45:41.
fwiw
-
Found many of these errors in your log. I'm not sure if any of this applies to you.
https://forum.netgate.com/topic/110858/unbound-error-bind-address-already-in-use-fatal-error-could-not-open-ports -
From that thread
Hint: Do NOT ever add Unbound to Service Watchdog. Especially not if using pfBlockerNG.
I did have unbound monitored by Service Watchdog and just removed it. I also use pfBlockerNG.
-
@asv345h Hope it helps. Let us know how it goes.
-
Are you using block rule auto generated by pfBlocker as well as DNSBL?
pfctl at 98% implies something very odd going on. I would expect the system log to be showing a load of entries there.
If you can I would disable pfBlocker as a test. I imagine the problems will all go away...
Steve
-
@stephenw10
Will disable pfBlocker and see. Just happened again as I was reading you response!I am using DNSBL, but not using auto-gen block rules. I just use the aliases directly like this:
-
Question : are you pushing DNS (unbound requests) over the WAN or a VPN tunnel ?
-
@gertjan
WAN using DNS over TLS to 1.1.1.1 -
I advise you to use a remote syslogger : Status => System Logs => Settings and set a "Remote log servers".
This will help you to see what happens real time, without the need to login by ssh or GUI.Checkout what happens with unbound when WAN goes up and down - knowing that you use pfBlocker or DNSBL. You will be surprised.
Be aware of the fact that when an interface goes down, like a WAN, all related connections like VPN's and attached services (unbound, pfBlocker DNSBL, etc) will restart. This represents a boat load of code. It will take seconds, even on a fast system, to stabilize.
I propose that you use a WAN - no VPN, no pfBlocker or DNSBL and see what happens.
Then add one functionality after another and analyze the timing. -
So my WAN just suffered the same kind of transient disruptions I've been seeing. However, this time, no pfSense meltdown. The GUI was responsive and both unbound and pfctl processes are both well behaved. After the WAN went back to normal I didn't have to reboot pfSense to get it back.
The difference seems to be that, following the advice on this thread, I disabled pfBlockerNG (both ip and DNSBL lists) and stopped monitoring unbound with Service Watchdog yesterday. I'll enable each one it turn and see what happens next time.
@Gertjan
Yesterday I setup a Splunk server and am sending all pfSense logs to it. Any advice on remote logging for pfSense? Splunk is working just fine but the logs are not as well formatted so harder to scan. -
both ssh and the gui will perform a dns query to resolve the ip address the connection comes from.
the timeout of the dns query is long, hence the sluggishness
i kinda recollect that using the AddressFamily=inet flag allows connecting faster through ssh. this seems to works even when passing the flag to the client, though i'm unsure why.
setting a reasonable timeout such as 5 seconds in resolv.conf should help. i kinda remember the default is 30
-
@skullnobrains said in pfSense hangs when WAN is unstable or lost:
both ssh and the gui will perform a dns query to resolve the ip address the connection comes from.
Dono if OpenSSL or PHP (GUI) are doing so, but true.
On the other hand : a device from LAN connecting to pfSense is probably DHCP registered. So, normally, unbound will know right away the IP of it's own interface - and the IP of the connecting device.
For example, the /etc/host file will contain this info.I guess, when SSL access is slow (from LAN), it is because the entire system hovers around 100 % occupation.
-
I use DHCP static mappings for all my devices. My management workstation, the one I use to connect to pfSense, as well as all the others has an entry in /etc/hosts
-
Do you have multiple gateways?
If not you can eliminate a lot of unnecessary churning by disabling the gateway monitoring action (edit the gateway).
With only one gateway the system will just keep trying it and erroring out but won't run all the gateway up/down scripts the restart a load of things.Steve
-
Do you have multiple gateways?
I do, 1 WAN and 4 VPNs and I now think that was part of the problem. In the end I decided to just do a fresh install and a complete do-over.
-
I ended up resolving (not solving) the issue by doing a fresh install and then reconfiguring the setup by hand. I had tried reloading the config from a backup first but the issue persisted. I now have the same hardware and config setup as before but the issue seems to be gone. I can power-cycle my cable modem, no problem. Before I would have had to reboot pfSense to get it out of whatever strange state it was in. Also, the transient WAN instabilities, which would have cause a meltdown and reboot, are handled just fine by pfSense as expected.
And, for your amusement. I recently found this homemade cable splitter hidden away in the ceiling which would explain my WAN issues.
-
I do think this is some combination between WAN instability, VPNs and pfBlockerNG as I have 5 VPNs, 1 WAN, pfBlockerNG and my ISP has been having issues tonight which has caused Unbound to stop running several times and even DHCPd once.
I have gateway monitoring action disabled on the VPNs but I have seen things flap around a bit in the past if an OpenVPN connection becomes unstable, which sometimes happens when my friends Cable is having a bad day (one of my VPN endpoints).
My guess is when an OpenVPN connection doesn't stay up long enough for all services to restart, they get restarted again while in an unstable state causing a failure to restart. I believe pfBlockerNG in particular slows down the firewall restart quite significantly, possibly causing a conflict when things are bouncing around.
Is this perhaps something that can be looked into by Netgate (maybe even the pfBlockerNG maintainer?) as it really shouldn't be possible for pfSense to end up in a state where manual intervention is required just because the WAN or VPN went wobbly for bit. Shouldn't there be some sort of limit in how quickly the scripts can be run so that we can be sure everything is stable before running them again?
Its frustrating enough that DHCP and DNS hangs when the gateway is bouncing, let alone them failing to restart at all.
It also tends to cause:
pf_busy PF was wedged/busy and has been reset. @ 2019-01-21 00:34:57 Filter Reload There were error(s) loading the rules: pfctl: DIOCADDALTQ: Device busy - The line in question reads [0]: @ 2019-01-21 00:34:58
