[Resolved] pfSense hangs when WAN is unstable or lost



  • Sometimes when WAN connection is lost or becomes unstable, pfSense will become unresponsive.

    • Can't login to web interface

    • Can't ssh

    • Must attach to serial connections and reboot.

    It's the same each time. The WAN starts acting up then unbound process takes over the cpu and I usually can't log in and must reboot via serial port. After that it all goes back to normal.

    Occasionally I am able to get to the gui or ssh but pfsense is very slow. Last time it happened I was able to get some screen shots.

    Even after the WAN side start working again pfsense stays in this state until I can reboot.

    I can't figure out why this is happing, tried a clean install and restore but eventually the same thing happens.

    Dashboard:
    0_1547029417437_Screen Shot 2019-01-09 at 10.21.11 AM.png

    WAN becomes unstable:
    0_1547028627605_Screen Shot 2018-12-20 at 8.50.11 AM.png

    unbound high cpu
    0_1547028703739_Screen Shot 2018-12-20 at 8.51.12 AM.png


  • Netgate Administrator

    Do you see anything logged after rebooting?

    A crash report?

    Steve



  • pfSense doesn't crash, just becomes very slow or unresponsive. If I am able to get the gui or ssh to respond and login, it's always unbound which is the cpu hog. Honestly, I'm usually so busy that after rebooting I just go back to work. I've been putting off troubleshooting this for months now. Next time this happens I'll grab the system log and post it.

    Can you think of any reason why unbound would be chewing cpu cycles like this after losing WAN access?


  • LAYER 8 Rebel Alliance

    You should check with your ISP why your WAN is unstable and/or goes down again and again in the first place. 😌
    Some WAN without 99.998% Uptime would make me crazy. 😬

    -Rico



  • @asv345h said in pfSense hangs when WAN is unstable or lost:

    Can you think of any reason why unbound would be chewing cpu cycles like this after losing WAN access?

    Unbound get's restarted when the state of the WAN interface changes. So if your WAN interface is flapping it will cause constant restarts, and if you use DNSBL with a lot of lists the unbound restarts cost additional CPU cycles.



  • @grimson said in pfSense hangs when WAN is unstable or lost:

    Unbound get's restarted when the state of the WAN interface changes. So if your WAN interface is flapping it will cause constant restarts

    Did not know that. Does gateway monitoring (dpinger) have any impact on the way unbound responds to WAN instability?

    and if you use DNSBL with a lot of lists the unbound restarts cost additional CPU cycles.

    I do use DNSBL and have 15 lists plus easylist - too many?



  • I also noticed a while back that my web gui felt very slow on the LAN when having issues with my modem on the WAN. I think in my case I had very bad latency on the WAN due to the modem. Restarting my modem solved my latency issues and the slow web GUI issue. I was a little puzzled about why my web gui was slow on the LAN, when my real issue was on the WAN. I never gave it much thought because the service provided by the ISP is usually solid and therefore not an issue.



  • @stephenw10

    system.log

    Same thing happened again this morning with the exception that unbound did not look to be a cpu hog. GUI was almost totally unresponsive but was able to ssh into system just fine.

    • Restarting webConfigurator had no effect.
    • This time I rebooted my Virgin modem and waited for it to come back online but pfSense remained in the same 'hung' state.
    • I eventually rebooted pfSense and, as usual, all was good again.

    I don't spend much time looking at the log so not sure what is normal. Can you see anything? Rebooted at 8:45:41.

    fwiw
    0_1547113492259_Screen Shot 2019-01-10 at 8.36.04 AM.png
    0_1547113505412_Screen Shot 2019-01-10 at 8.37.44 AM.png



  • Found many of these errors in your log. I'm not sure if any of this applies to you.
    https://forum.netgate.com/topic/110858/unbound-error-bind-address-already-in-use-fatal-error-could-not-open-ports



  • @Raffi_

    From that thread

    Hint: Do NOT ever add Unbound to Service Watchdog. Especially not if using pfBlockerNG.

    I did have unbound monitored by Service Watchdog and just removed it. I also use pfBlockerNG.



  • @asv345h Hope it helps. Let us know how it goes.


  • Netgate Administrator

    Are you using block rule auto generated by pfBlocker as well as DNSBL?

    pfctl at 98% implies something very odd going on. I would expect the system log to be showing a load of entries there.

    If you can I would disable pfBlocker as a test. I imagine the problems will all go away...

    Steve



  • @stephenw10
    Will disable pfBlocker and see. Just happened again as I was reading you response!

    I am using DNSBL, but not using auto-gen block rules. I just use the aliases directly like this:

    0_1547143832628_Screen Shot 2019-01-10 at 6.10.10 PM.png



  • Question : are you pushing DNS (unbound requests) over the WAN or a VPN tunnel ?



  • @gertjan
    WAN using DNS over TLS to 1.1.1.1



  • I advise you to use a remote syslogger : Status => System Logs => Settings and set a "Remote log servers".
    This will help you to see what happens real time, without the need to login by ssh or GUI.

    Checkout what happens with unbound when WAN goes up and down - knowing that you use pfBlocker or DNSBL. You will be surprised.

    Be aware of the fact that when an interface goes down, like a WAN, all related connections like VPN's and attached services (unbound, pfBlocker DNSBL, etc) will restart. This represents a boat load of code. It will take seconds, even on a fast system, to stabilize.

    I propose that you use a WAN - no VPN, no pfBlocker or DNSBL and see what happens.
    Then add one functionality after another and analyze the timing.



  • So my WAN just suffered the same kind of transient disruptions I've been seeing. However, this time, no pfSense meltdown. The GUI was responsive and both unbound and pfctl processes are both well behaved. After the WAN went back to normal I didn't have to reboot pfSense to get it back.

    The difference seems to be that, following the advice on this thread, I disabled pfBlockerNG (both ip and DNSBL lists) and stopped monitoring unbound with Service Watchdog yesterday. I'll enable each one it turn and see what happens next time.

    @Gertjan
    Yesterday I setup a Splunk server and am sending all pfSense logs to it. Any advice on remote logging for pfSense? Splunk is working just fine but the logs are not as well formatted so harder to scan.



  • both ssh and the gui will perform a dns query to resolve the ip address the connection comes from.

    the timeout of the dns query is long, hence the sluggishness

    i kinda recollect that using the AddressFamily=inet flag allows connecting faster through ssh. this seems to works even when passing the flag to the client, though i'm unsure why.

    setting a reasonable timeout such as 5 seconds in resolv.conf should help. i kinda remember the default is 30



  • @skullnobrains said in pfSense hangs when WAN is unstable or lost:

    both ssh and the gui will perform a dns query to resolve the ip address the connection comes from.

    Dono if OpenSSL or PHP (GUI) are doing so, but true.
    On the other hand : a device from LAN connecting to pfSense is probably DHCP registered. So, normally, unbound will know right away the IP of it's own interface - and the IP of the connecting device.
    For example, the /etc/host file will contain this info.

    I guess, when SSL access is slow (from LAN), it is because the entire system hovers around 100 % occupation.



  • I use DHCP static mappings for all my devices. My management workstation, the one I use to connect to pfSense, as well as all the others has an entry in /etc/hosts


  • Netgate Administrator

    Do you have multiple gateways?
    If not you can eliminate a lot of unnecessary churning by disabling the gateway monitoring action (edit the gateway).
    With only one gateway the system will just keep trying it and erroring out but won't run all the gateway up/down scripts the restart a load of things.

    Steve



  • @stephenw10

    Do you have multiple gateways?

    I do, 1 WAN and 4 VPNs and I now think that was part of the problem. In the end I decided to just do a fresh install and a complete do-over.



  • I ended up resolving (not solving) the issue by doing a fresh install and then reconfiguring the setup by hand. I had tried reloading the config from a backup first but the issue persisted. I now have the same hardware and config setup as before but the issue seems to be gone. I can power-cycle my cable modem, no problem. Before I would have had to reboot pfSense to get it out of whatever strange state it was in. Also, the transient WAN instabilities, which would have cause a meltdown and reboot, are handled just fine by pfSense as expected.

    And, for your amusement. I recently found this homemade cable splitter hidden away in the ceiling which would explain my WAN issues.
    0_1547716018355_IMG_5817.jpg



  • I do think this is some combination between WAN instability, VPNs and pfBlockerNG as I have 5 VPNs, 1 WAN, pfBlockerNG and my ISP has been having issues tonight which has caused Unbound to stop running several times and even DHCPd once.

    I have gateway monitoring action disabled on the VPNs but I have seen things flap around a bit in the past if an OpenVPN connection becomes unstable, which sometimes happens when my friends Cable is having a bad day (one of my VPN endpoints).

    My guess is when an OpenVPN connection doesn't stay up long enough for all services to restart, they get restarted again while in an unstable state causing a failure to restart. I believe pfBlockerNG in particular slows down the firewall restart quite significantly, possibly causing a conflict when things are bouncing around.

    Is this perhaps something that can be looked into by Netgate (maybe even the pfBlockerNG maintainer?) as it really shouldn't be possible for pfSense to end up in a state where manual intervention is required just because the WAN or VPN went wobbly for bit. Shouldn't there be some sort of limit in how quickly the scripts can be run so that we can be sure everything is stable before running them again?

    Its frustrating enough that DHCP and DNS hangs when the gateway is bouncing, let alone them failing to restart at all.

    It also tends to cause:

    pf_busy
    
        PF was wedged/busy and has been reset. @ 2019-01-21 00:34:57
    
    Filter Reload
    
        There were error(s) loading the rules: pfctl: DIOCADDALTQ: Device busy - The line in question reads [0]: @ 2019-01-21 00:34:58
    


  • Make sure that Watchdog is not monitoring unbound. I mistakenly had that activated and it definitely compounded the problem. After I reinstalled and reconfiguring all settings by hand, the issue went away. It's been a few days now and everything is rock solid.



  • I have the same issue at home. Im running on a config that started out early in the 2.0 chain.

    Anytime my cable provider has an issue which is rare the GUI becomes unresponsive. This has happened for years.



  • @asv345h said in [Resolved] pfSense hangs when WAN is unstable or lost:

    Make sure that Watchdog is not monitoring unbound. I mistakenly had that activated and it definitely compounded the problem. After I reinstalled and reconfiguring all settings by hand, the issue went away. It's been a few days now and everything is rock solid.

    I don't use the watchdog at all, although I might add DHCPd if that doesn't conflict?



  • @asv345h said in [Resolved] pfSense hangs when WAN is unstable or lost:

    And, for your amusement. I recently found this homemade cable splitter hidden away in the ceiling which would explain my WAN issues.

    WOW! There are some real "geniuses" in this world!!!

    I'm surprised the cable company didn't catch that, as they do a lot of work to ensure the integrity of their system, as they are required by law, to prevent interference to other services.