pfSense freezes after 19-23 hours uptime
-
Hi
I just bought a new box to give pfSense a try. Installation went smooth and configuration was fast (thanks for the good documentation!). Then the problems started.
SYMPTOMS
The box froze after having had an uptime of 19-23h. Problems were:- no interface could be reached anymore
- keyboard inputs were not responded to
- no error visible on the screen
A cold reset helped and anything came up again. Once in a while I got a kernel panic instead of a freeze (crash dumps still available if desired) with the advantage, that the box then rebooted itself.
TESTS
I tried different things:- troubleshooting the Intel I211 NIC's as described here
- tried both 2.4.4-p3 and 2.4.5
- disabled all additional packages. While pfBlockerNG is not the problem it could provoke the error sooner, when played around and often reloaded/updated the IP's
Error stayed the same.
SOLUTION
Then I stumbled upon this post. Problems are similar altough my connections did not automagically came back. But then I did the following and for now it seems to have resolved the problem:- "System - Routing - Edit Gateway" and activated both "Disable Gateway Monitoring" and "Disable Gateway Monitoring Action" (I do only have one gateway)
- "Interface - WAN" set "IPv6 Configuration Type = None" (my ISP does not provide IPv6 functionality)
QUESTION
My question is: why that? Can that be a driver problem with the Intel NIC (I have two 2-port I211 cards). I can't see the slightest error in the logs or syslog server before the system freezes. Was just wondering if somebody maybe had a good idea or could have a look at it. -
Pfsense checks a lot of things from the internet, so if there is no internet access then the GUI becomes very slow.
Many colleagues he/she when experiences this even - they think the system has crashed, but this is not always the case, in fact.
In this case, it may take up to 5 - 10 minutes for the GUI to be available.
This is not expected by many and they do a cold boot immediately, which is not a very good idea and the many - many cold start requires minimum at least a ZFS file system installation.Intel i211-based NICs are relatively problem-free in the pfsense system, even with basic settings.
On the WAN side, look for the problem, why your connection may be lost or why the WAN gateway parameters are deteriorating.
I suggest that, also turn off EEE and flow control on this interface.The gateway monitor should be well configured for the external gateway IP, if it responds to the ping or for external trusted IP e.g. DNS server 8.8.8.8 / 9.9.9.9/ 1.1.1.1 or similar.
I hope this can help
-
Thanks for the fast response!
In this case, it may take up to 5 - 10 minutes for the GUI to be available.
I have a broadband monitor from both WAN and LAN to the gateway. This shows downtimes for up to 40 min until I cold started the box (was never more patient than that during testing). So if it should come back then it takes waaaay to long.
Intel i211-based NICs are relatively problem-free in the pfsense system, even with basic settings.
Glad to hear :)
On the WAN side, look for the problem, why your connection may be lost or why the WAN gateway parameters are deteriorating.
I suggest that, also turn off EEE and flow control on this interface.I've had a different firewall before and did not experience downtimes. So it should not be a problem from the provider side? Will try disabling flow control if I ran into a problem again (and then get this post updated). But that would be for all ports and not only the troublesome, right? And what do you mean by EEE though?
The gateway monitor should be well configured for the external gateway IP, if it responds to the ping or for external trusted IP e.g. DNS server 8.8.8.8 / 9.9.9.9/ 1.1.1.1 or similar.
Sure, I have it set to loadbalanced servers from my work - so I do know if we encounter troubles but it for sure is better to set it to a service with even more reliable uptime.
-
-try rebooting via SSH rather than cold boot
-Many user-poorly configured resource-intensive processes, such as Suricata / Snort / pfblockerNG, can also cause extreme WAN-side parameters, which also indicate a crash-like state.
-PfSense is a bit of a different philosophy, but if you try it, you'll love it :-).
-EEE = energy efficiency ethernet, (it doesn’t make much sense on a busy NGFW)
use these:These ae tunables to improve network performance on Intel igb driver NICs
Flow Control (FC) 0=Disabled 1=Rx Pause 2=Tx Pause 3=Full FC
This tunable must be set according to your configuration. VERY IMPORTANT!
Set FC to 0 (<x>) on all interfaces
hw.igb.<x>.fc=0 #Also put this in System Tunables hw.igb.<x>.fc: value=0
and
Disable Energy Efficiency - set for each igb port in your system
This setting can cause Link flap errors if not disabled
Set for every igb interface in the system as per these examples
dev.igb.0.eee_disabled: value=1
dev.igb.1.eee_disabled: value=1
dev.igb.2.eee_disabled: value=1
dev.igb.3.eee_disabled: value=1These are mostly needed for IPS, but I think they only make your system better.
-
-try rebooting via SSH rather than cold boot
Well no interface is working. No SSH, no ping nothing on WAN/DMZ/LAN...
-Many user-poorly configured resource-intensive processes, such as Suricata / Snort / pfblockerNG, can also cause extreme WAN-side parameters, which also indicate a crash-like state.
Well I uninstalled close to all packages (apart acme, arpwatch and Status_Traffic_Totals). Still the problem persisted. Now I have pfBlockerNG re-installed and with around 400k IP's it should be well within the limits of my memory and parameter settings. But I had it to hourly update and read on another post that they had troubles with it - so I adjusted it to daily (still got the freezes though).
Thanks for the tips about Flow Control and EEE - will do that probably starting next week. I would like to see first though, that the box can run stable without big changes. Then I go ahead with the recommendations :)
-
Are you running bogons block on the interfaces??
-
Flow control and EEE are the default settings, so you can get rid of a lot of trouble in the beginning
I strongly recommend using pfBlockerNG-devel, read BBcan177's recommendations.
I would not postpone its setup (FC and EEE), I will go further ..... I always start with these settings as they form the basis of the system
I'm past the 50th installation :-) -
Are you running bogons block on the interfaces??
On WAN and DMZ yes, but not on LAN.
I would not postpone its setup (FC and EEE), I will go further ..... I always start with these settings as they form the basis of the system
Understood and will do but hey - I am on my first installation and need to get the vibes first ;) But it's noted down for beginning of next week!
I strongly recommend using pfBlockerNG-devel
Okay, will dig into that too.
Thanks for the hints guys!
-
You welcome and Cool_Corona didn't accidentally ask the bogons, ;-)
