Normal for pfsense LAN webgui to be temporarily inaccessible and pf interfaces and pf LAN pings do not reply if "Apply Changes" clicked or if dpinger Alarm latency events logged?
-
Hi,
FYI, I first searched and did find the following 2year old posts with similar issues and bugs and I will be testing some of the suggestions from these old posts in my environment to troubleshoot
https://forum.netgate.com/topic/153663/performance-impact-of-clicking-apply-changes/9
https://redmine.pfsense.org/issues/10414I've been running pfsense multi wan with several vlans and recently got new hardware and am running into issues.
Performed a clean install of 2.6.0-RELEASE onto an intel ssd dell in a R*30 server with onboard broadcom nics and enabled pfsense to use ram disks. PFsense has installed the following packages: ntopng, pfBlockerNG-devel, Status_Traffic_Totals, Tailscale. Also running several openvpn tunnels.
Current harware specs are
Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
16 CPUs: 1 package(s) x 8 core(s) x 2 hardware threads
Kernel PTI Disabled
MDS Mitigation Inactive
MBUF Usage 2% (20270/1000000)
Load average 0.67, 0.43, 0.43
Memory usage 1% of 261855 MiB
SWAP usage 0% of 1024 MiBPowerD is checked Enable PowerD. All set to Hiadaptive
Firewall Maximum Table Entries 400000 (left blank for the default. On this system the default size is: 400000)
The previous older pfsense on the older different machine had intel nics and was running an older version of pfsense and this same exact backed up XML configuration never experienced these issues below on any prevous pfsense versions or on any other hardware. I exported and edited the pfsense config, renamed nic names to match the new bge broadcom nics, and imported the nic updated config into the new dell 2.6.0-RELEASE machine. I noticed immediately on bootup that it was taking a very very long time for pfsense 2.6.0-RELEASE to initialize and respond to pings and then begin to allow any traffic to pass.
After further testing with the new dell R*30 server with onboard broadcom nics and 2.6.0-RELEASE I am experiencing many issues which I never experienced before on any previous hardware or pfsense versions.
On the LAN I connected another dfferent computer directly to a dedicated lan port on the pfsense gateway and it is constantly pinging pfsense lan. Most of the time there is no packet loss with a few scattered higher time=~15ms which I don't remember seeing times more than 1ms before.
The real issues are I can trigger on demand the pfsense web gui to become temporarily nonresponsive and for all pfsense static ip interfaces to temporarily stop respondng to pings by simply saving and clicking the 'Apply Changes' button. The pfsense webgui and pfsense pings will stop for many dozen seconds and then later some pings will get some responses, then flop to no responses, and then cluster receiving a few more responses, and then no responses, etc for many more seconds. All traffic through pfsense stops during this time and vlan to vlan traffic gets hit as well as vlan to wan. Most applications and sessions in all the vlans lose connections.Also, dpinger Alarm latency events will trigger the pfsense web gui to be nonresponsive and for the lan gateway to stop respondng to pings same as described above. I went to change the Routing Gateways advanced setting to increase the Packet Loss thresholds to 20 & 30 and saving and Apply Changes killed all the network connections again.
I also noticed that after any event triggers this behavior there is usually a few more randomly occuring clusters of no reply pings every few minutes. After a while and after no interacting with pfsense webgui the machine returns to experiencing no packet loss and throughput and speed tests all appear great.
I will try to setup another 2.6.0-RELEASE machine to test and try to troubleshoot this behavior.
I do not have the following enabled/checked: "State Killing on Gateway Failure Flush all states when a gateway goes down"
I do not have the following enabled/checked: "Hardware Checksum Offloading Disable hardware checksum offload"
I do have the following enabled/checked: "Hardware TCP Segmentation Offloading Disable hardware TCP segmentation offload"
I do have the following enabled/checked: "Hardware Large Receive Offloading Disable hardware large receive offload"
I do have the following enabled/checked: "hn ALTQ support Enable the ALTQ support for hn NICs."
I do not have the following enabled/checked: "Reset All States Reset all states if WAN IP Address changes"I have been reviewing the system logs and am unable to find any clues how to resolve this issue.
Here are some General System logs that appear around the time of the issues:
Oct 18 22:22:05 rc.gateway_alarm 88274 >>> Gateway alarm: WANGW (Addr:ip.ip.ip.ip Alarm:0 RTT:283.214ms RTTsd:1584.898ms Loss:0%)
Oct 18 22:22:05 check_reload_status 485 updating dyndns WANGW
Oct 18 22:22:05 check_reload_status 485 Restarting IPsec tunnels
Oct 18 22:22:05 check_reload_status 485 Restarting OpenVPN tunnels/interfaces
Oct 18 22:22:05 check_reload_status 485 Reloading filter
Oct 18 22:22:06 php-fpm 9329 /rc.openvpn: MONITOR: WANGW is available now, adding to routing group failoverWAN0_1
..
php-fpm 65556 /rc.filter_configure_sync: An error occurred while trying to find the interface got ip.ip.ip.ip . The rule has not been added.
Oct 18 21:43:59 php-fpm 65556 /rc.filter_configure_sync: An error occurred while trying to find the interface got ip.ip.ip.ip . The rule has not been added.
Oct 18 21:43:59 php-fpm 65556 /rc.filter_configure_sync: An error occurred while trying to find the interface got ip.ip.ip.ip . The rule has not been added.
Oct 18 21:43:59 php-fpm 65556 /rc.filter_configure_sync: An error occurred while trying to find the interface got ip.ip.ip.ip . The rule has not been added.
..
Oct 18 07:27:31 dpinger 91711 WANGW ip.ip.ip.ip: Alarm latency 16153us stddev 4735us loss 21%
Oct 18 07:28:43 dpinger 91711 WANGW ip.ip.ip.ip: Clear latency 142429us stddev 1362337us loss 5%When time permits I will try to lower the "Firewall Maximum Table Entries" and/or disable "Block bogon networks".
Anyone have any other ideas?
Thanks
-
You may well be hitting this: https://redmine.pfsense.org/issues/12827
A patch to mitigate that is in the recommended patches list in the System Patches package.
Steve
-
Yes, you are correct. That was impacting the system.
I first setup a test system in lab with same 2.6 config and performed a webgui update 2.6 to 2.7.0-DEVELOPMENT. On 2.7.0-DEVELOPMENT the system does not exhibit the issues described in first post.
On production machine I followed your instructions and applied patch "Disable pf counter data preservation to temporarily work around latency when reloading large rulesets (Redmine #12827)".
Issue appears resolved.
Thank you!
