-
Hi,
We have a handful of firewalls which we're interested in moving to pfSense Plus. A prerequisite to this is to upgrade to 2.6.0.
During my testing, I experienced an odd situation where if I restored the config from one of our firewalls everything was fine, but if I restored from another, the firewall would fail to boot, hanging on the second "Configuring firewall..." message.
What's strange about this is that both firewalls are basically identical as far as config goes - one just happens to be carrying more customer config as it's been around for longer.
After a great deal of trial and error, I've managed to narrow the issue down to the NAT reflection.
It seems that the more inbound NAT rules that exist, the longer the firewall takes to configure the firewall, until it hits a point where it fails to boot at all (at least in my testing, leaving it for over 45 minutes). It's my guess that this is why the "lighter config" firewall seems to boot without issue - this is only running 64 inbound NAT rules.
On the firewall in question, we have approx. 250 port forward rules on the WAN interface, some of these have NAT reflection disabled, but where required, NAT reflection is provided by the NAT proxy feature. (Pure NAT mode has never worked for reasons beyond my understanding).
If I change the config file to disable NAT reflection globally, the firewall will boot every time without fail - changing this to use pure NAT exhibits the same behaviour as Proxy NAT, which suggests it's not the proxy NAT processes which are the issue, but the configuration logic.
Currently, I'm in a situation where I can't move up to 2.6.0 - where we have NAT reflection enabled, it's required by the customers and not an option to disable.
Any thoughts appreciated.
Many thanks.
-
-
J jimp moved this topic from Problems Installing or Upgrading pfSense Software on
-
Any thoughts on this one?
-
@ChrisCCC What does “disable NAT reflection globally” mean? There is a setting to enable reflection on ALL rules. Otherwise Reflection can also be enabled for each individual rule in the rule settings.
Can the number of rules be reduced by using split DNS?
I’ve never had to use proxy mode. Per https://docs.netgate.com/pfsense/en/latest/nat/reflection.html there are quantity limitations. IIRC it creates a process per rule…memory issue?
-
@SteveITS thanks for your reply.
When I say "disable globally", I'm referring to to setting "NAT Reflection mode for port forwards" to disabled under System -> Advanced -> Firewall & NAT. The majority of our rules have NAT reflection disabled at the per-rule level where we've been able to disable it. For the rules where NAT reflection is required, the individual rule is set to "Use system default" and so follows the global setting.
Unfortunately, split DNS isn't a suitable solution in this use-case.
I have run into the issue with proxy mode limitations in the past and we reduced the number of rules with NAT reflection active, which resolved this. This is a distinct issue from that and also impacts "Pure NAT" mode.
I've performed some investigation, with my limited comprehension of the pfSense codebase. If I exclude the "if" block on line 2654 of /src/etc/filter.inc, the issue is resolved for both reflection modes.
Further to this, if I leave the if block uncommented, the specific problematic lines are 2656 for "NAT + Proxy" mode and 2667 for "Pure NAT" mode.
Looking at these lines, they appear to be appending lines to a variable that's going to be subsequently pushed into pf. Could this be a kernel issue?
-
Some more notes on what I think I've worked out so far:
The function filter_configure_sync() configures the firewall. This function builds a list of pf commands in a variable named $rules, which is then written into a file named /tmp/rules.debug, this file is then imported using pfctl.
Line 305 calls the function filter_nat_rules_generate(), saving the output into a variable named $natrules, which is subsequently appended to the $rules variable mentioned in the previous paragraph. Commenting line 305, or line 439 resolves this issue. This suggests there's either invalid pf commands being generated, or something has changed in pf which is causing this issue when attempting to load previously working rules.
-
@ChrisCCC I'm not the right person to ask about the code. I can read/write PHP but am not familiar with pfSense's code base.
I'd wonder why Pure NAT doesn't work. We've always used that but only ever need a few reflected rules. Can you enable Pure NAT for some/any of the rules? Perhaps the clause, "...it must be possible to accurately determine the interface and gateway IP address used for communication with the target at the time the rules are loaded."
It's not PHP running out of memory is it? The limit is 512 M on 64bit/CE as I recall. That seems unlikely but something that takes an increasingly long time like that is often a loop that is growing exponentially. It may not get noticed with 5 or 10 but is apparent with 100 or 200.
A sledgehammer approach would be to try it on 2.7 dev which is a new OS and PHP 8.
-
I noticed that after upgrading from 22.2 to 23.05 the 1:1 NAT seems to only accept traffic to the first external IP address in the list (see screenshot). This worked fine prior to the upgrade.
-
@SteveITS Thanks Steve.
Unfortunately Pure NAT has never worked for us. Given the relatively small amount of traffic that hits these rules, proxy mode has always served these edge cases well enough.
Re. your comment on PHP resource. I'm fairly certain this isn't the issue. I think I've followed the PHP as far as it goes and it appears to be consistent with 2.5.1, up to the point the generated pf rules are imported using pfctl. This makes me think the issue is more of a kernel issue, though it's certainly possible I've missed something.
I think I'll try your suggestion of testing 2.7. Definitely a sledgehammer approach, but if it works, once 2.7 is released, we can safely just skip over 2.6.
-
OK, so it seems to be good news. Whatever is causing this bootup issue in 2.6 doesn't appear to be an issue in 2.7. There are other buggy behaviours (CARP, specifically seems to have some issues), but I would expect this as it's still in development.
My only concern now is when 2.7 is actually likely to release. It's been coming for a while now.....
