Upgrading to pfSense 2.9.0 causes NAT 1:1 to drop IPSec return traffic

[thomas.sec]

Hello,

I’m using pfSense as a firewall in front of my infrastructure, with 1:1 NAT configured to forward traffic to several OPNsense firewalls.

One of my OPNsense firewalls has an IPSec tunnel to a client network. The tunnel works perfectly with pfSense 2.8.x.

After upgrading pfSense to 2.9.0, I started experiencing the following issue:

The IPSec tunnel comes up normally and is shown as UP.
I can establish one Telnet connection over the tunnel successfully.
After that, subsequent packets/connections appear to be dropped (return traffic, paquet are sent in tunnel).
I have to wait approximately 10 minutes before another Telnet connection can be established successfully.
The same behavior then repeats.

Rolling back to pfSense 2.8.x immediately fixes the problem, without changing any configuration on the OPNsense firewall or the IPSec tunnel.

So far, this looks like a regression related to 1:1 NAT and IPSec return traffic in pfSense 2.9.0.

Has anyone experienced a similar issue after upgrading to 2.9.0? Are there any changes in 2.9.0 regarding NAT, state tracking, or IPSec traffic that could explain this behavior? I don't see anything related in changelog.

Thanks in advance for any suggestions or pointers.

[tinfoilmatt]

1:1 NAT ≠ policy-based routing

Preliminarily, that would seem to be the most glaringly obvious issue with your setup.

It's otherwise a little hard to follow your description, particularly with regard to the OPNsense systems. Do they have tunnels to pfSense? Could you maybe post a diagram? Some screenshots of what you're looking at?

[thomas.sec]

Hope this helps. I’ve attached a network diagram to give you a better overview of the setup.
IPSec tunnel is always up (pfsense 2.8.X or 2.9.0), but traffic inside is not working with 2.9.0.

[tinfoilmatt]

How about a screenshot of the pfSense host's NAT ruleset?

Also, just to confirm—IPsec is configured only on the OPNsense host (tunneled to the customer peer), correct?

[thomas.sec]

Yes correct. IPSec is only configured on the OPNsense host.

Here is NAT ruleset:

Details:

Advanced reflection params:

Pass fw rules :

[tinfoilmatt]

And how about the "WAN_C[]TE" interface firewall ruleset? Your floating rule is lacking the 'Quick' action, which behavior I believe did change somewhat from 2.8.x to 2.9.0.

Do you have a "Default gateway IPv4" specified under System > Routing > Gateways? Or is it set to "Automatic"?

Any outbound NAT configured?

[tinfoilmatt]

Easiest thing to try guess-and-check would be ticking the 'Quick' box for your floating rule.

[thomas.sec]

And how about the "WAN_C[]TE" interface firewall ruleset?

Do you have a "Default gateway IPv4" specified under System > Routing > Gateways? Or is it set to "Automatic"?

Any outbound NAT configured?

We’ll try checking the “Quick” box on the firewall rules tonight and keep you posted. Thank you for the time you’ve already spent on this.

[tinfoilmatt]

We’ll try checking the “Quick” box on the firewall rules tonight

On only the one, essentially "pass all" floating rule—just to be clear. But keep in mind that this will essentially 'open up'/eliminate the pfSense firewall completely in both directions.

If any of this ends up being relevant, it'd be better to put a more specific 'pass' rule on the "WAN_C[]TE" interface.

[tinfoilmatt]

it'd be better to put a more specific 'pass' rule on the "WAN_C[]TE" interface.

I see that that's probably the first 'pass' rule utilizing the "IP_LAN_NAT_1_1" alias.

[thomas.sec]

Adding the “quick” option to the floating rule did not fix the issue.
We reverted back to version 2.8.1 again and it works.

We will try to get more logs or diagnostics because we don't understand the issue yet. We will keep you posted. Thanks

[luckman212]

@thomas.sec suggest you take a snapshot of /tmp/rules.debug from the working 2.8.1 system. When you decide to resume troubleshooting, it will probably be useful to diff compare that with the ruleset that gets generated by 2.9.0 to pinpoint the reason for this failure.

[tinfoilmatt]

You could try to diff either the configs and/or the ruleset (at /tmp/rules.debug) between 2.8.1 and 2.9.0.

[thomas.sec]

We’re running a CARP HA pair: one node on pfSense 2.8, the other on 2.9. I dumped rules.debug on both and compared them.

Aside from the usual per-node IPs, the rule sets match. The smaller state table on 2.9 is expected, that instance has 4 GB RAM, the 2.8 one has 16 GB.

The only extra bits on 2.9 are the new NAT64 defaults (RFC 6052):

table <_nat64reserved_> { 64:ff9b::… reserved v4 embeddings … }

block in  log quick inet6 from any to <_nat64reserved_>
block out log quick inet6 from any to <_nat64reserved_>

Those only hit IPv6 NAT64 toward non-global IPv4. They don’t touch IKE/NAT-T/ESP over IPv4, or the IPv4 traffic inside the tunnel.

(We also tested quick on a floating pass any/any, that’s on us, not 2.9. It wouldn’t block IPsec anyway.)

So from the ruleset side, nothing here looks like it would break IPsec (?).

[tinfoilmatt]

What's OPNsense's 'WAN' interface address?

[thomas.sec]

@tinfoilmatt 172.16.32.203 (CARP VIP)
WAN interfaces IP are: 172.16.32.11, 172.16.32.12

(opnsense fw are in HA too)

[tinfoilmatt]

Interface "INTERCO_WAN" is in the same subnet? What's that subnet, 172.16.32.xxx/xx?

[thomas.sec]

Subnet is 172.16.32.0/24

Pfsense INTERCO_WAN has 172.16.32.251, .252 and .254 (carp)

All in the same subnet.

[tinfoilmatt]

How are you verifying the dropped return traffic? I assume you would've said explicitly if you were seeing filtered traffic (i.e., appearing in the pfSense firewall log).

Have you performed any packet capturing on the OPNsense 'WAN' interface and both pfSense system's interfaces, "INTERCO_WAN" and "WAN_C[]TE" while testing? Would also be interesting to see what's making it and being replied to on the peer side, too.

[thomas.sec]

So far, we have only performed the following test:

From a host in Internal Subnet B, we send a TCP packet (with Test-NetConnection... Windows host) on port 443 or 389 to a host in the Customer Internal Subnet

• pf 2.9.0: The request succeeds once every ~10 minutes; otherwise, it is dropped (timeout).

• pf 2.8.1: The request succeeds consistently.

On the OPNsense side, we can see the packets being sent through the IPSec tunnel interface every time.

Unfortunately, we have not yet had a chance to trace the packets on pfSense.
We will perform further testing over the next few days, particularly on the pfSense side.

Thank you for your patience.