Port forward egress packets not being rewritten

[rmc47]

Hi folks,

Struggling a little with some port forwarding.  Previously, we were running pfSense 2.1.5 on a desktop PC, and all was good. We upgraded to 2.3.2.1, then migrated (via backup then restore of config) to a PCEngines board running the same.

Network looks like:

Public interwebs <–> ADSL router (which I don't have control over, so can't put into bridge mode) 192.168.0.1 <--> 192.168.0.2 PFSense 10.105.56.250 <--> 10.105.56.1 Server in question

There's a bunch of other 10.105.x.0/24 subnets on the LAN side of PFSense as well, each of which (including the .56.0/24) is a tagged VLAN on igb0; 192.168.0.2 is untagged on igb2. We're not running multi-WAN. The only gateway is 192.168.0.1

The ADSL router is set up to DMZ everything through to PFSense. This hasn't changed since everything was working pre-upgrade.

Outbound NAT from 10.105.56.1 to the internet works fine.

However, we have a port forwarding rule pointing inwards:

pfctl -sn | grep 1050

rdr on igb2 inet proto tcp from any to 192.168.0.2 port = 1050 -> <int_server>round-robin

And associated filter rule:

pfctl -sr | grep 1050

pass in log quick on igb2 reply-to (igb2 192.168.0.1) inet proto tcp from any to <int_server>port = 1050 flags S/SA keep state label "USER_RULE: NAT Test" dnpipe(2, 1)

If I mirror both igb0 and igb2 on the switch to a machine running Wireshark so I can see packets on both interfaces, I see:

a.b.c.d:56748 –> 192.168.0.2:1050 SYN - from ADSL router's MAC to igb2's MAC
a.b.c.d:56748 --> 10.105.56.1:1050 SYN - from igb0's MAC to the internal server's MAC
10.105.56.1:1050 -> a.b.c.d:56748 SYN ACK - from the server's MAC to igb0's MAC
10.105.56.1:1050 -> a.b.c.d:56748 SYN ACK - from igb2's MAC to the ADSL router's MAC

The fourth packet is where the problem shows up - pf seems to be failing to do the address translation of the outbound packet, but obviously does do the right thing on the inbound packet.

If I look at the state table at this point, it's in the half-established state:

pfctl -ss | grep 1050

igb2 tcp 10.105.56.1:1050 (192.168.0.2:1050) <- a.b.c.d:56748      SYN_SENT:ESTABLISHED
igb0_vlan56 tcp a.b.c.d:56748 -> 10.105.56.1:1050      ESTABLISHED:SYN_SENT

I don't think I'm doing anything crazy, but obviously it isn't wanting to play :-). I've tried rebooting, disabling the rules and re-enabling, disabling and enabling NAT Reflection (though I believe it should not apply here as I'm coming in from a WAN IP).

Any thoughts very much appreciated…

[Edited to add: I've just gone back to the original hardware, post-upgrade to 2.3.2 (not p1), and the behavior is the same, so I don't believe it's hardware-related.]

Thanks,
Rob</int_server></int_server>

[rmc47]

Ahh… a clue: this only seems to apply when I have a pipe selected on the filter rule.

Dropping the pipe off so the rule becomes:

pass in log quick on igb2 reply-to (igb2 192.168.0.1) inet proto tcp from any to <int_server>port = 1050 flags S/SA keep state label "USER_RULE: NAT Test"

…everything works correctly.

The limiter is configured to allow a fixed amount of bandwidth, with no schedule, no mask, and no advanced options set.

Cheers,
Rob</int_server>

[doktornotor]

Limiters with NAT are broken everywhere but on 2.4 devel snapshots.

[rmc47]

Thanks - just found the bug having established the connection! https://redmine.pfsense.org/issues/4326

That explains why it broke after the upgrade from 2.1.5 :-(.