Source address not NATed during OpenVPN startup?

bPsdTZpW · Feb 22, 2022, 9:59 PM

pfSense sometimes fails to NAT the LAN source address for packets sent to the WAN while an OpenVPN tunnel is initializing. I noticed this bug when my ISP's modem logged errors like:

num       date/time                       source IP       dest IP                 proto   reason
---------------------------------------------------------------------------------------------------------
146	2022-02-22T13:32:09.389745	192.168.x.y	<e.g., Google IP>	TCP	IP Source Address

while I was restarting OpenVPN. A packet capture on the WAN interface during this time shows un-NATed packets exiting WAN:

13:32:08.340316 <pfsense MAC> > <ISP modem MAC>, ethertype IPv4 (0x0800), length 139: (tos 0x0, ttl 127, id 0, offset 0, flags [DF], proto TCP (6), length 125)
    192.168.x.y.60004 > <e.g., Google IP>.443: Flags [P.], cksum 0x9f97 (correct), seq 3564024189:3564024274, ack 2514772943, win 1026, length 85
13:32:08.647718 <pfsense MAC> > <ISP modem MAC>, ethertype IPv4 (0x0800), length 139: (tos 0x0, ttl 127, id 0, offset 0, flags [DF], proto TCP (6), length 125)
    192.168.x.y.60004 > <e.g., Google IP>.443: Flags [P.], cksum 0x9f97 (correct), seq 0:85, ack 1, win 1026, length 85

(Note, the devices' clocks are not precisely synchronized)

I have put various "reject" floating rules on outbound WAN to prevent these packets from exiting [1], but they are ineffective: the packets still exit, and the rules show no activity.

The bug can be reproduced nearly every time by restarting OpenVPN, then refreshing a web page repeatedly until after OpenVPN is up again.

I use a gateway group as the default gateway, with the VPN interface at high priority and WAN_IGB0 at lower priority.

[1] e.g. action:block, quick, interface:WAN_IGB0, direction:out, family:IPV4+IPV6, protocol:any, source:RFC1918, destination:any, extra options:log, no advanced options.

stephenw10 · Feb 22, 2022, 10:19 PM

Do you see states opened by that traffic or is it being passed by some existing state?
If so what interface is the state shown on?

Steve

bPsdTZpW · Feb 22, 2022, 10:31 PM

@stephenw10 I see many states of this form:

IGB1_LAN_ADMIN 	tcp 	192.168.x.y:60304 -> <website IP>:443 	CLOSED:SYN_SENT 	1 / 1 	52 B / 80 B

IGB1_LAN_ADMIN is one of my LAN interfaces. It has the 192.168.x.y address.

stephenw10 · Feb 22, 2022, 10:48 PM

No actual states on WAN/igb0 for that traffic?

And 192.168.x.y is actually the LAN interface address? And appears as the source address for packets leaving WAN?

If there are no states opened on WAN then a block rule on WAN can't prevent that traffic.

bPsdTZpW · Feb 22, 2022, 10:56 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

No actual states on WAN/igb0 for that traffic?

Right.

And 192.168.x.y is actually the LAN interface address? And appears as the source address for packets leaving WAN?

Yep and yep. And the ISP's modem is getting these packets with 192.168.x.y source addresses.

If there are no states opened on WAN then a block rule on WAN can't prevent that traffic.

So how is this traffic going out WAN? Is NAT disabled during filter reload, but WAN still operating, so that un-NATed packets get sent out WAN, but states get created on LAN?!

stephenw10 · Feb 22, 2022, 10:57 PM

@bpsdtzpw said in Source address not NATed during OpenVPN startup?:

192.168.x.y

Just to be clear that is a single IP throughput and is the LAN interface IP?

What's probably happening here is the traffic is being passed on the LAN but policy routed out of the WAN. Which is odd but conceivable!

Do you have a LAN side gateway defined?

Are you using policy based routing anywhere?

Steve

bPsdTZpW · Feb 22, 2022, 11:09 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

@bpsdtzpw said in Source address not NATed during OpenVPN startup?:

192.168.x.y

Just to be clear that is a single IP throughput and is the LAN interface IP?

What's probably happening here is the traffic is being passed on the LAN but policy routed out of the WAN. Which is odd but conceivable!

Do you have a LAN side gateway defined?

Are you using policy based routing anywhere?

Steve

I have no LAN side gateways, and no policy routing. The only gateways are WAN_IGB0 and VPNIFC (VPN). I route packets to the VPN using a gateway group. And this thing is, the packets get routed incorrectly only during VPN initialization. I suspect something incorrect during filter reload.

FWIW, I have a static route to the ISP modem via WAN_IGB0's gateway (which lets me admin the modem through WAN just as if it were an external website). The network for the static route is the modem's (RFC1918) IP address/32.

stephenw10 · Feb 22, 2022, 11:11 PM

Ok so you have a policy based routing rule on the LAN for traffic from the LAN via the gateway group?

Is the system default route via the WAN?

bPsdTZpW · Feb 22, 2022, 11:14 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

Ok so you have a policy based routing rule on the LAN for traffic from the LAN via the gateway group?

I have a gateway group for traffic to the WAN, with the VPN gateway at high priority, and WAN_IGB0's gateway at low priority.

Is the system default route via the WAN?

The system default route (via system/routing/gateways) is the gateway group.

bPsdTZpW · Feb 22, 2022, 11:47 PM

Clue: checking system/advanced/networking/Reset All States results in fewer (but not necessarily no) bad packets exiting WAN during VPN startup.

stephenw10 · Feb 23, 2022, 1:39 AM

Ah, OK I so not rule on LAN with the group defined. It's only used in the system default gateway to route traffic?

Hmm, I could see that being an issue. If you policy route the traffic on LAN all the outgoing traffic will get tagged route-to for the current gateway. Can you test that?

Traffic leaving with the LAN interface IP as source is odd though. That should only ever be possible if it's NAT'd to it. It almost seems like it's falling back to the LAN IP as the next interface when the assigned OpenVPN goes down...

bPsdTZpW · Feb 23, 2022, 1:50 AM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

Ah, OK I so not rule on LAN with the group defined. It's only used in the system default gateway to route traffic?

Right. Only as the system default gateway.

Hmm, I could see that being an issue. If you policy route the traffic on LAN all the outgoing traffic will get tagged route-to for the current gateway. Can you test that?

OK.

Traffic leaving with the LAN interface IP as source is odd though. That should only ever be possible if it's NAT'd to it. It almost seems like it's falling back to the LAN IP as the next interface when the assigned OpenVPN goes down...

The even odder thing is that the bad packets get passed when the VPN is coming up, at about the time it issues the message

Initialization Sequence Completed

It does seem like it's falling back to the LAN IP, though.

bPsdTZpW · Feb 23, 2022, 1:57 AM

@bpsdtzpw So policy-routing from LAN to the gateway group gives the same behavior.

bPsdTZpW · Feb 23, 2022, 2:04 AM

The bad packets get passed around the time the following appears in the system log:

Feb 22 17:55:49 	php-fpm 	412 	/rc.start_packages: Restarting/Starting all packages.
Feb 22 17:55:48 	check_reload_status 	441 	Starting packages
Feb 22 17:55:48 	php-fpm 	98676 	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - <former IP> -> <former IP> - Restarting packages.
Feb 22 17:55:46 	php-fpm 	98676 	/rc.newwanip: Creating rrd update script
Feb 22 17:55:43 	php-fpm 	98676 	/rc.newwanip: IP Address has changed, killing states on former IP Address <former IP>.
Feb 22 17:55:43 	php-fpm 	98676 	/rc.newwanip: Default gateway setting Interface <vpn> Gateway as default.
Feb 22 17:55:43 	php-fpm 	98676 	/rc.newwanip: Gateway, switch to: <vpn>
Feb 22 17:55:43 	php-fpm 	98676 	<VPN gateway>|<VPN virtual IP>|<vpn>|9.916ms|0.516ms|0.0%|online|none

bPsdTZpW · Feb 23, 2022, 2:20 AM

Another clue: if I change the NAT rule from LAN to WAN_IGB0 to use the WAN's IP address directly (e.g. w.x.y.z) instead of "interface address", the behavior is the same. This seems to indicate that the NAT rules aren't being used when the bad packets are passed.

stephenw10 · Feb 23, 2022, 3:44 PM

Mmm, if if was NATing to the LAN IP or using the OBN rules at all you would see it in the created states.

Since it's being passed by a state opened on LAN you could try adding a block rule on LAN to prevent it as a workaround.

Steve

bPsdTZpW · Feb 23, 2022, 5:56 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

Mmm, if if was NATing to the LAN IP or using the OBN rules at all you would see it in the created states.

Since it's being passed by a state opened on LAN you could try adding a block rule on LAN to prevent it as a workaround.

Steve

What block rule could I use? From the point of view of the LAN interface, the packets are perfectly OK (src:LAN device, dest:internet).

stephenw10 · Feb 23, 2022, 6:12 PM

It's source: LANaddress destination: Internet though and it's outbound which should never happen.

You want to make it as specific as possible so I'd use a floating, quick, outbound rule, source: LANaddress destination: some-test-address. Make sure that does something useful and does block expected traffic before changing the destination to some thing wider.

Steve

bPsdTZpW · Feb 23, 2022, 6:29 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

It's source: LANaddress destination: Internet though and it's outbound which should never happen.

You want to make it as specific as possible so I'd use a floating, quick, outbound rule, source: LANaddress destination: some-test-address. Make sure that does something useful and does block expected traffic before changing the destination to some thing wider.

Steve

I think I see. I already have a rule like this, and it doesn't work. From the original post:

I have put various "reject" floating rules on outbound WAN [by which I meant floating, WAN, outbound] to prevent these packets from exiting [1]...[1] e.g. action:block, quick, interface:WAN_IGB0, direction:out, family:IPV4+IPV6, protocol:any, source:RFC1918, destination:any, extra options:log, no advanced options.

I also tried putting such a rule on the LAN interface (out, reject, quick, src RFC1918, dest <test IP>) and, as expected, it did nothing.

stephenw10 · Feb 23, 2022, 10:46 PM

Hmm, check the actual state as it appears in the state table. Try using pfctl -vvss

If it creates a state it should be possible to add a rule that prevents it.

Steve