Source address not NATed during OpenVPN startup?

bPsdTZpW · Feb 22, 2022, 11:14 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

Ok so you have a policy based routing rule on the LAN for traffic from the LAN via the gateway group?

I have a gateway group for traffic to the WAN, with the VPN gateway at high priority, and WAN_IGB0's gateway at low priority.

Is the system default route via the WAN?

The system default route (via system/routing/gateways) is the gateway group.

bPsdTZpW · Feb 22, 2022, 11:47 PM

Clue: checking system/advanced/networking/Reset All States results in fewer (but not necessarily no) bad packets exiting WAN during VPN startup.

stephenw10 · Feb 23, 2022, 1:39 AM

Ah, OK I so not rule on LAN with the group defined. It's only used in the system default gateway to route traffic?

Hmm, I could see that being an issue. If you policy route the traffic on LAN all the outgoing traffic will get tagged route-to for the current gateway. Can you test that?

Traffic leaving with the LAN interface IP as source is odd though. That should only ever be possible if it's NAT'd to it. It almost seems like it's falling back to the LAN IP as the next interface when the assigned OpenVPN goes down...

bPsdTZpW · Feb 23, 2022, 1:50 AM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

Ah, OK I so not rule on LAN with the group defined. It's only used in the system default gateway to route traffic?

Right. Only as the system default gateway.

Hmm, I could see that being an issue. If you policy route the traffic on LAN all the outgoing traffic will get tagged route-to for the current gateway. Can you test that?

OK.

Traffic leaving with the LAN interface IP as source is odd though. That should only ever be possible if it's NAT'd to it. It almost seems like it's falling back to the LAN IP as the next interface when the assigned OpenVPN goes down...

The even odder thing is that the bad packets get passed when the VPN is coming up, at about the time it issues the message

Initialization Sequence Completed

It does seem like it's falling back to the LAN IP, though.

bPsdTZpW · Feb 23, 2022, 1:57 AM

@bpsdtzpw So policy-routing from LAN to the gateway group gives the same behavior.

bPsdTZpW · Feb 23, 2022, 2:04 AM

The bad packets get passed around the time the following appears in the system log:

Feb 22 17:55:49 	php-fpm 	412 	/rc.start_packages: Restarting/Starting all packages.
Feb 22 17:55:48 	check_reload_status 	441 	Starting packages
Feb 22 17:55:48 	php-fpm 	98676 	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - <former IP> -> <former IP> - Restarting packages.
Feb 22 17:55:46 	php-fpm 	98676 	/rc.newwanip: Creating rrd update script
Feb 22 17:55:43 	php-fpm 	98676 	/rc.newwanip: IP Address has changed, killing states on former IP Address <former IP>.
Feb 22 17:55:43 	php-fpm 	98676 	/rc.newwanip: Default gateway setting Interface <vpn> Gateway as default.
Feb 22 17:55:43 	php-fpm 	98676 	/rc.newwanip: Gateway, switch to: <vpn>
Feb 22 17:55:43 	php-fpm 	98676 	<VPN gateway>|<VPN virtual IP>|<vpn>|9.916ms|0.516ms|0.0%|online|none

bPsdTZpW · Feb 23, 2022, 2:20 AM

Another clue: if I change the NAT rule from LAN to WAN_IGB0 to use the WAN's IP address directly (e.g. w.x.y.z) instead of "interface address", the behavior is the same. This seems to indicate that the NAT rules aren't being used when the bad packets are passed.

stephenw10 · Feb 23, 2022, 3:44 PM

Mmm, if if was NATing to the LAN IP or using the OBN rules at all you would see it in the created states.

Since it's being passed by a state opened on LAN you could try adding a block rule on LAN to prevent it as a workaround.

Steve

bPsdTZpW · Feb 23, 2022, 5:56 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

Mmm, if if was NATing to the LAN IP or using the OBN rules at all you would see it in the created states.

Since it's being passed by a state opened on LAN you could try adding a block rule on LAN to prevent it as a workaround.

Steve

What block rule could I use? From the point of view of the LAN interface, the packets are perfectly OK (src:LAN device, dest:internet).

stephenw10 · Feb 23, 2022, 6:12 PM

It's source: LANaddress destination: Internet though and it's outbound which should never happen.

You want to make it as specific as possible so I'd use a floating, quick, outbound rule, source: LANaddress destination: some-test-address. Make sure that does something useful and does block expected traffic before changing the destination to some thing wider.

Steve

bPsdTZpW · Feb 23, 2022, 6:29 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

It's source: LANaddress destination: Internet though and it's outbound which should never happen.

You want to make it as specific as possible so I'd use a floating, quick, outbound rule, source: LANaddress destination: some-test-address. Make sure that does something useful and does block expected traffic before changing the destination to some thing wider.

Steve

I think I see. I already have a rule like this, and it doesn't work. From the original post:

I have put various "reject" floating rules on outbound WAN [by which I meant floating, WAN, outbound] to prevent these packets from exiting [1]...[1] e.g. action:block, quick, interface:WAN_IGB0, direction:out, family:IPV4+IPV6, protocol:any, source:RFC1918, destination:any, extra options:log, no advanced options.

I also tried putting such a rule on the LAN interface (out, reject, quick, src RFC1918, dest <test IP>) and, as expected, it did nothing.

stephenw10 · Feb 23, 2022, 10:46 PM

Hmm, check the actual state as it appears in the state table. Try using pfctl -vvss

If it creates a state it should be possible to add a rule that prevents it.

Steve

bPsdTZpW · Feb 23, 2022, 11:32 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

Hmm, check the actual state as it appears in the state table. Try using pfctl -vvss

If it creates a state it should be possible to add a rule that prevents it.

Steve

So after the bad packets have passed, pfctl -vvss gives many states of this form:

all tcp <website IP>:443 <- <LAN IP>:64970       CLOSED:SYN_SENT
   [0 + 16777216]  [1164774331 + 33554432]
   age 00:01:20, expires in 00:00:40, 1:1 pkts, 52:80 bytes, rule 122
   id: 4c80166200000000 creatorid: c79f1419 gateway: 0.0.0.0
   origif: igb1

which presumably correspond to some of the bad packets.

However, there are also a few state pairs that appear to correspond to working, NATed packets. The first state of the pair is very similar to the bad state, above, so I don't see how I could filter on it:

all tcp <website IP>:443 <- <LAN IP>:64972       TIME_WAIT:TIME_WAIT
   [4088558557 + 132096] wscale 7  [1887699560 + 1282998272] wscale 8
   age 00:01:20, expires in 00:01:13, 268:128 pkts, 27167:143271 bytes, rule 122
   id: 4d80166200000000 creatorid: c79f1419 gateway: 0.0.0.0
   origif: igb1

all tcp <WAN IP>:32245 (<LAN IP>:64972) -> <website IP>:443       ESTABLISHED:ESTABLISHED
   [3162374760 + 8323072] wscale 8  [4088558557 + 132096] wscale 7
   age 00:01:20, expires in 23:59:24, 268:128 pkts, 27167:143271 bytes, rule 74
   id: 4e80166200000000 creatorid: c79f1419 gateway: <WAN gateway>
   origif: igb0

This really seems like a bug.

stephenw10 · Feb 23, 2022, 11:38 PM

It does. The only way I could possibly see anything sourced from the LAN IP itself would be some sort of proxy running. So Squid, HAProxy or NAT reflection in NAT+Proxy mode.

What is rule 122 in your ruleset?

Steve

bPsdTZpW · Feb 23, 2022, 11:56 PM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

It does.

The only way I could possibly see anything sourced from the LAN IP itself would be some sort of proxy running. So Squid, HAProxy or NAT reflection in NAT+Proxy mode.

Nope, nope, and nope. The setup is quite vanilla. I have one package: service_watchdog.

What is rule 122 in your ruleset?

The only rule having any such number in the output of pfctl -vvsa is

@122(0) block drop in log quick on igb0 inet6 proto udp from any to any port = nameserver label "USER_RULE: Ports to monitor" ridentifier 1628204890
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: pid 61558 State Creations: 0     ]

which makes not the least bit of sense. This is an inbound WAN block/log rule I use to see how many attempts at common ports the bad guys are making. Am I reading the numbers wrong?

stephenw10 · Feb 24, 2022, 12:30 AM

Mmm, indeed. Can you see what rule 122 is or was when the OpenVPN is up?

This starts to look like a stale state somehow.

bPsdTZpW · Feb 24, 2022, 12:37 AM

@stephenw10 said in Source address not NATed during OpenVPN startup?:

Mmm, indeed. Can you see what rule 122 is or was when the OpenVPN is up?

That rule is from after OpenVPN came up. I don't know what the numbering was before it came up; it would be tricky to get; I'd probably need to write a script.

This starts to look like a stale state somehow.

Well, I did find that setting Reset All States in System/Advanced/Networking reduces (but does not zero) the number of bad packets.