(IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP

stephenw10 · Feb 22, 2021, 8:03 PM

Run: pfctl -ss -vvvv

Find the state(s) open with port 51820. Note the rule number that opened them.

Now run: pfctl -sr -vvvv

Check what rule opened the state.

Steve

kevindd992002 · Feb 23, 2021, 3:40 PM

With the WAN rule enabled, running "pfctl -sr -vvvv" shows the user rule I created and like I said, the open state is as expected:

pfctl -ss -vvvv | grep 51820
igb0 udp {server public IP}:51820 <- {client public IP}:19988       MULTIPLE:MULTIPLE

pfctl -sr -vvvv
@68(1613966106) pass in quick on igb0 reply-to (igb0 {server WAN gateway IP}) inet proto udp from any to {server public IP} port = 51820 keep state label "USER_RULE: pfSense WireGuard Server"
  [ Evaluations: 21757     Packets: 6025      Bytes: 4078532     States: 1     ]
  [ Inserted: pid 28901 State Creations: 1     ]

Right after disabling the rule and deleting the open state on the server, it opens a new state but this time in reverse and naturally no rules are triggered because it becomes an outbound call:

pfctl -ss -vvvv | grep 51820
igb0 udp {server public IP}:51820 -> {client public IP}:19988       MULTIPLE:MULTIPLE

pfctl -sr -vvvv
@68(1613966106) pass in quick on igb0 reply-to (igb0 {server WAN gateway IP}) inet proto udp from any to {server public IP} port = 51820 keep state label "USER_RULE: pfSense WireGuard Server"
  [ Evaluations: 24229     Packets: 6933      Bytes: 4638020     States: 0     ]
  [ Inserted: pid 28901 State Creations: 1     ]

So how do both sides have outbound connections and still achieve a tunnel? I'm just curious here. The open state on the client side never changes direction. It's always client to server and the source and destination port is always 51820.

stephenw10 · Feb 23, 2021, 1:19 PM

You have client IP twice in the second state. I assume that's a typo?

Are you saying the state that opens without the WAN rule enabled shows it was created by the same rule number?

Steve

kevindd992002 · Feb 23, 2021, 3:50 PM

@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

You have client IP twice in the second state. I assume that's a typo?

Are you saying the state that opens without the WAN rule enabled shows it was created by the same rule number?

Steve

Edited, sorry about that.

No. After I disable the WAN rule and delete the open state on the server side, a new state in the server side is recreated but seems to be outbound. So the same rule number does not show it was used because naturally it is outbound. But then after a while (like how I checked again now), the open state on the server seems to be as expected again BUT the rule state details under the WAN tab does not reflect the correct "bytes".

stephenw10 · Feb 23, 2021, 4:14 PM

Ok, but the rule that is creating that state is not '@68' I assume?

I imagine the WireGuard process has a record of the client IP and source port and start sending traffic back to it. The 'allow everything out' rule will pass that and create a new state. Since that isn't the rule on WAN that doesn't show the traffic.

If you removed the rule on the server side WAN and then restarted WireGuard or rebooted I would expect it to fail to come back up.

Steve

kevindd992002 · Feb 23, 2021, 4:24 PM

@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

Ok, but the rule that is creating that state is not '@68' I assume?

I imagine the WireGuard process has a record of the client IP and source port and start sending traffic back to it. The 'allow everything out' rule will pass that and create a new state. Since that isn't the rule on WAN that doesn't show the traffic.

If you removed the rule on the server side WAN and then restarted WireGuard or rebooted I would expect it to fail to come back up.

Steve

Correct, it's not '@68' for sure because the connection was outbound.

What you're saying makes sense. Since the client side has an open state, the server can reach it on that client IP and source port outbound. That's actually clever.

That will be a good test. I will expect it to fail too. I'll try later and report back.

However, do you have any ideas on the inaccurate "bytes" on the rule state details?

And why I see the "matchaddr failed" errors on both sides now?

stephenw10 · Feb 23, 2021, 6:35 PM

@kevindd992002 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

matchaddr failed

Yeah that should not be there. Do you have more that one peer defined?
https://redmine.pfsense.org/issues/11502

Steve

kevindd992002 · Feb 23, 2021, 11:36 PM

@stephenw10

Not at all. I only have one peer serup on both sides.

stephenw10 · Feb 23, 2021, 11:40 PM

Hmm

kevindd992002 · Feb 24, 2021, 12:00 AM

@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

Hmm

Can I file it as a bug?

stephenw10 · Feb 28, 2021, 4:55 PM

Yes, if you have details there and can replicate it.

Kuser · Mar 12, 2021, 1:38 PM

@kevindd992002

I think I'm seeing the same behavior.
My setup might be a bit different but I have two netgate appliances: SG-3100 & SG-5100 and site2site wireguard vpn setup according to the howto. After disabling the WAN inbound firewall rules for 51820 on both devices and even killing states the link still comes online.

Since disabling the rules didn't work I set an explicit reject instead of allow, but connection still comes up... So it seems wireguard doesn't care about my WAN rules.

stephenw10 · Mar 12, 2021, 1:56 PM

That is expected if both sides have a remote IP and there in nothing NATing in between them. Both ends open an outbound UDP state with 51280 as the source and destination port. The traffic arriving from the other side can use that open state to connect.

This should be a new thread in the WireGuard or Firewall subs at this point. It's no longer IPSec related at all.

Steve

Kuser · Mar 12, 2021, 1:59 PM

@stephenw10
Thanks for the answer, not my intention to hijack the post. It just seemed relevant to the WG related questions. You are correct both sides have remote IP defined, but also explicit firewall WAN -rules to reject the connection.

Is this expected, even though I restarted one of the firewalls (e.g no active states)

stephenw10 · Mar 12, 2021, 2:02 PM

If the reject rules are inbound they won't do anything over the default block rule. The default outbound allow rules will still allow the state to open on each end letting the traffic enter from the remote side.

Steve

Kuser · Mar 12, 2021, 2:05 PM

@stephenw10

Thanks again for the clarification, so this basically means that the WAN inbound rules as in howto are not necessary at all in this scenario?

https://docs.netgate.com/pfsense/en/latest/recipes/wireguard-s2s.html

Wish you a great day.

stephenw10 · Mar 12, 2021, 2:30 PM

There is no harm in adding them to the specific remote IPs and it may allow the tunnel to come up faster. Without then it can only establish once both sides are sending outbound traffic.

Steve

kevindd992002 · Mar 12, 2021, 3:57 PM

@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

There is no harm in adding them to the specific remote IPs and it may allow the tunnel to come up faster. Without then it can only establish once both sides are sending outbound traffic.

Steve

Does that mean both sides can be behind a CGNAT device (WAN interface is private IP) and the WG tunnel can still be established because both ends will just open a state when they initiate an outbound connection?

stephenw10 · Mar 13, 2021, 4:19 PM

Unlikely.
It would work as long as they were both using dyndns to give the other something to open a state to and the provider is not using source port randomisation. They probably are though.

Steve