WG - Full tunnel problematic

80sCyborgNinja

@slugger Many thanks for the help. Yes, I'm connected via ethernet. I set both MTU & MSS to 1200 and I'm still seeing the same issues. I ran a traceroute from my android phone and noticed the trace fails at the WG0 gateway.

Thanks!

Adding trace to local dns from android device.

Slugger

@80scyborgninja So the packet hits the wg interface then disappears. Ok, so you need to find out where that packet is going after it hits the wg interface.

Do a packet capture on the wg interface. Make sure you see the packet coming in on that interface (this will also confirm that the wg config is absolutely correct).
Assuming #1 checks out, do a packet capture on your internet interface to see if the data is going out the internet as expected.

3a. If not, then time to find out where it's going. Check your firewall rules to make sure it's not being blocked. If you're sure the packet isn't being blocked then start packet capturing all your other interfaces until you find which one it's being directed out of. Assuming you find it and it's not going out the interface you want it to then should just be a matter of fixing the routing issue.

3b. If it did go out the internet interface and you never got a reply back then inspect the packet captures further (i.e. in wireshark) and start looking for things like constant retransmits, etc. If you're seeing that in the captures then we're back to a likely mtu issue. But check for the mss clamping, etc. Make sure all of those values look good.

Isn't network troubleshooting fun? Good luck. :)

80sCyborgNinja

@slugger Ok I took this a bit further over the weekend. I ran a packet capture over the weekend and had some interesting results. I used the WG client address as the host address for running the packet capture. The LAN results looked fine AFAICT, I noticed normal interaction between the LAN and WG interface. When looking at the WG interface I noticed TCP retransmits. When looking at the OVPN interface I noticed WG packets looking like the following:

IP 10.0.10.2.38338 > 142.250.64.106.443: UDP, length 1350
IP 10.0.10.2.46173 > 142.250.64.106.443: tcp 0
IP 10.0.10.2.46175 > 142.250.64.106.443: tcp 0

In wireshark, the OVPN interface also showed TCP retransmits. I was unable to get any packet capture fot the WAN interface using the WG client as the host address. I disabled the OVPN interface and tested again and still had the same results ( no internet access ). I'm still stumped on this one. Thanks!

Slugger

@80scyborgninja When capturing WAN, the src address will have been translated by the time it's captured so you'll have to capture on destination. So I would do a ping to 8.8.4.4 from the client and capture icmp traffic to 8.8.4.4 on wan.

80sCyborgNinja

@slugger Ok I see, thanks. I'm not seeing anything for 8.8.4.4 on the WAN running the ping as you described. Running the same ping out the WG interface I see [No response seen to ICMP request] in the packet capture. I also ran the ping from the firewall out the WG interface and noticed the same issue. All pings started from my android client. Thanks!

Slugger

@80scyborgninja Miscommunication, I think. :)

Start a packet capture on the WAN interface (Diagnostics > Packet Capture). Capture icmp to 8.8.4.4. Start that up. Then on your phone, ping 8.8.4.4. Confirm the pings are going out the WAN interface.

If they are and there's no response seen on WAN then something weird is going on beyond pfSense. If you don't see any of the pings going out the WAN then step back and capture on WG0 and make sure you see the traffic coming in on wg. Once you confirm that and you reconfirm it's not going out WAN then you've pretty much confirmed a routing issue in your pfsense config and it shouldn't take much to track that down and fix it.

80sCyborgNinja

@slugger Ok I ran captures as you described. The WAN interface had no output as seen via attached screenshot. The WG interface had output as seen via attached screenshot. This is the strangest issue I've ever run into with my firewall lol! I really appreciate the help!

Slugger

@80scyborgninja Well the nice thing is this now seems like one of two things:

The traffic coming in on wg0 is being blocked/dropped by pfsense
The routing of that traffic coming in wg0 is setup to send it somewhere other than WAN

That's about all there is at this stage since nothing is going out WAN. So check your firewall rules for wg0 and if that's not it then it's a routing issue.

Slugger

Check floating rules as well. Something is preventing those packets from going back out the wan interface. It's either firewall rules blocking it or misconfigured (policy) routing.

dma_pf

@slugger said in WG - Full tunnel problematic:

@80scyborgninja Well the nice thing is this now seems like one of two things:

The traffic coming in on wg0 is being blocked/dropped by pfsense

This is a guess on my part. But doesn't the any/any rule that is currently on the WireGuard tab need to be moved to the Wg0 tab? Per the docs:

Assigned WireGuard interfaces get their own individual rule tabs and will only match traffic on that specific tunnel interface. Rules on assigned WireGuard interface tabs also get reply-to which ensures that traffic entering a specific assigned WireGuard interface exits back out the same interface. Without that, return traffic will follow the default gateway.

Is it possible that the lack of a reply-to tag is messing up the routing?

Slugger

@dma_pf Good call. I didn't notice that the any/any was on the WireGuard tab. Yeah, that's probably it. I just instinctively create interfaces for all my openvpn and wg tunnels and never use the generic tabs for rules.

80sCyborgNinja

@Slugger @dma_pf I've checked all of the rules and made adjustments as described earlier and I'm still seeing the same issue unfortunately. Could it be my OVPN PIA interface messing things up? Thanks

dma_pf

@80scyborgninja said in WG - Full tunnel problematic:

I've checked all of the rules and made adjustments as described earlier and I'm still seeing the same issue unfortunately. Could it be my OVPN PIA interface messing things up? Thanks

First thing I see is you do not need the PIA rule. What that rule is saying is to allow any traffic coming across from PIA into pfsense. There should never be any traffic coming across from PIA that was not initiated by your networks. All traffic initiated from your networks would automatically be allowed back in through the PIA interface because it is stateful. You would never want to allow any uninitiated traffic across the tunnel.

I'd try changing the source on the WG0 firewall rule to any for now. Enable logging and see what happens to the traffic in the firewall logs. Which interface (PIA or WAN) are you ultimately trying to route the traffic out to the internet on? What interface do you have set as the default Gateway in System/Routing?

80sCyborgNinja

@dma_pf I removed the PIA rule as you had recommended. I also changed the source to any on the WG0 rule and enabled logging. When I ping from the WG client to 8.8.4.4 I see nothing in the system firewall logs for destination 8.8.4.4. I am still unable to ping from the client to internet hosts.

I am trying to route traffic from my wireguard client out the WG interface to the internet using full tunnel on the client. Currently I can only hit the LAN network from my WG client.The default gateway is set to my WAN DHCP gateway. Thanks!

Adding a screenshot of the states as firewall logging doesn't generate for the WG0 interface.

dma_pf

@80scyborgninja I'm seeing a couple of things that are different in your settings than mine.

In the Peer settings for your Wg0 tunnel the "Peer Wireguard Address" is blank. I populated it with my tunnel network with the /24 mask as described here. So in your case I would try 10.0.10.2/24.
In the Interface settings in the phone's app I have 2 things that are different than you. The Address field is populated with a mask of /24, yours is set to/32. And I set my MTU value to match the MSS value set for the Wg0 interface in pfsense. I think @Slugger had you set it for 1420. I would make sure they match.

dma_pf

@80scyborgninja One other difference. In the Wg0 firewall rule I have a gateway defined instead of any. In my particular case it's my VPN provider. I'd suggest trying to set it to your WAN for now, as using PIA is going to require an additional NAT rule.

80sCyborgNinja

@dma_pf Ok I made all of the changes you suggested, but still the same problem. Are you able to ping out your WG interface using pfsense Diagnostics to 8.8.4.4? I can't do that and feel that others would probably be able to with a working setup. Thanks

dma_pf

@80scyborgninja said in WG - Full tunnel problematic:

@dma_pf Ok I made all of the changes you suggested, but still the same problem. Are you able to ping out your WG interface using pfsense Diagnostics to 8.8.4.4? I can't do that and feel that others would probably be able to with a working setup. Thanks

Yes, i just tried it and it works perfectly.

80sCyborgNinja

@dma_pf damn ok I'll try and trouble shoot that a bit. Real weird. Thanks

80sCyborgNinja

@dma_pf Hmm oddly, my WG gateway IP is the IP of my WG peer and I can't seem to change it as it is set to dynamic. Not sure if that is expected, but would explain all of my issues I think. Thanks