WG - Full tunnel problematic
-
Hi All,
I'm experiencing a very similar issue to this post https://forum.netgate.com/topic/161733/wg-full-nat-not-routing-traffic. I can't seem to figure out why my android wg peer is unable to full tunnel. I'm able to access LAN resources, but my WG peer IP can't seem to hit the internet. I'm seeing this in the states dump:
WG0 to Internet
WG0 tcp 10.0.10.2:44165 -> 34.107.221.82:80 CLOSED:SYN_SENT 1 / 0 60 B / 0 BWG0 to LAN
WG0 tcp 10.0.10.2:46172 -> 10.0.0.15:8009 ESTABLISHED:ESTABLISHED 16 / 17 3 KiB / 8 KiBI'm also using a Private Internet Access OVPN client setup with my pfSense firewall. See attached screenshots for further details.
Thanks!
-
Best guess is an mtu issue on the wg interface. What type of Internet connection do you have? If it uses pppoe, you probably will need to set the mtu and mss values on your wg interface to 1412 or lower. The mtu needs to be 80 less than the mtu of your WAN interface. Whatever you set the mtu to, set the mss to the same value. The default mtu for wg interfaces is 1420, but if your internet uses pppoe then you need to lower the mtu and make sure the mss clamping is set properly
-
@slugger My internet connection is broadband highspeed. I changed the WG0 interface MTU and MSS values. I'm still seeing the same issues as before unfortunately. Any other ideas? FWIW, I can run wireguard on a RPI3b+ and everything works with respect to full tunnel. I'd love to get this working on my firewall.
Thanks!
-
The MSS should be set to 1420 (pfsense will subtract the 40 from the number you enter). And since you're using 1420 I'm assuming you're connected via standard ethernet and not pppoe or the like? I don't think the MSS will fix your problem, but either way you should set it to 1420.
It still looks like an mtu problem to me. Assuming the MSS change doesn't fix it, I'd start lowering the MTU & MSS down to see if you can find a spot where it starts to work. Set both to 1200 and see if that works. If it does suddenly start working then raise it up incrementally until you get to the max that works for you. If 1200 still doesn't work then it's not likely an mtu issue and then I'm out of ideas, unfortunately. Maybe try pings & traceroutes out to the internet. Do they work? If not, how far out does the traceroute get? If the traceroute isn't getting beyond pfsense then you're back to a pfsense/LAN issue.
-
@slugger Many thanks for the help. Yes, I'm connected via ethernet. I set both MTU & MSS to 1200 and I'm still seeing the same issues. I ran a traceroute from my android phone and noticed the trace fails at the WG0 gateway.
Thanks!
Adding trace to local dns from android device.
-
@80scyborgninja So the packet hits the wg interface then disappears. Ok, so you need to find out where that packet is going after it hits the wg interface.
-
Do a packet capture on the wg interface. Make sure you see the packet coming in on that interface (this will also confirm that the wg config is absolutely correct).
-
Assuming #1 checks out, do a packet capture on your internet interface to see if the data is going out the internet as expected.
3a. If not, then time to find out where it's going. Check your firewall rules to make sure it's not being blocked. If you're sure the packet isn't being blocked then start packet capturing all your other interfaces until you find which one it's being directed out of. Assuming you find it and it's not going out the interface you want it to then should just be a matter of fixing the routing issue.
3b. If it did go out the internet interface and you never got a reply back then inspect the packet captures further (i.e. in wireshark) and start looking for things like constant retransmits, etc. If you're seeing that in the captures then we're back to a likely mtu issue. But check for the mss clamping, etc. Make sure all of those values look good.
Isn't network troubleshooting fun? Good luck. :)
-
-
@slugger Ok I took this a bit further over the weekend. I ran a packet capture over the weekend and had some interesting results. I used the WG client address as the host address for running the packet capture. The LAN results looked fine AFAICT, I noticed normal interaction between the LAN and WG interface. When looking at the WG interface I noticed TCP retransmits. When looking at the OVPN interface I noticed WG packets looking like the following:
IP 10.0.10.2.38338 > 142.250.64.106.443: UDP, length 1350 IP 10.0.10.2.46173 > 142.250.64.106.443: tcp 0 IP 10.0.10.2.46175 > 142.250.64.106.443: tcp 0In wireshark, the OVPN interface also showed TCP retransmits. I was unable to get any packet capture fot the WAN interface using the WG client as the host address. I disabled the OVPN interface and tested again and still had the same results ( no internet access ). I'm still stumped on this one. Thanks!
-
@80scyborgninja When capturing WAN, the src address will have been translated by the time it's captured so you'll have to capture on destination. So I would do a ping to 8.8.4.4 from the client and capture icmp traffic to 8.8.4.4 on wan.
-
@slugger Ok I see, thanks. I'm not seeing anything for
8.8.4.4on the WAN running the ping as you described. Running the same ping out the WG interface I see[No response seen to ICMP request]in the packet capture. I also ran the ping from the firewall out the WG interface and noticed the same issue. All pings started from my android client. Thanks!
-
@80scyborgninja Miscommunication, I think. :)
Start a packet capture on the WAN interface (Diagnostics > Packet Capture). Capture icmp to 8.8.4.4. Start that up. Then on your phone, ping 8.8.4.4. Confirm the pings are going out the WAN interface.
If they are and there's no response seen on WAN then something weird is going on beyond pfSense. If you don't see any of the pings going out the WAN then step back and capture on WG0 and make sure you see the traffic coming in on wg. Once you confirm that and you reconfirm it's not going out WAN then you've pretty much confirmed a routing issue in your pfsense config and it shouldn't take much to track that down and fix it.
-
@slugger Ok I ran captures as you described. The WAN interface had no output as seen via attached screenshot. The WG interface had output as seen via attached screenshot. This is the strangest issue I've ever run into with my firewall lol! I really appreciate the help!
-
@80scyborgninja Well the nice thing is this now seems like one of two things:
- The traffic coming in on wg0 is being blocked/dropped by pfsense
- The routing of that traffic coming in wg0 is setup to send it somewhere other than WAN
That's about all there is at this stage since nothing is going out WAN. So check your firewall rules for wg0 and if that's not it then it's a routing issue.
-
Check floating rules as well. Something is preventing those packets from going back out the wan interface. It's either firewall rules blocking it or misconfigured (policy) routing.
-
@slugger said in WG - Full tunnel problematic:
@80scyborgninja Well the nice thing is this now seems like one of two things:
- The traffic coming in on wg0 is being blocked/dropped by pfsense
This is a guess on my part. But doesn't the any/any rule that is currently on the WireGuard tab need to be moved to the Wg0 tab? Per the docs:
Assigned WireGuard interfaces get their own individual rule tabs and will only match traffic on that specific tunnel interface. Rules on assigned WireGuard interface tabs also get reply-to which ensures that traffic entering a specific assigned WireGuard interface exits back out the same interface. Without that, return traffic will follow the default gateway.
Is it possible that the lack of a reply-to tag is messing up the routing?
-
@dma_pf Good call. I didn't notice that the any/any was on the WireGuard tab. Yeah, that's probably it. I just instinctively create interfaces for all my openvpn and wg tunnels and never use the generic tabs for rules.
-
-
@80scyborgninja said in WG - Full tunnel problematic:
I've checked all of the rules and made adjustments as described earlier and I'm still seeing the same issue unfortunately. Could it be my OVPN PIA interface messing things up? Thanks
First thing I see is you do not need the PIA rule. What that rule is saying is to allow any traffic coming across from PIA into pfsense. There should never be any traffic coming across from PIA that was not initiated by your networks. All traffic initiated from your networks would automatically be allowed back in through the PIA interface because it is stateful. You would never want to allow any uninitiated traffic across the tunnel.
I'd try changing the source on the WG0 firewall rule to any for now. Enable logging and see what happens to the traffic in the firewall logs. Which interface (PIA or WAN) are you ultimately trying to route the traffic out to the internet on? What interface do you have set as the default Gateway in System/Routing?
-
@dma_pf I removed the PIA rule as you had recommended. I also changed the source to any on the WG0 rule and enabled logging. When I ping from the WG client to
8.8.4.4I see nothing in the system firewall logs for destination8.8.4.4. I am still unable to ping from the client to internet hosts.I am trying to route traffic from my wireguard client out the WG interface to the internet using full tunnel on the client. Currently I can only hit the LAN network from my WG client.The default gateway is set to my WAN DHCP gateway. Thanks!
Adding a screenshot of the states as firewall logging doesn't generate for the WG0 interface.
-
@80scyborgninja I'm seeing a couple of things that are different in your settings than mine.
-
In the Peer settings for your Wg0 tunnel the "Peer Wireguard Address" is blank. I populated it with my tunnel network with the /24 mask as described here. So in your case I would try 10.0.10.2/24.
-
In the Interface settings in the phone's app I have 2 things that are different than you. The Address field is populated with a mask of /24, yours is set to/32. And I set my MTU value to match the MSS value set for the Wg0 interface in pfsense. I think @Slugger had you set it for 1420. I would make sure they match.
-
-
@80scyborgninja One other difference. In the Wg0 firewall rule I have a gateway defined instead of any. In my particular case it's my VPN provider. I'd suggest trying to set it to your WAN for now, as using PIA is going to require an additional NAT rule.
