Wireguard suddenly refuses to handshake

sLy1337

@cmcdonald @xxGBHxx I do face the same issue here with latest WG package:

pfSense-pkg-WireGuard 	0.1.5_3 	pfSense package WireGuard (EXPERIMENTAL)
wireguard-kmod 	0.0.20211105 	WireGuard implementation for the FreeBSD kernel
wireguard-tools-lite 	1.0.20210914_1 	Fast, modern and secure VPN Tunnel (lite flavor)

I have created a S2S tunnel via wireguard. This worked like 14d without any issues. One site regularly reconnects (VDSL). However, today the connection dropped and no handshake could be established.

From my home (site1, client) no connection is being initiated. I can see on the remote site (server) that no UDP packet arrives. I manually tested sending a UDP packet from my home pfSense and this arrives and is visible on the remote site. Looks like the wg process does not even try to connect or hangs somewhere in between.

Any suggestions where to troubleshoot? As wg being a silent tool, I dont see any error messages..

sLy1337

I checked further and found a message in the logs basically saying that the route for my S2S Gateway was changed.

/usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 8.8.8.8 and adding a new route through 10.1.1.1

10.1.1.1 is another gateway (LTE), not the primary (should be the gateway group..)

I checked the gateway and activated the "Disable Gateway Monitoring Action" checkmark. And now its working again... but does this make any sense? In any case, just wanted to let you know, so maybe someone faces the same issue.

robearded

@sly1337 as you pointed out, the wireguard package is pretty silent and does not output logs, which unfortunately makes it hard for us consumers to see where the problem is (maybe there are logs somewhere in a file, in a location that the developers knows, but I'm not aware of that).

The only thing that I found is by downgrading the wireguard-kmod package to an older version, the handshake and connection are made. I've added the steps on how to downgrade that package in an earlier post from this thread (or direct link: https://forum.netgate.com/post/1010944 ). This is the only (easy) solution I've found to fix this problem, I've been running this since I've found it (3 weeks ago) and so far there is no problem. I'm constantly using the wireguard connection (I use it as a Gateway for some of my servers) and had no downtime since.

sLy1337

@robearded Yep, I know. Actually I used the older version before, however did not work in my case. As mentioned, it worked when enabling "Disable Gateway Monitoring Action" in the gateway. Thanks anyways!

xxGBHxx

@sly1337 said in Wireguard suddenly refuses to handshake:

I checked further and found a message in the logs basically saying that the route for my S2S Gateway was changed.
/usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 8.8.8.8 and adding a new route through 10.1.1.1 
10.1.1.1 is another gateway (LTE), not the primary (should be the gateway group..)

I checked the gateway and activated the "Disable Gateway Monitoring Action" checkmark. And now its working again... but does this make any sense? In any case, just wanted to let you know, so maybe someone faces the same issue.

I just tried that and it made no difference to my setup.

Before that (but after the problem appeared) I did a reboot of both my ESXi host (and updated it) and of the PfSesnse server. Since then the OpenVPN tunnel and one of the WG tunnels came back up exactly as they were before. The other is permanently down and has never come back up. I'm hoping @cmcdonald will be able to get me to do any troubleshooting while it's in this state as having 1 WG tunnel down and the other up and working fine seems like a fairly good opportunity to test things.

Ultimately though this is happening enough to be an "issue" and it's not an issue I have with any of the VPN vendors own WG stacks on my mobile or my desktop. I do need to find the time to set up FreeBSD but I'm not convinced that's really going to help as I can't actually replicate when, or if, it's going to happen.

robearded

After I had to restart my modem (so the WAN went down for some minutes on the pfsense) this problem happened to me again. I can't seem to be able to initiate the handshake whatever I do. I also checked "Disable Gateway Monitoring Action" checkbox for the wireguard gateway, however it's still not working

xxGBHxx

@robearded So as of 3h ago my other WG connection has gone down for no reason again and now I can't get either of the tunnels up.

This is so infuriating :(

I can't believe it's just a handful of us.

G

cmcdonald

Hey all,

Sorry you guys are having issues with handshakes.

I do have an idea that might help illuminate where the problem lies, basically creating the tunnel from barebones at the shell and bypassing the GUI.

Here are the steps (adjust addresses, names, etc. as necessary for your situation):

Disable "WireGuard" in WireGuard > Settings. This keeps the daemon from starting. This is important because we are going to build the tunnel up manually at the shell. Though you can still create your tunnel and peer config as normal, just don't start the service.
Login to pfSense shell via SSH or console cable.
Use these commands to create the interface, assign addresses, add to interface group (if necessary) and sync WireGuard conf:

This command creates a interface of type wg with name tun_wg0:

$ ifconfig wg create name tun_wg0

This command assigns an IPv4 inet address of 10.11.12.1/24 on tun_wg0:

$ ifconfig tun_wg0 inet 10.11.12.1/24

This command assigns tun_wg0 to interface group WireGuard:

$ ifconfig tun_wg0 group WireGuard

This command syncs tun_wg0 configuration using WireGuard userland tools wg :

$ wg syncconf tun_wg0 /usr/local/etc/wireguard/tun_wg0.conf

Note: /usr/local/etc/wireguard/tun_wg0.conf can be generated by the package, just keep the service disabled for the time-being.

This command brings up tun_wg0 administratively:

$ ifconfig tun_wg0 up

At this point you should have a WireGuard tunnel interface built manually. You can now proceed with assigning it to pfSense, etc.

This test is useful because it bypasses all WireGuard package semantics and only uses pfSense core logic...useful for isolating potential issues.

cmcdonald

Youtube Video

This video walks through several scenarios that should be applicable here, hopefully this helps!

This video is more aimed at discussing the various ways I test and live with WireGuard on a daily basis, but I walk through the setup and configuration of each scenario.

robearded

@robearded said in Wireguard suddenly refuses to handshake:

After I had to restart my modem (so the WAN went down for some minutes on the pfsense) this problem happened to me again. I can't seem to be able to initiate the handshake whatever I do. I also checked "Disable Gateway Monitoring Action" checkbox for the wireguard gateway, however it's still not working

It turns out this happened because of a change I did on the wireguard server without realizing it will break stuff. I'm not however sure if I did this change before the first occurrence of this problem, from what I remember it is after, but it is definitely possible to be before.
I had some DNAT iptables rules in the PostUp and PostDown config of wireguard to forward some ports to pfsense, so I can use the public IP of the wg server as a public IP for the server running pfsense. I figured out that it would be easier to DNAT all ports (except SSH) and just do the firewalling inside pfsense, so I don't have to log in to the wg server everything I want to forward another port, and I can do it all from pfsense. Well, the problem is, that I didn't realized I also forwarded the port WG was listening on.

Woodsomeister

Hello,

i'm facing the same problem with the error status of

/usr/local/pkg/wireguard/includes/wg_service.inc: Static Routes: Gateway IP could not be found ...

No handshake seems to happen anymore. Reinstalled the wireguard package several times, and also tried a new version of the kmod implementation. No success.

Jarhead

@woodsomeister Post pictures of your wireguard tunnel, peers, interface and routes/gateway.

Woodsomeister

This post is deleted!

Woodsomeister

@jarhead

Routes were added for the marked Wireguard interface and i also configured the interface using the right IPv4 static IP (10.0.41.1, peer is 10.0.41.2). (no screenshot)
Unfortunatly i had to switched back to IPSec, so that's why everything is deactivated.

Jarhead

@woodsomeister You have to add the tunnel as an allowed IP.
Did this ever work?

Woodsomeister

@jarhead For several weeks. In the other side is also a Opnsense under my control. Everything was working as expected, but after a restart some days ago i went into this problem.

itheadquarters

@xxgbhxx I tried the version 0.0.20210606_1 solution and now wireguard crashes and I get crash reports. I tried uninstalling wireguard and installing but now the WG service won't stay running.
What's my next step?

n3IVI0

DO NOT DO THIS.

Changing the wireguard-kmod package will cause kernel panics and take down your pfSense box.

I have the same problem with WireGuard not handshaking, but this solution DOES NOT fix it. FYI.

n3IVI0

@itheadquarters Same thing happened to me. You need to SSH into your box, run 'pkg install wireguard-kmod'. This will put your package back to wireguard-kmod-0.0.20211105. This doesn't fix the handshake problem, but it will stop the kernel panics.

You can check your version numbers with 'pkg info -l wireguard-kmod'

xxGBHxx

@itheadquarters Sorry no notifications you'd replied to me. That wasn't my suggestion by the way.

I've given up with pfSense' implementation of Wireguard. I find it unstable and flakey as hell.

I purposely don't touch it once it's working and pray it's going to keep working. I've had a few unexplained random drops since I posted this and I just bypass the vpn until it comes back up which might be minutes, hours or days.

To illustrate this I have relatively recently set up a new connection to one of the VPN providers I've set the connection up to before. The settings are identical (obviously not they key's). Everything I can see is set up exactly the same as the working one and it has NEVER worked.

I've deleted and re-built that VPN 10+ times and it will never connect. In anger I set up and configured a brand new Opnsense VM using the exact same keys and settings and it worked instantly first time. It's never dropped. Not even once.

I can only go by what I find. If you're lucky, you get no issues but I have found, with my setup, Wireguard is just broken.

As soon as I get a few days free I'm going to be converting over to Opnsense full time not elast as its fairly clear to me that Netgate have walked away from the community version of PFSense.