Wireguard suddenly refuses to handshake

xxGBHxx

Hi,

I have an infuriating problem.

I have Wireguard set up and configured to connect to two completely separate VPN providers. This is a perfectly bog standard connection with some conditional routing and has been working fine for one of the connections for 22 days. The other I only set up last week to the second VPN and that was set up, configured and working within minutes. No problems with either connection and it was working perfectly fine.

Today I made the fatal mistake of updating to the latest Wiregaurd. Since doing that and not changing ANYTHING other than hitting update neither tunnel connects at all. They both never handshake and report no errors. As I mentioned absolutely NOTHING has been changed.

I have re-booted and this has made absolutely no difference. Neither connection will handshake.

But this isn't the first time.

22 days ago I'd had 2 VPN connections going to the same VPN provider. In that case I had added a new network connection and had suffered the exact same problem (neither connection to the same VPN provider would handshake). In that situation I tried EVERYTHING (deleting connections and re-creating, creating new keys, connecting to different peers etc. I had assumed this was something to do with the VPN provider. Nothing worked to the point I eventually had to re-install pfSense from scratch and everything started working again perfectly.

The believe the problem is with pfSense and Wireguard. Once it happens there is seemingly NOTHING I can do to unlock Wireguard again and get it to handshake.

What am I doing wrong? How can I "flush" Wireguard out and get it to connect again without needing to re-install the firewall from scratch?

G

MoonKnight

@xxgbhxx

I have 4 VPN tunnels running (24/7) via WireGuard, and I hit the update button. Everything is still running smoothly without any problems.
So I'm not sure why you are experiencing problems after updating to the newest WireGuard.
I would probably do a re-install of the WireGuard package before I do a clean install of pfSense. What version of pfSense are you using? I'm on version 2.5.2.

xxGBHxx

@ciscox Yes everything is running the very latest.

There is something very broken somewhere.

So as I've been here before I took a snapshot of the FW as I'm running it in VMWare. When I couldn't get anything working I went back to the fully working and perfectly ok snapshot and now THAT isn't working either.

I had considered re-installing the package but it becomes problematic as without the Wireguard tunnel on the Firewall I have absolutely zero connectivity. While I can simply enable that for the LAN I'm struggling to enable the firewall itself to talk out.

I am wondering if it's the VMX3 interfaces I'm using in VMWare that are just breaking with pfSense. I know I've had some issues with the way the virtual interfaces on VMWare interact with pfSense but I'm starting to wonder if it may be interaction with the type of virtual interface I'm using. That said in their VMWare guide they state you can use either type of interface.

I will try to re-install it.

xxGBHxx

So finally managed to get the firewall connected by setting up a new OpenVPN connection and using that. I was then able to re-install and bang, instant connection with both VPN's

I have no idea why it's done it twice now but at least I didn't need to go through the pain of re-installing again. Thanks for the kick to go do it.

Doesn't in any way explain why my snapshotted VM with a working config ALSO didn't work.

I'm sure there's issues there somewhere. Guess I'm never going to find out what it is.

G

robearded

I have the exact problem, last handshake is 15 hours ago (and that is kind of the same time I pressed the update button). My pfsense is connected through wireguard to a VPS for a "reverse proxy" like setup (using the wireguard connection as a WAN). Pfsense doesn't want to connect to the server anymore since the wireguard package upgrade.

Unfortunately, I can't use another VPN for this (like OpenVPN) because it can't handle the amount of bandwidth I'm running through Wireguard.

I can connect from my laptop to a wireguard server running on the pfsense, however pfsense doesn't want to connect to another remote.

robearded

Sorry for the double post, but I'm not able to edit my post anymore.
I've found the solution of this problem.

SOLUTION

The problem seems to be with the latest version of the package "wireguard-kmod" (0-0.0.20211105) that the wireguard package depends on. By manually installing an older version of wireguard-kmod, the problem is fixed.

Steps to downgrade:

Download "wireguard-kmod" version 0.0.20210606_1 from here: https://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/All/wireguard-kmod-0.0.20210606_1.txz
Go to your pfsense dashboard -> Diagnostics -> Command prompt.
On the Upload File option, select your downloaded wireguard-kmod package and upload it. After upload is finished, you should receive a message that your file is uploaded in /tmp/wireguard-kmod-0.0.20210606_1.txz
On the Execute Shell Command option (it's located on the same page) run this command: pkg install -y /tmp/wireguard-kmod-0.0.20210606_1.txz

Let the command finish execution and you should see an output saying package installed successful. Restart your wireguard service and it should be able to connect to remotes again.

xxGBHxx

@robearded said in Wireguard suddenly refuses to handshake:

Sorry for the double post, but I'm not able to edit my post anymore.
I've found the solution of this problem.

SOLUTION

The problem seems to be with the latest version of the package "wireguard-kmod" (0-0.0.20211105) that the wireguard package depends on. By manually installing an older version of wireguard-kmod, the problem is fixed.

Steps to downgrade:

Download "wireguard-kmod" version 0.0.20210606_1 from here: https://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/All/wireguard-kmod-0.0.20210606_1.txz

Go to your pfsense dashboard -> Diagnostics -> Command prompt.

On the Upload File option, select your downloaded wireguard-kmod package and upload it. After upload is finished, you should receive a message that your file is uploaded in /tmp/wireguard-kmod-0.0.20210606_1.txz

On the Execute Shell Command option (it's located on the same page) run this command: pkg install -y /tmp/wireguard-kmod-0.0.20210606_1.txz

Let the command finish execution and you should see an output saying package installed successful. Restart your wireguard service and it should be able to connect to remotes again.

Nice work!

That could have saved me a massive amount of hassle a few weeks ago.

G

cmcdonald

@xxgbhxx What VPN providers are you having issues with?

xxGBHxx

@cmcdonald I have been using IVPN.net and WeVPN.

IVPN.net provides a specific setup guide for pfSense and Wireguard which is what I followed. This worked perfectly fine on both the old Wireguard (previous guide) and the new Wireguard (new guide) implementations until things just randomly stop working.

I started using WeVPN in the last few weeks in order to prove to myself it wasn't IVPN.net (or me!) causing the issues. Set it up in about 2 mins, worked instantly and has remained stable until I upgraded Wireguard as above.

Both vendors provide apps for their VPN and I've used the Android and Windows apps on Wireguard and OpenVPN. Both work flawlessly for both my ISP's every single time and have never failed.

I have also had IVPN.net verify the servers at their end that there were not any strange/dead connections and they could see none at all.

G

cmcdonald

@xxgbhxx Just tested with IVPN.net and had no issues.

Are you sure you are using the correct endpoint port?

As per https://www.ivpn.net/setup/router/pfsense-wireguard/

Endpoint Port: Choose one of 53, 2049, 2050, 30587, 41893, 48574, or 58237, all are equally secure

I saw no handshakes until I read read the manual and chose one of the above ports.

22.01-DEVELOPMENT][admin@pfSense.home.arpa]/root: wg
interface: tun_wg0
  public key: qck10oARb58oS+0owfGFRK4K8tpxVPZxLTw5nccb9go=
  private key: (hidden)
  listening port: 51820

peer: ANhVUMAQgStPVNRHW8mg0ZtN1YI1QHyXfNCO8+USNQQ=
  endpoint: 72.11.137.148:53
  allowed ips: ::/0, 0.0.0.0/0
  latest handshake: 1 minute, 19 seconds ago
  transfer: 184 B received, 10.08 KiB sent
  persistent keepalive: every 30 seconds
[22.01-DEVELOPMENT][admin@pfSense.home.arpa]/root:

xxGBHxx

@cmcdonald I've had it working many times on multiple different ports to multiple different peers. When it works, it works.

When it fails however it seems to get into a lock and nothing short of re-installing the entire FW is sure to fix it. I wish there was more troubleshooting I could do to help.

When it happened the last time I tried to do some packet captures but all that showed was the packets getting sent by the FW but never being answered. Didn't matter what port, what peer, what key the result was the same.

This is NOT just limited to IVPN as it affected WeVPN the last time in exactly the same way.

I think the best I can do if you really want to help is that I try and get it to do it again. When it's happening then I can run any test you like.

cmcdonald

@xxgbhxx That is very strange. Reinstalling the firewall is a pretty heavy-handed solution. If you're confident that the issue is with pfSense (and thus FreeBSD), I would suggest standing up a FreeBSD VM based on the same version of pfSense and configuring a tunnel there. If the issue is still present on vanilla FreeBSD, then that is a good indication we are looking at an issue related to the kernel code and not something higher up the stack.

xxGBHxx

@cmcdonald Sorry been a week or so due to work and running out of space on my VMWare server. Not yet managed to get FreeBSD installed as I only sorted it out today.

In the interim I re-setup the two VPN's again and they've both been up and working for the past week or so without an issue. They have re-connected through ISP drops but for the past 18 hours the IVPN VPN has failed to handshake. I'm not touching it again to tryand getit to re-connect. The other WG VPN is up and working without an issue.

So anything you'd like me to do to test or anything I should do before I try and reconnect it?

G

sLy1337

@cmcdonald @xxGBHxx I do face the same issue here with latest WG package:

pfSense-pkg-WireGuard 	0.1.5_3 	pfSense package WireGuard (EXPERIMENTAL)
wireguard-kmod 	0.0.20211105 	WireGuard implementation for the FreeBSD kernel
wireguard-tools-lite 	1.0.20210914_1 	Fast, modern and secure VPN Tunnel (lite flavor)

I have created a S2S tunnel via wireguard. This worked like 14d without any issues. One site regularly reconnects (VDSL). However, today the connection dropped and no handshake could be established.

From my home (site1, client) no connection is being initiated. I can see on the remote site (server) that no UDP packet arrives. I manually tested sending a UDP packet from my home pfSense and this arrives and is visible on the remote site. Looks like the wg process does not even try to connect or hangs somewhere in between.

Any suggestions where to troubleshoot? As wg being a silent tool, I dont see any error messages..

sLy1337

I checked further and found a message in the logs basically saying that the route for my S2S Gateway was changed.

/usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 8.8.8.8 and adding a new route through 10.1.1.1

10.1.1.1 is another gateway (LTE), not the primary (should be the gateway group..)

I checked the gateway and activated the "Disable Gateway Monitoring Action" checkmark. And now its working again... but does this make any sense? In any case, just wanted to let you know, so maybe someone faces the same issue.

robearded

@sly1337 as you pointed out, the wireguard package is pretty silent and does not output logs, which unfortunately makes it hard for us consumers to see where the problem is (maybe there are logs somewhere in a file, in a location that the developers knows, but I'm not aware of that).

The only thing that I found is by downgrading the wireguard-kmod package to an older version, the handshake and connection are made. I've added the steps on how to downgrade that package in an earlier post from this thread (or direct link: https://forum.netgate.com/post/1010944 ). This is the only (easy) solution I've found to fix this problem, I've been running this since I've found it (3 weeks ago) and so far there is no problem. I'm constantly using the wireguard connection (I use it as a Gateway for some of my servers) and had no downtime since.

sLy1337

@robearded Yep, I know. Actually I used the older version before, however did not work in my case. As mentioned, it worked when enabling "Disable Gateway Monitoring Action" in the gateway. Thanks anyways!

xxGBHxx

@sly1337 said in Wireguard suddenly refuses to handshake:

I checked further and found a message in the logs basically saying that the route for my S2S Gateway was changed.
/usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 8.8.8.8 and adding a new route through 10.1.1.1 
10.1.1.1 is another gateway (LTE), not the primary (should be the gateway group..)

I checked the gateway and activated the "Disable Gateway Monitoring Action" checkmark. And now its working again... but does this make any sense? In any case, just wanted to let you know, so maybe someone faces the same issue.

I just tried that and it made no difference to my setup.

Before that (but after the problem appeared) I did a reboot of both my ESXi host (and updated it) and of the PfSesnse server. Since then the OpenVPN tunnel and one of the WG tunnels came back up exactly as they were before. The other is permanently down and has never come back up. I'm hoping @cmcdonald will be able to get me to do any troubleshooting while it's in this state as having 1 WG tunnel down and the other up and working fine seems like a fairly good opportunity to test things.

Ultimately though this is happening enough to be an "issue" and it's not an issue I have with any of the VPN vendors own WG stacks on my mobile or my desktop. I do need to find the time to set up FreeBSD but I'm not convinced that's really going to help as I can't actually replicate when, or if, it's going to happen.

robearded

After I had to restart my modem (so the WAN went down for some minutes on the pfsense) this problem happened to me again. I can't seem to be able to initiate the handshake whatever I do. I also checked "Disable Gateway Monitoring Action" checkbox for the wireguard gateway, however it's still not working

xxGBHxx

@robearded So as of 3h ago my other WG connection has gone down for no reason again and now I can't get either of the tunnels up.

This is so infuriating :(

I can't believe it's just a handful of us.

G