Wireguard suddenly refuses to handshake

xxGBHxx · Nov 17, 2021, 6:32 PM

@ciscox Yes everything is running the very latest.

There is something very broken somewhere.

So as I've been here before I took a snapshot of the FW as I'm running it in VMWare. When I couldn't get anything working I went back to the fully working and perfectly ok snapshot and now THAT isn't working either.

I had considered re-installing the package but it becomes problematic as without the Wireguard tunnel on the Firewall I have absolutely zero connectivity. While I can simply enable that for the LAN I'm struggling to enable the firewall itself to talk out.

I am wondering if it's the VMX3 interfaces I'm using in VMWare that are just breaking with pfSense. I know I've had some issues with the way the virtual interfaces on VMWare interact with pfSense but I'm starting to wonder if it may be interaction with the type of virtual interface I'm using. That said in their VMWare guide they state you can use either type of interface.

I will try to re-install it.

xxGBHxx · Nov 17, 2021, 8:43 PM

So finally managed to get the firewall connected by setting up a new OpenVPN connection and using that. I was then able to re-install and bang, instant connection with both VPN's

I have no idea why it's done it twice now but at least I didn't need to go through the pain of re-installing again. Thanks for the kick to go do it.

Doesn't in any way explain why my snapshotted VM with a working config ALSO didn't work.

I'm sure there's issues there somewhere. Guess I'm never going to find out what it is.

G

robearded · Nov 18, 2021, 6:40 PM

I have the exact problem, last handshake is 15 hours ago (and that is kind of the same time I pressed the update button). My pfsense is connected through wireguard to a VPS for a "reverse proxy" like setup (using the wireguard connection as a WAN). Pfsense doesn't want to connect to the server anymore since the wireguard package upgrade.

Unfortunately, I can't use another VPN for this (like OpenVPN) because it can't handle the amount of bandwidth I'm running through Wireguard.

I can connect from my laptop to a wireguard server running on the pfsense, however pfsense doesn't want to connect to another remote.

robearded · Nov 18, 2021, 8:21 PM

Sorry for the double post, but I'm not able to edit my post anymore.
I've found the solution of this problem.

SOLUTION

The problem seems to be with the latest version of the package "wireguard-kmod" (0-0.0.20211105) that the wireguard package depends on. By manually installing an older version of wireguard-kmod, the problem is fixed.

Steps to downgrade:

Download "wireguard-kmod" version 0.0.20210606_1 from here: https://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/All/wireguard-kmod-0.0.20210606_1.txz
Go to your pfsense dashboard -> Diagnostics -> Command prompt.
On the Upload File option, select your downloaded wireguard-kmod package and upload it. After upload is finished, you should receive a message that your file is uploaded in /tmp/wireguard-kmod-0.0.20210606_1.txz
On the Execute Shell Command option (it's located on the same page) run this command: pkg install -y /tmp/wireguard-kmod-0.0.20210606_1.txz

Let the command finish execution and you should see an output saying package installed successful. Restart your wireguard service and it should be able to connect to remotes again.

xxGBHxx · Oct 6, 2022, 3:13 AM

@robearded said in Wireguard suddenly refuses to handshake:

Sorry for the double post, but I'm not able to edit my post anymore.
I've found the solution of this problem.

SOLUTION

The problem seems to be with the latest version of the package "wireguard-kmod" (0-0.0.20211105) that the wireguard package depends on. By manually installing an older version of wireguard-kmod, the problem is fixed.

Steps to downgrade:

Download "wireguard-kmod" version 0.0.20210606_1 from here: https://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/All/wireguard-kmod-0.0.20210606_1.txz

Go to your pfsense dashboard -> Diagnostics -> Command prompt.

On the Upload File option, select your downloaded wireguard-kmod package and upload it. After upload is finished, you should receive a message that your file is uploaded in /tmp/wireguard-kmod-0.0.20210606_1.txz

On the Execute Shell Command option (it's located on the same page) run this command: pkg install -y /tmp/wireguard-kmod-0.0.20210606_1.txz

Let the command finish execution and you should see an output saying package installed successful. Restart your wireguard service and it should be able to connect to remotes again.

Nice work!

That could have saved me a massive amount of hassle a few weeks ago.

G

cmcdonald · Nov 19, 2021, 4:29 PM

@xxgbhxx What VPN providers are you having issues with?

xxGBHxx · Nov 19, 2021, 4:56 PM

@cmcdonald I have been using IVPN.net and WeVPN.

IVPN.net provides a specific setup guide for pfSense and Wireguard which is what I followed. This worked perfectly fine on both the old Wireguard (previous guide) and the new Wireguard (new guide) implementations until things just randomly stop working.

I started using WeVPN in the last few weeks in order to prove to myself it wasn't IVPN.net (or me!) causing the issues. Set it up in about 2 mins, worked instantly and has remained stable until I upgraded Wireguard as above.

Both vendors provide apps for their VPN and I've used the Android and Windows apps on Wireguard and OpenVPN. Both work flawlessly for both my ISP's every single time and have never failed.

I have also had IVPN.net verify the servers at their end that there were not any strange/dead connections and they could see none at all.

G

cmcdonald · Nov 21, 2021, 3:37 PM

@xxgbhxx Just tested with IVPN.net and had no issues.

Are you sure you are using the correct endpoint port?

As per https://www.ivpn.net/setup/router/pfsense-wireguard/

Endpoint Port: Choose one of 53, 2049, 2050, 30587, 41893, 48574, or 58237, all are equally secure

I saw no handshakes until I read read the manual and chose one of the above ports.

22.01-DEVELOPMENT][admin@pfSense.home.arpa]/root: wg
interface: tun_wg0
  public key: qck10oARb58oS+0owfGFRK4K8tpxVPZxLTw5nccb9go=
  private key: (hidden)
  listening port: 51820

peer: ANhVUMAQgStPVNRHW8mg0ZtN1YI1QHyXfNCO8+USNQQ=
  endpoint: 72.11.137.148:53
  allowed ips: ::/0, 0.0.0.0/0
  latest handshake: 1 minute, 19 seconds ago
  transfer: 184 B received, 10.08 KiB sent
  persistent keepalive: every 30 seconds
[22.01-DEVELOPMENT][admin@pfSense.home.arpa]/root:

xxGBHxx · Nov 21, 2021, 9:06 PM

@cmcdonald I've had it working many times on multiple different ports to multiple different peers. When it works, it works.

When it fails however it seems to get into a lock and nothing short of re-installing the entire FW is sure to fix it. I wish there was more troubleshooting I could do to help.

When it happened the last time I tried to do some packet captures but all that showed was the packets getting sent by the FW but never being answered. Didn't matter what port, what peer, what key the result was the same.

This is NOT just limited to IVPN as it affected WeVPN the last time in exactly the same way.

I think the best I can do if you really want to help is that I try and get it to do it again. When it's happening then I can run any test you like.

cmcdonald · Nov 22, 2021, 3:02 PM

@xxgbhxx That is very strange. Reinstalling the firewall is a pretty heavy-handed solution. If you're confident that the issue is with pfSense (and thus FreeBSD), I would suggest standing up a FreeBSD VM based on the same version of pfSense and configuring a tunnel there. If the issue is still present on vanilla FreeBSD, then that is a good indication we are looking at an issue related to the kernel code and not something higher up the stack.

xxGBHxx · Dec 2, 2021, 11:14 PM

@cmcdonald Sorry been a week or so due to work and running out of space on my VMWare server. Not yet managed to get FreeBSD installed as I only sorted it out today.

In the interim I re-setup the two VPN's again and they've both been up and working for the past week or so without an issue. They have re-connected through ISP drops but for the past 18 hours the IVPN VPN has failed to handshake. I'm not touching it again to tryand getit to re-connect. The other WG VPN is up and working without an issue.

So anything you'd like me to do to test or anything I should do before I try and reconnect it?

G

sLy1337 · Dec 9, 2021, 8:53 AM

@cmcdonald @xxGBHxx I do face the same issue here with latest WG package:

pfSense-pkg-WireGuard 	0.1.5_3 	pfSense package WireGuard (EXPERIMENTAL)
wireguard-kmod 	0.0.20211105 	WireGuard implementation for the FreeBSD kernel
wireguard-tools-lite 	1.0.20210914_1 	Fast, modern and secure VPN Tunnel (lite flavor)

I have created a S2S tunnel via wireguard. This worked like 14d without any issues. One site regularly reconnects (VDSL). However, today the connection dropped and no handshake could be established.

From my home (site1, client) no connection is being initiated. I can see on the remote site (server) that no UDP packet arrives. I manually tested sending a UDP packet from my home pfSense and this arrives and is visible on the remote site. Looks like the wg process does not even try to connect or hangs somewhere in between.

Any suggestions where to troubleshoot? As wg being a silent tool, I dont see any error messages..

sLy1337 · Dec 9, 2021, 9:29 AM

I checked further and found a message in the logs basically saying that the route for my S2S Gateway was changed.

/usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 8.8.8.8 and adding a new route through 10.1.1.1

10.1.1.1 is another gateway (LTE), not the primary (should be the gateway group..)

I checked the gateway and activated the "Disable Gateway Monitoring Action" checkmark. And now its working again... but does this make any sense? In any case, just wanted to let you know, so maybe someone faces the same issue.

robearded · Dec 9, 2021, 9:43 AM

@sly1337 as you pointed out, the wireguard package is pretty silent and does not output logs, which unfortunately makes it hard for us consumers to see where the problem is (maybe there are logs somewhere in a file, in a location that the developers knows, but I'm not aware of that).

The only thing that I found is by downgrading the wireguard-kmod package to an older version, the handshake and connection are made. I've added the steps on how to downgrade that package in an earlier post from this thread (or direct link: https://forum.netgate.com/post/1010944 ). This is the only (easy) solution I've found to fix this problem, I've been running this since I've found it (3 weeks ago) and so far there is no problem. I'm constantly using the wireguard connection (I use it as a Gateway for some of my servers) and had no downtime since.

sLy1337 · Dec 9, 2021, 2:45 PM

@robearded Yep, I know. Actually I used the older version before, however did not work in my case. As mentioned, it worked when enabling "Disable Gateway Monitoring Action" in the gateway. Thanks anyways!

xxGBHxx · Dec 10, 2021, 11:08 AM

@sly1337 said in Wireguard suddenly refuses to handshake:

I checked further and found a message in the logs basically saying that the route for my S2S Gateway was changed.
/usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 8.8.8.8 and adding a new route through 10.1.1.1 
10.1.1.1 is another gateway (LTE), not the primary (should be the gateway group..)

I checked the gateway and activated the "Disable Gateway Monitoring Action" checkmark. And now its working again... but does this make any sense? In any case, just wanted to let you know, so maybe someone faces the same issue.

I just tried that and it made no difference to my setup.

Before that (but after the problem appeared) I did a reboot of both my ESXi host (and updated it) and of the PfSesnse server. Since then the OpenVPN tunnel and one of the WG tunnels came back up exactly as they were before. The other is permanently down and has never come back up. I'm hoping @cmcdonald will be able to get me to do any troubleshooting while it's in this state as having 1 WG tunnel down and the other up and working fine seems like a fairly good opportunity to test things.

Ultimately though this is happening enough to be an "issue" and it's not an issue I have with any of the VPN vendors own WG stacks on my mobile or my desktop. I do need to find the time to set up FreeBSD but I'm not convinced that's really going to help as I can't actually replicate when, or if, it's going to happen.

robearded · Dec 12, 2021, 6:57 PM

After I had to restart my modem (so the WAN went down for some minutes on the pfsense) this problem happened to me again. I can't seem to be able to initiate the handshake whatever I do. I also checked "Disable Gateway Monitoring Action" checkbox for the wireguard gateway, however it's still not working

xxGBHxx · Dec 21, 2021, 2:11 PM

@robearded So as of 3h ago my other WG connection has gone down for no reason again and now I can't get either of the tunnels up.

This is so infuriating :(

I can't believe it's just a handful of us.

G

cmcdonald · Dec 21, 2021, 2:12 PM

Hey all,

Sorry you guys are having issues with handshakes.

I do have an idea that might help illuminate where the problem lies, basically creating the tunnel from barebones at the shell and bypassing the GUI.

Here are the steps (adjust addresses, names, etc. as necessary for your situation):

Disable "WireGuard" in WireGuard > Settings. This keeps the daemon from starting. This is important because we are going to build the tunnel up manually at the shell. Though you can still create your tunnel and peer config as normal, just don't start the service.
Login to pfSense shell via SSH or console cable.
Use these commands to create the interface, assign addresses, add to interface group (if necessary) and sync WireGuard conf:

This command creates a interface of type wg with name tun_wg0:

$ ifconfig wg create name tun_wg0

This command assigns an IPv4 inet address of 10.11.12.1/24 on tun_wg0:

$ ifconfig tun_wg0 inet 10.11.12.1/24

This command assigns tun_wg0 to interface group WireGuard:

$ ifconfig tun_wg0 group WireGuard

This command syncs tun_wg0 configuration using WireGuard userland tools wg :

$ wg syncconf tun_wg0 /usr/local/etc/wireguard/tun_wg0.conf

Note: /usr/local/etc/wireguard/tun_wg0.conf can be generated by the package, just keep the service disabled for the time-being.

This command brings up tun_wg0 administratively:

$ ifconfig tun_wg0 up

At this point you should have a WireGuard tunnel interface built manually. You can now proceed with assigning it to pfSense, etc.

This test is useful because it bypasses all WireGuard package semantics and only uses pfSense core logic...useful for isolating potential issues.

cmcdonald · Dec 22, 2021, 5:54 PM

Youtube Video

This video walks through several scenarios that should be applicable here, hopefully this helps!

This video is more aimed at discussing the various ways I test and live with WireGuard on a daily basis, but I walk through the setup and configuration of each scenario.