Wireguard suddenly refuses to handshake

cmcdonald

@xxgbhxx Just tested with IVPN.net and had no issues.

Are you sure you are using the correct endpoint port?

As per https://www.ivpn.net/setup/router/pfsense-wireguard/

Endpoint Port: Choose one of 53, 2049, 2050, 30587, 41893, 48574, or 58237, all are equally secure

I saw no handshakes until I read read the manual and chose one of the above ports.

22.01-DEVELOPMENT][admin@pfSense.home.arpa]/root: wg
interface: tun_wg0
  public key: qck10oARb58oS+0owfGFRK4K8tpxVPZxLTw5nccb9go=
  private key: (hidden)
  listening port: 51820

peer: ANhVUMAQgStPVNRHW8mg0ZtN1YI1QHyXfNCO8+USNQQ=
  endpoint: 72.11.137.148:53
  allowed ips: ::/0, 0.0.0.0/0
  latest handshake: 1 minute, 19 seconds ago
  transfer: 184 B received, 10.08 KiB sent
  persistent keepalive: every 30 seconds
[22.01-DEVELOPMENT][admin@pfSense.home.arpa]/root:

xxGBHxx

@cmcdonald I've had it working many times on multiple different ports to multiple different peers. When it works, it works.

When it fails however it seems to get into a lock and nothing short of re-installing the entire FW is sure to fix it. I wish there was more troubleshooting I could do to help.

When it happened the last time I tried to do some packet captures but all that showed was the packets getting sent by the FW but never being answered. Didn't matter what port, what peer, what key the result was the same.

This is NOT just limited to IVPN as it affected WeVPN the last time in exactly the same way.

I think the best I can do if you really want to help is that I try and get it to do it again. When it's happening then I can run any test you like.

cmcdonald

@xxgbhxx That is very strange. Reinstalling the firewall is a pretty heavy-handed solution. If you're confident that the issue is with pfSense (and thus FreeBSD), I would suggest standing up a FreeBSD VM based on the same version of pfSense and configuring a tunnel there. If the issue is still present on vanilla FreeBSD, then that is a good indication we are looking at an issue related to the kernel code and not something higher up the stack.

xxGBHxx

@cmcdonald Sorry been a week or so due to work and running out of space on my VMWare server. Not yet managed to get FreeBSD installed as I only sorted it out today.

In the interim I re-setup the two VPN's again and they've both been up and working for the past week or so without an issue. They have re-connected through ISP drops but for the past 18 hours the IVPN VPN has failed to handshake. I'm not touching it again to tryand getit to re-connect. The other WG VPN is up and working without an issue.

So anything you'd like me to do to test or anything I should do before I try and reconnect it?

G

sLy1337

@cmcdonald @xxGBHxx I do face the same issue here with latest WG package:

pfSense-pkg-WireGuard 	0.1.5_3 	pfSense package WireGuard (EXPERIMENTAL)
wireguard-kmod 	0.0.20211105 	WireGuard implementation for the FreeBSD kernel
wireguard-tools-lite 	1.0.20210914_1 	Fast, modern and secure VPN Tunnel (lite flavor)

I have created a S2S tunnel via wireguard. This worked like 14d without any issues. One site regularly reconnects (VDSL). However, today the connection dropped and no handshake could be established.

From my home (site1, client) no connection is being initiated. I can see on the remote site (server) that no UDP packet arrives. I manually tested sending a UDP packet from my home pfSense and this arrives and is visible on the remote site. Looks like the wg process does not even try to connect or hangs somewhere in between.

Any suggestions where to troubleshoot? As wg being a silent tool, I dont see any error messages..

sLy1337

I checked further and found a message in the logs basically saying that the route for my S2S Gateway was changed.

/usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 8.8.8.8 and adding a new route through 10.1.1.1

10.1.1.1 is another gateway (LTE), not the primary (should be the gateway group..)

I checked the gateway and activated the "Disable Gateway Monitoring Action" checkmark. And now its working again... but does this make any sense? In any case, just wanted to let you know, so maybe someone faces the same issue.

robearded

@sly1337 as you pointed out, the wireguard package is pretty silent and does not output logs, which unfortunately makes it hard for us consumers to see where the problem is (maybe there are logs somewhere in a file, in a location that the developers knows, but I'm not aware of that).

The only thing that I found is by downgrading the wireguard-kmod package to an older version, the handshake and connection are made. I've added the steps on how to downgrade that package in an earlier post from this thread (or direct link: https://forum.netgate.com/post/1010944 ). This is the only (easy) solution I've found to fix this problem, I've been running this since I've found it (3 weeks ago) and so far there is no problem. I'm constantly using the wireguard connection (I use it as a Gateway for some of my servers) and had no downtime since.

sLy1337

@robearded Yep, I know. Actually I used the older version before, however did not work in my case. As mentioned, it worked when enabling "Disable Gateway Monitoring Action" in the gateway. Thanks anyways!

xxGBHxx

@sly1337 said in Wireguard suddenly refuses to handshake:

I checked further and found a message in the logs basically saying that the route for my S2S Gateway was changed.
/usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 8.8.8.8 and adding a new route through 10.1.1.1 
10.1.1.1 is another gateway (LTE), not the primary (should be the gateway group..)

I checked the gateway and activated the "Disable Gateway Monitoring Action" checkmark. And now its working again... but does this make any sense? In any case, just wanted to let you know, so maybe someone faces the same issue.

I just tried that and it made no difference to my setup.

Before that (but after the problem appeared) I did a reboot of both my ESXi host (and updated it) and of the PfSesnse server. Since then the OpenVPN tunnel and one of the WG tunnels came back up exactly as they were before. The other is permanently down and has never come back up. I'm hoping @cmcdonald will be able to get me to do any troubleshooting while it's in this state as having 1 WG tunnel down and the other up and working fine seems like a fairly good opportunity to test things.

Ultimately though this is happening enough to be an "issue" and it's not an issue I have with any of the VPN vendors own WG stacks on my mobile or my desktop. I do need to find the time to set up FreeBSD but I'm not convinced that's really going to help as I can't actually replicate when, or if, it's going to happen.

robearded

After I had to restart my modem (so the WAN went down for some minutes on the pfsense) this problem happened to me again. I can't seem to be able to initiate the handshake whatever I do. I also checked "Disable Gateway Monitoring Action" checkbox for the wireguard gateway, however it's still not working

xxGBHxx

@robearded So as of 3h ago my other WG connection has gone down for no reason again and now I can't get either of the tunnels up.

This is so infuriating :(

I can't believe it's just a handful of us.

G

cmcdonald

Hey all,

Sorry you guys are having issues with handshakes.

I do have an idea that might help illuminate where the problem lies, basically creating the tunnel from barebones at the shell and bypassing the GUI.

Here are the steps (adjust addresses, names, etc. as necessary for your situation):

Disable "WireGuard" in WireGuard > Settings. This keeps the daemon from starting. This is important because we are going to build the tunnel up manually at the shell. Though you can still create your tunnel and peer config as normal, just don't start the service.
Login to pfSense shell via SSH or console cable.
Use these commands to create the interface, assign addresses, add to interface group (if necessary) and sync WireGuard conf:

This command creates a interface of type wg with name tun_wg0:

$ ifconfig wg create name tun_wg0

This command assigns an IPv4 inet address of 10.11.12.1/24 on tun_wg0:

$ ifconfig tun_wg0 inet 10.11.12.1/24

This command assigns tun_wg0 to interface group WireGuard:

$ ifconfig tun_wg0 group WireGuard

This command syncs tun_wg0 configuration using WireGuard userland tools wg :

$ wg syncconf tun_wg0 /usr/local/etc/wireguard/tun_wg0.conf

Note: /usr/local/etc/wireguard/tun_wg0.conf can be generated by the package, just keep the service disabled for the time-being.

This command brings up tun_wg0 administratively:

$ ifconfig tun_wg0 up

At this point you should have a WireGuard tunnel interface built manually. You can now proceed with assigning it to pfSense, etc.

This test is useful because it bypasses all WireGuard package semantics and only uses pfSense core logic...useful for isolating potential issues.

cmcdonald

Youtube Video

This video walks through several scenarios that should be applicable here, hopefully this helps!

This video is more aimed at discussing the various ways I test and live with WireGuard on a daily basis, but I walk through the setup and configuration of each scenario.

robearded

@robearded said in Wireguard suddenly refuses to handshake:

After I had to restart my modem (so the WAN went down for some minutes on the pfsense) this problem happened to me again. I can't seem to be able to initiate the handshake whatever I do. I also checked "Disable Gateway Monitoring Action" checkbox for the wireguard gateway, however it's still not working

It turns out this happened because of a change I did on the wireguard server without realizing it will break stuff. I'm not however sure if I did this change before the first occurrence of this problem, from what I remember it is after, but it is definitely possible to be before.
I had some DNAT iptables rules in the PostUp and PostDown config of wireguard to forward some ports to pfsense, so I can use the public IP of the wg server as a public IP for the server running pfsense. I figured out that it would be easier to DNAT all ports (except SSH) and just do the firewalling inside pfsense, so I don't have to log in to the wg server everything I want to forward another port, and I can do it all from pfsense. Well, the problem is, that I didn't realized I also forwarded the port WG was listening on.

Woodsomeister

Hello,

i'm facing the same problem with the error status of

/usr/local/pkg/wireguard/includes/wg_service.inc: Static Routes: Gateway IP could not be found ...

No handshake seems to happen anymore. Reinstalled the wireguard package several times, and also tried a new version of the kmod implementation. No success.

Jarhead

@woodsomeister Post pictures of your wireguard tunnel, peers, interface and routes/gateway.

Woodsomeister

This post is deleted!

Woodsomeister

@jarhead

Routes were added for the marked Wireguard interface and i also configured the interface using the right IPv4 static IP (10.0.41.1, peer is 10.0.41.2). (no screenshot)
Unfortunatly i had to switched back to IPSec, so that's why everything is deactivated.

Jarhead

@woodsomeister You have to add the tunnel as an allowed IP.
Did this ever work?

Woodsomeister

@jarhead For several weeks. In the other side is also a Opnsense under my control. Everything was working as expected, but after a restart some days ago i went into this problem.