Second peer not passing traffic

myms · Jan 23, 2021, 6:07 AM

Hey folks – so excited to see WireGuard in pfSense! Thanks to Netgate for sponsoring the work to get the kernel port in.

Testing it now and having trouble with my second peer passing traffic. First peer works fine. What information can I provide to help debug?

stephenw10 · Jan 23, 2021, 1:56 PM

How are they connected? What are they configured to pass?

The actual actual wg.conf files from /etc/wg/wg0.conf would be best but you'd want to redact at least the private keys unless this is an internal test setup only.

Steve

wingsman2 · Jan 23, 2021, 7:42 PM

I am also experiencing this issue and I have found that when you add a second peer wireguard seems to break. On cli with one peer if you run "wg" it pops out realtive info for your config but as soon as you add a second peer the "wg" command throws "Unable to access interface: Cannot allocate memory." Config file looks good between having one to two peers. I can confirm that it has plenty of memory. Also the first peer which is configured identically, aside from the assigned IP, is still able to route traffic perfectly fine.

login-to-view

stephenw10 · Jan 23, 2021, 9:04 PM

It looks like you don't have an endpoint defined for that one peer we can see. Which seems invalid, no idea how it would try to connect there.
If neither peer has as endpoint set I could imagine it just failing out.

stephenw10 · Jan 23, 2021, 9:23 PM

Yes, I'm unable to replicate that with endpoints defined:

[21.02-DEVELOPMENT][admin@5100.stevew.lan]/root: wg
interface: wg0
  public key: xxxxx=
  private key: (hidden)
  listening port: 51820

peer: xxxxx=
  endpoint: x.x.x.105:51820
  allowed ips: 172.28.20.0/24

peer: xxxxx=
  endpoint: x.x.x.5:51820
  allowed ips: 172.30.2.0/24, 0.0.0.0/0

wingsman2 · Jan 23, 2021, 9:51 PM

The endpoints are blank because they are the clients that are mobile and have no static IP so they need to be dynamic. The peer you can see works perfectly fine. It's when I add another peer like it when it throws that error.

On the client I use the persistent keepalive setting to keep the tunnel up and functioning when client is running. This would allow my client to connect to my vpn even behind nats.

wingsman2 · Jan 23, 2021, 9:43 PM

# Peer: phone
[Peer]
PublicKey = xxxx=
AllowedIPs = 10.30.x.0/24, 192.168.x.0/24

# Peer: phone2
[Peer]
PublicKey = xxxx=
AllowedIPs = 10.30.x.0/24, 192.168.x.0/24

stephenw10 · Jan 23, 2021, 9:50 PM

Ah, OK I see.

Let me update and test here....

jimp · Jan 26, 2021, 3:38 PM

It's possible something was fixed here, but I can't replicate this behavior.

I added a second tunnel configured like the first and wg outputs the config from both.

: wg
interface: wg1
  public key: LMtMofK78bDgsPgkjJRr/8zkHTFV/ZGVVVkBV9/wKV4=
  private key: (hidden)
  listening port: 51821

peer: efIIIX7tjFdFhWVsPlgQHJcYDgidNjFllpS12+9D2EU=
  allowed ips: 10.6.211.4/32

peer: wtvPLZrlCvB+dvxkwbtAkmnSqCRXmC9+14LE/SvktSc=
  allowed ips: 10.6.211.3/32

peer: KN2+TVoXjngczbcDgwdUG6C4T9tJz4k1dgr92Zf0YUc=
  allowed ips: 10.6.211.2/32

interface: wg0
  public key: PUVBJ+zuz/0mRPEB4tIaVbet5NzVwdWMX7crGx+/wDs=
  private key: (hidden)
  listening port: 51820

peer: V0qTImT7Vdr0eaZTAUg8D/sXP3Q6zGWRBa8+sSIHISw=
  allowed ips: 10.17.210.4/32

peer: bfad7zPslGzPJtSqf+SRS4+0KSiX3gf6g1KsYeiSPUE=
  allowed ips: 10.17.210.3/32

peer: 71nJC+0wt52pWSUpg/xYCCK3XbMcfanZ8UlDa4HZQQc=
  allowed ips: 10.6.210.2/32

Also, FYI, those are public keys, so not worth redacting, even if they weren't throwaways from my lab.

wingsman2 · Jan 26, 2021, 3:51 PM

This post is deleted!

jimp · Jan 26, 2021, 3:58 PM

My configs for reference:

: pkg info -E pfSense
pfSense-2.5.0.a.20210125.2350

: cat /etc/wg/wg0.conf
# This WireGuard config file has been created automatically. Do not edit!
# Description: Remote Access

[Interface]
PrivateKey = <blah>
ListenPort = 51820

# Peer: test
[Peer]
PublicKey = 71nJC+0wt52pWSUpg/xYCCK3XbMcfanZ8UlDa4HZQQc=
AllowedIPs = 10.6.210.2/32

# Peer: someone
[Peer]
PublicKey = bfad7zPslGzPJtSqf+SRS4+0KSiX3gf6g1KsYeiSPUE=
AllowedIPs = 10.6.210.3/32

# Peer: another
[Peer]
PublicKey = V0qTImT7Vdr0eaZTAUg8D/sXP3Q6zGWRBa8+sSIHISw=
AllowedIPs = 10.6.210.4/32

: cat /etc/wg/wg1.conf
# This WireGuard config file has been created automatically. Do not edit!
# Description: Second RA

[Interface]
PrivateKey = <blah again>
ListenPort = 51821

# Peer: alice
[Peer]
PublicKey = KN2+TVoXjngczbcDgwdUG6C4T9tJz4k1dgr92Zf0YUc=
AllowedIPs = 10.6.211.2/32

# Peer: bob
[Peer]
PublicKey = wtvPLZrlCvB+dvxkwbtAkmnSqCRXmC9+14LE/SvktSc=
AllowedIPs = 10.6.211.3/32

# Peer: sue
[Peer]
PublicKey = efIIIX7tjFdFhWVsPlgQHJcYDgidNjFllpS12+9D2EU=
AllowedIPs = 10.6.211.4/32

wingsman2 · Jan 26, 2021, 5:09 PM

I don't know how I reach this but I just reformatted and reinstalled a fresh pfSense VM. Configured everything to get it working as expected and I still run into it giving me that error when I run the wg command.

[2.5.0-DEVELOPMENT][admin@localrooter]/root: cat /etc/wg/wg0.conf
# This WireGuard config file has been created automatically. Do not edit!
# Description: WGVPN

[Interface]
PrivateKey = xxxxxxxx=
ListenPort = 51820

# Peer:
[Peer]
PublicKey = xxxxxx=
AllowedIPs = 10.30.x.0/24, 192.168.x.0/24

# Peer:
[Peer]
PublicKey = xxxxx=
AllowedIPs = 10.30.x.0/24, 192.168.x.0/24

[2.5.0-DEVELOPMENT][admin@localrooter]/root: wg
Unable to access interface wg0: Cannot allocate memory
[2.5.0-DEVELOPMENT][admin@localrooter]/root:

I have assigned the wg0 interface with OPT1. I do have some static information I have configured for gateways and outbound NAT but that shouldn't be effecting this.

wingsman2 · Jan 26, 2021, 5:13 PM

I can also confirm the VM does have plenty of memory to play with

last pid: 33282;  load averages:  0.83,  0.60,  0.54                                                                                                                                                                 up 0+00:40:15  12:04:50
54 processes:  1 running, 53 sleeping
CPU:  1.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.0% idle
Mem: 60M Active, 7944K Inact, 334M Wired, 86M Buf, 3513M Free
Swap: 1024M Total, 1024M Free

Ritte · Jan 26, 2021, 5:29 PM

I think I've identified the problem. I had same problem as OP.
My problem was that in PFsense -> WireGuard -> Peers I had assigned 0.0.0.0/0 as "Allowed IPs". Which you make think based on the description in pfsense:

"List of CIDR-masked subnets which can be reached via this peer."

I shared 0.0.0.0/0 both in the wireguard iOS-app and PFSense to experiment with a roadwarrior VPN.

Then I saw @jimp's example and quickly realized that his AllowedIPs (for the server!) are actually the tunnel IP to clients.

So in my case I assigned 192.168.18.2/32 to my iphone and 192.168.18.3/32 to my ipad (the thunnel has 192.168.18.0/24). In the Wireguard IOS-app, I use the same adress under "Adresses" in the interface.

In the peer-section of the client (iOS-app for me), I add the networks I want to reach via pfsense (for a split VPN):
192.168.1.0/24, 192.168.10.0/24 etc or 0.0.0.0/0 if I want to have a roadwarrior-VPN.

I guess the reasoning that "allowedIPs" in the server configuration is telling the wireguard server how to reach the client, but for the client it's which traffict to send over VPN.

This is my old non-working config:

# Description:
[Interface]
PrivateKey = RedactedJustBecause
ListenPort = 51820

# Peer: ios-iphone
[Peer]
PublicKey = RedactedJustBecause
AllowedIPs = 0.0.0.0/0

# Peer: IOS-ipad
[Peer]
PublicKey = RedactedJustBecause
AllowedIPs = 0.0.0.0/0

The working config looks like this:

# Description:

[Interface]
PrivateKey = RedactedJustBecause

# Peer: ios-iphone
[Peer]
PublicKey = RedactedJustBecause
AllowedIPs = 192.168.18.2/32

# Peer: IOS-ipad
[Peer]
PublicKey = RedactedJustBecause
AllowedIPs = 192.168.18.3/32

Thanks for the good work by integrating Wireguard! I'll experiment with it and give feedback.

jimp · Jan 26, 2021, 5:23 PM

That's right you can't have the same Allowed IPs on two peers of the same VPN, otherwise it has no idea where to send the traffic. It uses the combination of the peer key and Allowed IPs to figure out how to route things internally inside WireGuard.

Also there is no "client" or "server" with WireGuard -- each side has peers, and the "Allowed IPs" on the peer are the networks on the peer side which can be reached via that peer.

So as you found, duplicating the same value across multiple peers breaks. We may need some input validation to prevent that from being allowed, but it could get tricky.

wingsman2 · Jan 26, 2021, 5:25 PM

Taking @ritte approach I believe he may have gotten it. I was assuming the allowedIPs was what the server allowed the client to access, not what the server see's in the peer. I adjusted my config to match this new info and everything seems to be working correctly. wg command is not throwing out any errors. Will continue to test.

Ritte · Jan 26, 2021, 5:48 PM

@jimp

Is the allowedIPs treated as internal "endpoint" for a mobile peer (when setting it in PFSENSE)? I guess when doing site-site VPN this "problem" should not be faced since an external endpoint exists.

stephenw10 · Jan 26, 2021, 5:45 PM

It doesn't include the external public IP if that's what you mean, no.

It would include the IP in the 'tunnel subnet' that mobile peer had been given.

Steve

Ritte · Jan 26, 2021, 5:56 PM

@stephenw10
Yes, I realized this was quickly written. But when you have an external IP and an Endpoint, then this should not be an issue. PFsense peer knows how to route the traffic to Site B Peer.

But with mobile peers, pfsense peer needs to have allowedIPs as Internal IPs to properly route the traffic.

Have I understood it correctly?

stephenw10 · Jan 26, 2021, 6:50 PM

Yes. With multiple peers you need to set Allowed-IPs to determine which peer WG routes to.
https://www.wireguard.com/#cryptokey-routing

But to avoid confusion 'Endpoint' is a WG term the defines the external IP.

Steve