@prudentcircle said in Does WireGuard work in a High Availability (pfsync, "CARP") mirrored firewall environment?:

Can the same logic be applied to Wireguard?

IPsec or OpenVPN are doing that, if a node is standby and it's configured on a CARP IP. But as Wireguard does not have an interface binding it's a bit more complicated.

@prudentcircle said in Does WireGuard work in a High Availability (pfsync, "CARP") mirrored firewall environment?:

If CARP is capable of judging who is active and who is standby, can this be used as a signal for where to run one instance of Wireguard and kill all the other Wireguard processes in the cluster?

Perhaps - I'm not sure.

@prudentcircle said in Does WireGuard work in a High Availability (pfsync, "CARP") mirrored firewall environment?:

IPSec and OpenVPN could tell who is active and who is standby because they are bound to interfaces, right? So those two VPN protocols can form a cluster without relying on something like CARP because they are bound to interfaces, right?

To the first part: yes. To the second: I don't understand what you mean by forming a cluster without relying on CARP etc. A cluster is a cluster because of things like CARP, keepalived or stuff. What do you mean by "form a cluster without relying on sth like CARP"?

@prudentcircle said in Does WireGuard work in a High Availability (pfsync, "CARP") mirrored firewall environment?:

Then what about making Wireguard dependent on CARP and run only single instance of Wireguard where CARP status is confirmed to be active?

Wouldn't change a thing as the problem with Wireguard still remains, that it is interface-agnostic and doesn't bind to the VIP (virtual IP) of a cluster. You simply don't want Wireguard to use your interface IP instead of the cluster IP as your communication would always come from the wrong IP and you can't that easily set it up to work on a fixed interface.

@prudentcircle said in Does WireGuard work in a High Availability (pfsync, "CARP") mirrored firewall environment?:

I know that VyOS is doing something similar to this. They combine VRRP and transition scripts to do this to make sure that if a node becomes a VRRP master, Wireguard comes up and if you are not a master anymore, kill wireguard.

Could maybe work. Still don't know how they'd treat WG to fix it's tendency to use the wrong IP or wrong interface though.

@prudentcircle said in Does WireGuard work in a High Availability (pfsync, "CARP") mirrored firewall environment?:

Is it because PfSense software itself is limited in some software designs?
or Is it because CARP has inherent limitations and is different from VRRP?

Nothing to do with FreeBSD or pfSense, wireguard is simply weird that way. And as I don't know what VyOs does with keepalived and if it's really using VRRP and runs WG only on the VRRP IP - I can't say that.
Also check that post in VyOS forums, that describes exactly what I said. Wireguard simply ignores the VRRP interface and communicates via the physical IP what you don't want in a cluster:
-> https://forum.vyos.io/t/wireguard-does-not-work-with-vrrp-ip-address/14909

@cypherpunk AFAIK, the pfSense Wireguard implementation does not support multicast.

Mentioned in the doc here.

@McMurphy said in Where to set MTU:

The maximum packet size for the internet link before fragmentation is 1472 (+28 = 1500)

in your case 1440 is fine for IPv4 only Tunnel. If the Tunnel also shall transport IPv6-Trafic you shall not use a MT bigger 1420. The reason is the slightly bigger overhead of IPv6 compared to IPv4.

Using tracepath you can check out pmtu and packet transfer, to find optimal results

See: https://schroederdennis.de/vpn/wireguard-mtu-size-1420-1412-best-practices-ipv4-ipv6-mtu-berechnen/ (german language)

@Bob-Dig
Awww. Suggesting that is like taking a xmas present back from a child. ☹️

Good idea - I'll try pruning the clients back to maybe 2 or 3 and experiment from there. If I have no luck with that, I'll check out opendwt (I used to run ddwrt - i didn't realise openwrt was unlocked/unlicensed for x86).

@Bob-Dig This is an added layer of security, if the device/machine is stolen for example they would have the private key. So by blocking by public IP we can stop the WG connect being used elsewhere. At least to certain networks using a VLAN firewall rule.

I have just install and is up and runing pf sense CE in my infrastrature. But regrading WireGuard with all the user I have is impossible to generate peer for every user on manual base.

So I create a new Ubuntu Server with pivpn with wireguard, then a port foward to that server.

At this point I have 700mb download and 650mb upload using wireguard and this configuration.

I know this is stupid, but creating peer manual for many users is also stupid. If someone convert the pivpn files to pfsense would be great.

@gabacho4 Did you ever figure it out?

@gunnyp

Self-inflicted!
Forgot to change the IP address of the managed switch that sits between the servers and the pfSense vm.
IVPN is up and running.

Good news. I've managed to figure out my major bottleneck. My main Win10 machine at home has a stu*** Intel I225-V 2.5 Gbit on board ethernet NIC that I was using. It has caused weird issues in the past with random disconnects and is globally known for its issues.. I have plugged in a ASUS USB 2.5 Gbit adapter and sure enough I am able to max out my wire speed now using the wireguard tunnel.

OpenVPN also works equally well now as on the other machine, and I figured out that enabling DCO in the OpenVPN client side can bring me close to peak speeds of 400Mbit wire speed. I haven't enabled DCO on the OpenVPN server side yet. I am still using OpenVPN via TCP, and I read for DCO thats not supported/recommended - even though interestingly enough it already gives me a performance boost already now just enabling it on the client side. Switching OpenVPN server to using UDP and enabling DCO on the server side as well might further improve things I imagine - something I might try in the next days.

Hope this thread will help somebody down the road troubleshooting the same.

@ahking19 Thanks for your reply. Both machines are on wired ethernet on the same router at home. I think I am not near maxing out the Netgate CPU, but I've attached a screenshot below to be sure (not sure how to interpret that plot exactly):
e7b8a2a8-aea9-4b8a-a82e-09a91455386a-image.png

Latency wise: I have about 9-15 ms ping through the OpenVPN tunnel from client to my netgate.
Using wireguard is about 9-11ms.

@compsmith said in Road Warrior need access all spokes in hub/spoke multisite:

Anyone out there have any insight to get this to work?

Okay, i got it fixed up myself.

It was the "Advanced Outbound NAT Entry" (Firewall / NAT / Outbound).
Rules defined there does not work if not "Outbound NAT Mode" is set at least to "Hybrid Outbound NAT rule generation". I did not marked that and therefor the defined rules below was ignored.
And without an appropiate Outbound NAT rule it is impossible to route via Wireguard all internet traffic from clients via the tunnel.
OpenVPN does not need such a rule or (most likely) generate the needed rules automatically (similar to IPsec). The standard "Outbound NAT Mode" is set to the entry "Automatic outbound NAT rule generation.
(IPsec passthrough included)". So I suspect the outbound NAT rules will be generated for both IPsec and OpenVPN automaticaly, but not automaticaly for Wireguard.

[Troubleshooting/Fix]

Pinging from Client -> Youtube.com (142.250.180.14).

PCAP on em0:WAN

#Tunnel Disabled: 18:48:21.808185 xx:xx:xx:xx:66:8a > xx:xx:xx:xx:28:27, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 990, offset 0, flags [none], proto ICMP (1), length 84) WAN_address > 142.250.180.14: ICMP echo request, id 61165, seq 7, length 64 18:48:22.516482 xx:xx:xx:xx:28:27 > xx:xx:xx:xx:66:8a, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 110, id 0, offset 0, flags [none], proto ICMP (1), length 84) 142.250.180.14 > WAN_address: ICMP echo reply, id 61165, seq 7, length 64 18:48:22.817329 xx:xx:xx:xx:66:8a > xx:xx:xx:xx:28:27, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 63388, offset 0, flags [none], proto ICMP (1), length 84) WAN_address > 142.250.180.14: ICMP echo request, id 61165, seq 8, length 64 18:48:23.518344 xx:xx:xx:xx:28:27 > xx:xx:xx:xx:66:8a, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 110, id 0, offset 0, flags [none], proto ICMP (1), length 84) 142.250.180.14 > WAN_address: ICMP echo reply, id 61165, seq 8, length 64 18:48:23.822630 xx:xx:xx:xx:66:8a > xx:xx:xx:xx:28:27, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 5156, offset 0, flags [none], proto ICMP (1), length 84) WAN_address > 142.250.180.14: ICMP echo request, id 61165, seq 9, length 64 18:48:24.526677 xx:xx:xx:xx:28:27 > xx:xx:xx:xx:66:8a, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 110, id 0, offset 0, flags [none], proto ICMP (1), length 84) 142.250.180.14 > WAN_address: ICMP echo reply, id 61165, seq 9, length 64 #Tunnel Enabled: 18:48:24.824198 xx:xx:xx:xx:66:8a > xx:xx:xx:xx:28:27, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 30834, offset 0, flags [none], proto ICMP (1), length 84) 10.100.0.101 > 142.250.180.14: ICMP echo request, id 30738, seq 10, length 64 18:48:25.833285 xx:xx:xx:xx:66:8a > xx:xx:xx:xx:28:27, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 23547, offset 0, flags [none], proto ICMP (1), length 84) 10.100.0.101 > 142.250.180.14: ICMP echo request, id 30738, seq 11, length 64 18:48:26.834520 xx:xx:xx:xx:66:8a > xx:xx:xx:xx:28:27, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 64833, offset 0, flags [none], proto ICMP (1), length 84) 10.100.0.101 > 142.250.180.14: ICMP echo request, id 30738, seq 12, length 64 18:48:27.840448 xx:xx:xx:xx:66:8a > xx:xx:xx:xx:28:27, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 50822, offset 0, flags [none], proto ICMP (1), length 84) 10.100.0.101 > 142.250.180.14: ICMP echo request, id 30738, seq 13, length 64 18:48:28.843054 xx:xx:xx:xx:66:8a > xx:xx:xx:xx:28:27, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 62192, offset 0, flags [none], proto ICMP (1), length 84) 10.100.0.101 > 142.250.180.14: ICMP echo request, id 30738, seq 14, length 64

So it appears there's no NAT for the 10.100.0.100/31 traffic. I tried creating the Hybrid NAT rule below:

Interface: Wireguard Source: Wireguard networks Source Port: any Static Port: unchecked Destination: any Destination Port: any NAT Address: WAN_address

However I was still seeing traffic not being NAT'ed. So I tried adjusting it:

Interface: WAN Source: Wireguard networks Source Port: any Static Port: unchecked Destination: any Destination Port: any NAT Address: WAN_address

Alas, this STILL did not work. On a hunch, I tried using the subnet specifically instead of "Wireguard networks" and suddenly it worked:

Interface: WAN Source: 10.100.0.100/31 Source Port: any Static Port: unchecked Destination: any Destination Port: any NAT Address: WAN_address

Is this behavior desired? It seems incongruent with the rest of the UI. The alias "Wireguard networks" does not appear to include the peer network. It seems like a rather simple addition to the Wireguard package.

Now I have encrypted traffic that cannot be easily cracked over the air :)

How is this still not a feature???
This should be the feature to be included in year 2024!!

I want this feature so bad that I might visit forum once a week to keep this post on the top page...

@Jarhead I do it like that. It might be less secure, but how much?
I wish we could get rid of the Resolver ACL too. 😁

@Jarhead said in Why use Allowed IP's?:

why would we ever set specific Allowed IP's if they really aren't doing anything needed? (like creating routes for example)

If you have more than one other peer, you can do 0.0.0.0/0 only on one.