Wireguard Site-to-Site Setup - Errors on Interface
-
Hi all -
I recently setup a site-to-site wireguard VPN tunnel following the recipe available in the Netgate docs here:
https://docs.netgate.com/pfsense/en/latest/recipes/wireguard-s2s.html
Everything is working well, but on both sites I see "Errors In" slowly ticking up when looking at the Interface Statistics dashboard widget. Does anyone have any idea why this might be happening? Both sites are very similar: Both are running pfSense 2.5.2, Wireguard 0.1.5_3, and have same type of connection (fiber) from the same provider. Looking at Status > Interfaces I do see that the Wireguard interface has an MTU of 1500 - is that expected (I thought Wireguard MTU was 1420)?
Thanks in advance for your help, I really appreciate it.
-
@tman222 said in Wireguard Site-to-Site Setup - Errors on Interface:
I do see that the Wireguard interface has an MTU of 1500 - is that expected (I thought Wireguard MTU was 1420)
1420 would be the correct MTU that you would want to use. There's a significant amount of overhead in the Wireguard packets so the MTU has to be lowered.
Christian MacDonald put out a video on site-to-site Wireguard that is worth watching: https://www.youtube.com/watch?v=2oe7rTMFmqc
-
@tman222 Sans a few exceptions, when an interface is assigned to pfSense via Interfaces>Assignments, these interfaces default to an MTU of 1500. This is too high.
I'll repost this breakdown here so anyone stumbling onto this post can easily find an explanation:
- 20-byte IPv4 header or 40 byte IPv6 header
- 8-byte UDP header
- 4-byte type
- 4-byte key index
- 8-byte nonce
- N-byte encrypted data
- 16-byte authentication tag
N(IPv6) : 1500-(40+8+4+4+8+16) = 1420 bytes
N(IPv4) : 1500-(20+8+4+$+8+16) = 1440 bytes -
Hi @dma_pf and @cmcdonald - thanks a lot for the replies.
I guess I should have watched the video all the way through as the step about setting the MTU and MSS on the WireGuard tunnel interfaces is missing in the WireGuard site-to-site recipe in the Netgate documentation...
That being said, I set the MTU and MSS both to 1420 for the two sites (as per video), but I'm still seeing the "Errors In" on the WireGuard tunnel interfaces slowly tick up (on both sites). Can you guys think of anything else that I should be checking to try to troubleshoot this? Also, should the MSS be set to 1420 or 1380 (i.e. subtracting 40 bytes for IPv4)?
Thanks again for all your help.
-
Looks like the packet error rates I'm currently seeing are 0.0004% on one site and 0.01% on the other. Maybe that is considered ok / to be expected? Thanks again.
-
@tman222 said in Wireguard Site-to-Site Setup - Errors on Interface:
Looks like the packet error rates I'm currently seeing are 0.0004% on one site and 0.01% on the other. Maybe that is considered ok / to be expected? Thanks again.
I don't personally have a site-to-site setup for Wireguard so I have no sense of what would be expected or acceptable. Maybe @cmcdonald has a feel for that as he's the Wireguard expert at pfsense.
But in general with Wireguard using UDP I would expect that some packets would be lost. Unlike TCP, UDP does not have a guarantee that all of the data sent will be delivered. As UDP is a "best efforts" protocol I would expect that after leaving pfsense some packets will get lost enroute.
-
@tman222 As a follow up. On my system I have 3 wireguard connections to a VPN provider. I show no incoming errors on any of the tunnels but I do see outgoing errors on each of the 3 interfaces. The error rates are .0002%, .0004%, and .0007%.
-
@dma_pf said in Wireguard Site-to-Site Setup - Errors on Interface:
@tman222 As a follow up. On my system I have 3 wireguard connections to a VPN provider. I show no incoming errors on any of the tunnels but I do see outgoing errors on each of the 3 interfaces. The error rates are .0002%, .0004%, and .0007%.
Thanks @dma_pf - I really appreciate the follow up. Seeing your numbers makes me feel a bit more comfortable that some level of packet errors are probably expected.
@dma_pf and @cmcdonald - What would be the best way to troubleshoot this further? Does WireGuard have logs I can review that may shed some light? Or do I need look at detailed interface statistics for the tunnel interfaces? Maybe decreasing the MTU further to 1400 or below might help?
Thanks again for all your help.
-
Well, after a few days using WireGuard in a site to site VPN configuration, I see no adverse affects from the few errors showing on the interfaces (even if they are ticking up slowly). It also looks like the errors aren't necessarily proportional to total traffic / total packets transferred either. For instance, I ran some iperf3 tests through the tunnel recently and didn't see the errors tick up materially during the tests, so perhaps it's only certain traffic that causes errors now and then.
Overall though, I'm quite happy with the performance I'm seeing from WireGuard. I was able to achieve almost 900 Mbit/s transfer speeds through the tunnel using a single iperf3 stream (each site is using a 1 Gbit/s fiber internet connection) - very impressive.
-
@tman222 I've seen similar speeds as you on my fiber connection as well. Sometimes pushing to around 940.
Interestingly, I've been monitoring the interface statistics since you brought them to my attention. Since my last posting there have been no additional errors. So my incoming still shows 0 errors on all three interfaces. Outgoing the amount of errors have remained the same on the three interfaces at 5, 18 and 17. With the amount of traffic that has gone out since my last post it now means that the error rate is .00008%, .0002% and .0003%.
I'm not sure what time period the Interface Statistics covers (the router was last rebooted 8 days 11 hours ago) but it shows that the three interfaces have pushed 62,701,113 packets in and 22,618,258 packets out. So if you look at the errors per all of the packets the error rate has been .00005%
-
Just wanted to follow up on this topic - after upgrading to the latest 23.01 release, I no longer see the error count ticking on the Wireguard interfaces (at least so far).
Hi @cmcdonald - were there any changes made to the Wireguard package that could have influenced this, or maybe it is related to the upgrade from FreeBSD 12.3 to 14?
Thanks in advance.
-
@tman222 Just out of curriosity: What boxes are on either end of that tunnel? Iām looking for what throughput can be expected for the SG-2100 ARM based boxes, but no-one seems to know :-)
(With 900mbps+ I know you are not
) -
@keyser said in Wireguard Site-to-Site Setup - Errors on Interface:
@tman222 Just out of curriosity: What boxes are on either end of that tunnel? Iām looking for what throughput can be expected for the SG-2100 ARM based boxes, but no-one seems to know :-)
(With 900mbps+ I know you are not)Hi @keyser - hardware on both sides fairly powerful (at least as far as firewalls concerned): System on one side is driven by a Xeon D-1518 CPU, System on the other side has a Intel Core i3 10100 CPU. Bear in mind that those results are from a single stream iperf3 test using default settings (i.e. large 1500 byte packets) and that the site to site latency is only a few milliseconds.
