erros seen in Wireguard interface

michmoor

For a while now i have been getting latency spikes from my monitoring station in Linode to my home exceeding 300ms over wireguard.
When the alerts fire off i do a sanity check and run my mtr traces from my linode instance to my WAN port. No packet loss things look good. In fact the latency numbers are within spec at around 3ms. So pings to the WAN port are normal.
Today i decided to dig in deeper since i had some time. Pings to my internal subnet are 300ms specifically to the LAN gateway address.
So traffic THROUGH the VPN tunnel is taking errors which indicates a problem there.

Running netstat -i i do see errors increasing on my tun_wg1 which is the interface used for my routing over Wireguard to my Zabbix instance in Linode.

CPU Util is no issue.
No interrupts seen that are even remotely concerning.
Quality graphs for my WAN which is monitoring google is fantastic - 4ms

Zabbix monitoring

OErrors

tun_wg1 1500 <Link#13> tun_wg1 2293246 20 0 2410051 1493 0

<2 sec later >
tun_wg1 1500 <Link#13> tun_wg1 2294179 20 0 2411065 1494 0

<30 sec later>
tun_wg1 1500 <Link#13> tun_wg1 2297669 20 0 2414697 1496 0

My Zabbix instance is seeing RX-ERR counters increase on the wg0 interface.

edit: restarting the wireguard service on the Zabbix side doesn't help the issue. As soon as i restart it the errors arrive. So this feels like a pfsense issue but i don't know where to look. Eventually this problem will go away but it does pop up every now and then.

 netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
docker0   1500  3223173      0      0 0       6868283      0      0      0 BMRU
eth0      1500 77751266      0      0 0      77945734      0      0      0 BMRU
lo       65536 13422449      0      0 0      13422449      0      0      0 LRU
tailscal  1280        0      0      0 0           326      0      0      0 MOPRU
veth64ba  1500  2702982      0      0 0       6021960      0      0      0 BMRU
vethee66  1500   320934      0      0 0        861684      0      0      0 BMRU
vethfd81  1500   199257      0      0 0        459679      0      0      0 BMRU
wg0       1420     5224     42      0 0          4870      0      0      0 OPRU
ztwdjmeo  2800  3546625      0      0 0       3672839      0      0      0 BMRU

stephenw10

So this is pfSense at both ends of the tunnel?

Do you see errors on the interfaces wireguard is running on?

Steve

michmoor

@stephenw10
one side is pfsense other side is just a VCP (wireguard client)

I put the netstat -i output from both ends.

stephenw10

Yeah I'm not seeing any other errors at the VCP end but I can't see anything for the parent NIC at the pfSense end. No errors on that?

Looks like receive errors are both ends on the wireguard interface? About the same rate?

michmoor

@stephenw10
Yes around the same rate.

An addendum to my troubleshooting last night. When i did reset the VPN on the VPC side errors went away and latency dropped.
Not sure who to blame then. pfSense or VPC side. I confirmed that when the latency and errors happen a restart of the service fixes it. Why it occurs is a mystery.

stephenw10

Hmm, sounds like the VPN connection was taking a different route perhaps?