Epyc 3251 and Wireguard
-
@jarhead I had 4 WG tunnels to the privacy VPN and 2 other tunnels. When it comes to physical interfaces, most are VLANs here, so no. Also I just wanted to mentioned it, don't really think that I could help here anyway.
-
Hmm, hard to see how the number of interfaces would affect Wireguard. It would increase the mbuf allocation.
Do you see any errors on the console or logged even if it doesn't panic/reboot?Not sure if it would better to get a crash report from igb at this point.
I could imagine the cxgbe driver is crashing completely but igb, when faces with the same traffic, just starts dropping it causing the massive loss/latency you are seeing.
Steve
-
@stephenw10
No errors anywhere I can see but now the other WG tunnel is going out too.
-
Your WAN is running on the cxl interface directly here right? Not on a VLAN or virtual interface?
-
@jarhead I had latency problems with the WG-tunnels to the privacy VPN too but no crashes. I thought it is their fault, usually it is.
-
Also just to confirm is your WG tunnel running on the WAN directly? It's not forwarding to localhost for example?
The crash looks like cxgbe crashing whilst trying to forward traffic.
Steve
-
@stephenw10 said in Epyc 3251 and Wireguard:
Your WAN is running on the cxl interface directly here right? Not on a VLAN or virtual interface?
No. WAN has been on igb0 since last night.
Just noticed in that picture how high the latency is on my WAN, haven't seen that before.
I wonder if that's what the whole problem is.
Again, not sure if it's been that high the whole time or not but I have to start there. It's usually around 3 to 4 ms.
Just turned off both WG tunnels and it's still around 150ms.Time to call the ISP. Oh joy.
-
No ISP call needed luckily.
Stupid mistake on my part.
When I put the guest wifi back this morning I moved the WAN port on my switch since I used that port for the WAN to igb0 yesterday.
So I plugged the new WAN port into a switchport that was set to 10/full. Causing the high latency obviously.
Changed that port to auto/auto and it's back up and running good.
Both WG tunnels up and seeing normal latency on both.Will leave it up this way for the day but I'm starting to think it has to be the chelsio card now. It just doesn't like wireguard for some reason. Can't explain why it did crash on the igb before but it's not now. Yet.
@stephenw10 I'm gonna want/need to move WAN back to cxl3 at some point. Other than mbuf, any other ideas of how to troubleshoot this?
-
Yeah, it looks like the cxgbe driver to me too. And to the developers that looked at this.
Previously when it was crashing on igb it never created a crash report right? But it was panicking and rebooting? Or just the stream of errors you originally posted?
It must be something about the WG traffic that is triggering an issue.
Try to grab the output of
netstat -m.Otherwise I'd be looking at the sysctl mac stats for the NIC in use.
Beyond that we might need a debug kernel or similar.
If you're able to leave it running on igb0 for a while that might help. You might be logging errors there that could point to a cause. Or if it does crash and generate a report that would be useful.
Steve
-
@stephenw10 said in Epyc 3251 and Wireguard:
Yeah, it looks like the cxgbe driver to me too. And to the developers that looked at this.
Previously when it was crashing on igb it never created a crash report right? But it was panicking and rebooting? Or just the stream of errors you originally posted?
Steve
Unfortunately I never let it complete the dump when it was on the igb interface. I've been thinking about it and I think that was caused by the bad config import. If you remember I found the gateways were all screwed up and who knows what else was. That's the only thing I can attribute to it crashing on the igb.
It's been running for a few hours now and it's perfect. This is with a new config I did from scratch and all new tunnels instead of trying to recreate the old ones.
It still crashes if I use the cxgbe and it doesn't with the igb.
So it must've been the config causing it then.When I can I'll put it back on the cxgbe and monitor what I can. Maybe do a sniff on it too.
-
Yeah a pcap might show something if it's some unusual traffic triggering it.
-
@stephenw10
Found a reasonably priced Dell Intel x520.
Took some figuring out since all of my sfp's are cisco, but once I got it running it's been good.
Even added a third Wireguard tunnel and no problems.Still can't believe Wireguard hates Chelsio!
-
Yeah, that's.... interesting. Good to find though!
Also I'd argue it's Chelsio that hates Wireguard.
Though I'm not sure if that's more unexpected.Steve
