Epyc 3251 and Wireguard

Jarhead

@stephenw10
Still no good.
Just created 1 tunnel. It comes up fine but as soon as I try to use it, gone.

stephenw10

Hmm, still showing issues in the Chelsio driver.

Panic:

Fatal trap 9: general protection fault while in kernel mode
cpuid = 12; apic id = 0c
instruction pointer	= 0x20:0xffffffff8065f3d9
stack pointer	        = 0x28:0xfffffe009a0fb540
frame pointer	        = 0x28:0xfffffe009a0fb570
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (irq323: t5nex0:3a2)
trap number		= 9
panic: general protection fault
cpuid = 12
time = 1661734572
KDB: enter: panic

Backtrace:

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100213 td 0xfffff80005df0000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe009a0fb250
vpanic() at vpanic+0x197/frame 0xfffffe009a0fb2a0
panic() at panic+0x43/frame 0xfffffe009a0fb300
trap_fatal() at trap_fatal+0x391/frame 0xfffffe009a0fb360
trap() at trap+0x67/frame 0xfffffe009a0fb470
calltrap() at calltrap+0x8/frame 0xfffffe009a0fb470
--- trap 0x9, rip = 0xffffffff8065f3d9, rsp = 0xfffffe009a0fb540, rbp = 0xfffffe009a0fb570 ---
cxgbe_transmit() at cxgbe_transmit+0x19/frame 0xfffffe009a0fb570
ether_output_frame() at ether_output_frame+0xb4/frame 0xfffffe009a0fb5a0
ether_output() at ether_output+0x676/frame 0xfffffe009a0fb620
ip_output() at ip_output+0x136c/frame 0xfffffe009a0fb770
ip_forward() at ip_forward+0x39e/frame 0xfffffe009a0fb840
ip_input() at ip_input+0x850/frame 0xfffffe009a0fb8f0
netisr_dispatch_src() at netisr_dispatch_src+0xca/frame 0xfffffe009a0fb940
ether_demux() at ether_demux+0x16a/frame 0xfffffe009a0fb970
ether_nh_input() at ether_nh_input+0x330/frame 0xfffffe009a0fb9d0
netisr_dispatch_src() at netisr_dispatch_src+0xca/frame 0xfffffe009a0fba20
ether_input() at ether_input+0x89/frame 0xfffffe009a0fba80
service_iq_fl() at service_iq_fl+0x5d2/frame 0xfffffe009a0fbb30
t4_intr() at t4_intr+0x2d/frame 0xfffffe009a0fbb50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe009a0fbbb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe009a0fbbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009a0fbbf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

But wireguard was no longer running on it when happened?

stephenw10

What's the WG tunnel connected to there? Another pfSense install?

Jarhead

@stephenw10 said in Epyc 3251 and Wireguard:

Hmm, still showing issues in the Chelsio driver.

I assume that's only because my WAN is on the chelsio at the time. I didn't check when I disconnected the chelsio card but I would also assume it would've shown as the igb0 at that time.

But wireguard was no longer running on it when happened?

Probably the cause right there. WG shutting down when I try to use it?

Jarhead

@stephenw10 said in Epyc 3251 and Wireguard:

What's the WG tunnel connected to there? Another pfSense install?

Unfortunately not.
That tunnel goes to an opnsense box. At least until the vlan0 is fixed.

stephenw10

Hmm, so the encrypted WG traffic still runs over the Chelsio NIC, the WAN?

Jarhead

@stephenw10 Not really sure what you're asking there.
My WAN is on the chelsio card (cxl3), the WG tunnel comes up with handshakes, but as soon as I try to access the other side it crashes.

stephenw10

Mmm, I'm unsure what you moved to igb0. I would have expected that to have to be the WAN for the WG interface to be running on it.

Jarhead

@stephenw10 I moved the WAN to igb0 and disconnected the chelsio card from the motherboard as a test.
The trouble still happened.
So I don't think focusing on the chelsio is the way to go.
It happens with the onboard nics also.
Because it still happened with the onboard nics, I reinserted the chelsio and moved WAN back to it.

stephenw10

Right, I would agree except that it appeared the error was still on the Chelsio NIC even when it was not carrying WG traffic as I understand it.

It would be good to get a crash report from the igb0 as WAN setup if that's possible. It would be very surprising to see the same error on igb sicne many people are running WG with an igb parent.

Jarhead

@stephenw10 said in Epyc 3251 and Wireguard:

Right, I would agree except that it appeared the error was still on the Chelsio NIC even when it was not carrying WG traffic as I understand it.

How are you coming up with that?

stephenw10

@jarhead said in Epyc 3251 and Wireguard:

But wireguard was no longer running on it when happened?

Probably the cause right there. WG shutting down when I try to use it?

I may have read that wrong. But what I meant to ask there was; was WG running on the Chelsio NIC when that crash report was generated?

Jarhead

@stephenw10 I'll go through the whole thing again, trying to be more clear.

New router. Backed up old, restored on new changing interfaces as needed.
Wireguard would crash.
Moved WAN and LAN to onboard igb nic's.
Wireguard would crash.
Since this proves it's not related to the chelsio card, as it wasn't even plugged in to the motherboard, I reinstalled the chelsio and moved WAN and LAN back to it.
Wireguard would crash.
I found some weird errors in my gateways, as in network 1 was using gateway 2, and network 2 using gateway 1 when they should be 1 to 1 and 2 to 2, so I uninstalled wireguard then reinstalled it and recreated one tunnel.
Wireguard crashed and that's the dump I posted here.

So focusing on the chelsio card seems to be not the way to go.

Have you guys used an Epyc 3251 in the office for testing at all?

stephenw10

Ah, OK. Sorry I misinterpreted the responses there then.

Is it possible to switch back to igb and try to generate a crash report?
The crash you saw there in cxgbe looks very similar to some we have seen in other drivers but that I expect to be fixed in igb.

I'm not aware of any testing that has been done on an Epyc platform device.

Steve

Jarhead

@stephenw10 Definitely possible, just don't know when I can get to it. Got a lot of painting to do tonight so maybe I can play around as I watch the paint dry.

stephenw10

Ha, sounds like a good option!

I'll keep digging here, see if anyone has any suggestions.

Jarhead

@stephenw10 Boy that paint took a long time to dry!

Gave me a lot of time to try this out.
Kept going back to that config being messed up so I started from scratch.
New install, chelsio card not connected, just changed my network address with WAN on igb0 and LAN on igb1.
Installed Wireguard. It ran fine.
Installed Chelsio. still ran fine on igb interfaces.
Moved WAN and LAN to chelsio. Still ran fine!
So the Chelsio is not causing the issue.
Spent some time (a lot! long night) setting the config back to my usual. Not importing anything, had my old router up and used it as a reference.
Wireguard crashed.
In the process of setting up my old router as the new endpoint so I can rule out opnsense being the cause. (secretly hoping it is so I can get rid of it!!)

Did get a new crash dump that might show something. This is still on the chelsio though.

textdump.tar.0
info.0

stephenw10

Mmm, pretty much identical crash and it's in the cxgbe driver.
I have no idea how the WG encrypted traffic could be triggering it though.

I'd be willing to bet it would not crash with igb as WAN. Though you said you were still seeing the errors logged with igb?

After reconfiguring it with your previous settings did it start crashing immediately?
Was it actually panicking and rebooting or just Wireguard erroring out?

Steve

Jarhead

@stephenw10
But it did crash with the igb driver.
When I have a clean config (meaning no other interfaces assigned) it runs fine on both igb and cxgbe.
When I put my config back on it, it crashes on igb and cxgbe.
Gonna try disabling all other interfaces when I get a chance later.

stephenw10

The firewall itself crashed and rebooted with the igb NIC as WAN?