Kernel panic on GRE over IPsec tunnel on Netgate 7100 1U

tsakki

I have a Netgate 7100-1U that suddenly crashes while trying to establish an IPv6 over GRE tunnel, right after printing 'Configuring IPsec VPN...' during boot. I have been using this tunnel for the last few months (the endpoint is a Linux VPS running strongSwan) and it had never crashed.

I am currently running pfSense Plus 23.01 BETA. The stack trace that gets printed on the serial console is the following:

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 10
fault virtual address   = 0x460
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80eb7fd6
stack pointer           = 0x28:0xfffffe00855f8f20
frame pointer           = 0x28:0xfffffe00855f8f20
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi1: netisr 3)
rdi:                0 rsi:                2 rdx:                1
rcx:                0  r8:                0  r9:  200000000000000
rax:                2 rbx:                0 rbp: fffffe00855f8f20<C8><C5>0: fffff80139140078 r11:                8 r12: fffffe00855f8f88
r13: fffff80139967678 r14:                0 r15: fffff80139967600
trap number             = 12
panic: page fault
[...]
Tracing pid 12 tid 100040 td 0xfffffe0085673740
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00855f8ce0
vpanic() at vpanic+0x182/frame 0xfffffe00855f8d30
panic() at panic+0x43/frame 0xfffffe00855f8d90
trap_fatal() at trap_fatal+0x409/frame 0xfffffe00855f8df0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00855f8e50
calltrap() at calltrap+0x8/frame 0xfffffe00855f8e50
--- trap 0xc, rip = 0xffffffff80eb7fd6, rsp = 0xfffffe00855f8f20, rbp = 0xfffffe00855f8f20 ---
if_inc_counter() at if_inc_counter+0x6/frame 0xfffffe00855f8f20
looutput() at looutput+0x4f/frame 0xfffffe00855f8f50
ip6_forward() at ip6_forward+0x888/frame 0xfffffe00855f9050
pf_refragment6() at pf_refragment6+0x164/frame 0xfffffe00855f90a0
pf_test6() at pf_test6+0x1380/frame 0xfffffe00855f9210
pf_check6_out() at pf_check6_out+0x40/frame 0xfffffe00855f9240
pfil_mbuf_out() at pfil_mbuf_out+0x35/frame 0xfffffe00855f9270
ip6_output() at ip6_output+0x1204/frame 0xfffffe00855f94b0
icmp6_reflect() at icmp6_reflect+0x2dd/frame 0xfffffe00855f9560
icmp6_error() at icmp6_error+0x37c/frame 0xfffffe00855f95d0
pf_route6() at pf_route6+0x7ff/frame 0xfffffe00855f96b0
pf_test6() at pf_test6+0xce3/frame 0xfffffe00855f9830
pf_check6_out() at pf_check6_out+0x40/frame 0xfffffe00855f9860
pfil_mbuf_out() at pfil_mbuf_out+0x35/frame 0xfffffe00855f9890
ip6_output() at ip6_output+0x1204/frame 0xfffffe00855f9ad0
icmp6_reflect() at icmp6_reflect+0x2dd/frame 0xfffffe00855f9b80
icmp6_input() at icmp6_input+0x143b/frame 0xfffffe00855f9d10
ip6_input() at ip6_input+0x92f/frame 0xfffffe00855f9df0
swi_net() at swi_net+0x138/frame 0xfffffe00855f9e60
[...]

The only unsupported element in my configuration is BIRD running with a custom Shellcmd (because FRR's ospf6d keeps crashing for no apparent reason) but it should be launched later on, when the boot is complete.

The issue only shows up when the GRE tunnel is brought up. If I kill the other endpoint, run ifconfig down gre0 and then initiate the IPsec IKE and child SA everything is OK until i bring gre0 up again.

Any help is appreciated.

stephenw10

So this started happening after upgrading to 23.01? And is repeatable every time?

For the last few months it worked as expected was that all in 22.05?

Steve