Kernel Panic when applying setting

junicast

Hi guys.
I'm running two physical machines with pfSense 2.4.4 on them in CARP mode. There is one 10 GbE Ethernet Port connected through which I run tagged vlan (802.1Q).

Hardware details:
Intel Server Board S1200SPL
Xeon E3 1230 v6 3,5 GHz
8 GB RAM ECC
2 x 120 GB SSD RAID 1 (ZFS)
Intel 10 Gigabit Ethernet SFP Fiber X710
Intel I210 onboard for dedicated HA

Every once in a while when I change a setting via WebGUI and then say apply the system will become unreachable and after some duration will reboot. I think it's some kind of a kernel panic. While it's unresponsive there's a lot of text running through the console screen. See photo.
In this case I changed "Block Bogon Networks" from checked to unchecked on several interfaces at a time. When I hit apply the error occured.

Also the diagnostic files:
0_1539350090049_debug.tar.gz

jimp

Looks like a crash in the ixl driver.

ixl0: RX queue 0 still enabled!
ixl0: aq_add_macvlan err -53, aq_error 14
ixl0: aq_add_macvlan err -53, aq_error 14

db:0:kdb.enter.default>  bt
Tracing pid 5083 tid 100522 td 0xfffff8000ee33000
free() at free+0x4e/frame 0xfffffe021cce05a0
ixl_init_locked() at ixl_init_locked+0x3ea/frame 0xfffffe021cce0610
ixl_ioctl() at ixl_ioctl+0x6d5/frame 0xfffffe021cce0670
ifioctl() at ifioctl+0xcf8/frame 0xfffffe021cce0740
kern_ioctl() at kern_ioctl+0x26d/frame 0xfffffe021cce07b0
sys_ioctl() at sys_ioctl+0x16c/frame 0xfffffe021cce0880
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe021cce09b0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe021cce09b0

jimp

from a quick glance at the source it's from VLAN hardware filtering failing somehow. You could try disabling that for that interface. There isn't a persistent way to do it from the GUI, but from the CLI you can run ifconfig ixl0 -vlanhwfilter and see if it stabilizes.

junicast

@jimp
I will give that a try. Thank you.

junicast

Sadly it is not stabilizing. When I create or update a VLAN I get a single message on stderr

ixl0: aq_add_macvlan err -53, aq_error 14

When I try to set an IP-address for such an interface I get this error message spamed in the console. Networking becomes highly unresponsive. Then the server reboots after lots of those kernel messages I sent earlier.
I guess this is rather for the bugtracker but for the forum, right?

Edit:
This is another topic adressing the same issue:
https://forum.netgate.com/topic/136201/new-version-2-4-4-interface-error-aq_add_macvlan-err-53-aq_error-14

jimp

Well, there may not be anything we can do for that since it appears to be a driver issue. It's probably something you'd need to try to reproduce on a stock FreeBSD 11.2 installation. If the same issue happens there, it should be reported to FreeBSD directly.

I don't think we have any hardware on hand that uses that driver, so it isn't something we can investigate internally.

junicast

Is there a way I can find out what commands were invoked by using the GUI so I can execute them manually on a stock FreeBSD?

jimp

It depends on what you're doing. In some cases you can look at the page and check what happens when Apply Changes is performed, but in other cases the actual config may be done by our pfSense PHP module and not shell commands. The net effect should be the same, though.

But find what you're trying to do and see how that is done on FreeBSD. If it's just VLANs there should be several rc.conf examples around that will do it for you by making a couple edits.

junicast

It happens for example when I change the interface name in pfSense. I cannot find out what php skript is invoked and which commands are being executed in the OS.

I just installed FreeBSD 11.2 plugged in an identical NIC and tinkered with vlan configurations. I get the same error message when I create a vlan interface upon ixl0.

ifconfig ixl0.500 create vlan 500 vlandev ixl0 inet 10.123.12.1/22
ixl0: aq_add_macvlan err -53, aq_error 14

The system doesn't crash though.

Edit:
Just for reference, here's the bug I issued at FreeBSD's.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233198

stephenw10

Are you able to test in FreeBSD 12? Or 11-stable? There have been some updates to that driver that look relevant since 11.2-release.
https://github.com/freebsd/freebsd/commit/eeec306a5944aa752fb8d2b210fe2c558ad92269#diff-b1b4b271e4bf3fff5da3a7c784c2a443

http://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.2/

Steve

Juve

I am seeinga similar behavior in 2.4.4-P3:

when using Web GUI: ixl0: aq_add_macvlan err -53, aq_error 14
when using command line: ifconfig lagg0.502 create vlan 502 vlandev lagg0 inet 10.123.12.1/22 : no problem

no kernel panic, bu no traffic is passing when configuring through Web GUI.

stephenw10

Can you test in a 2.5 snapshot? Or against a more recent FreeBSD directly?

Steve