Kernel Panic on Temporarily Disable CARP with ixgbe driver

fastpilot

Could be an use-after-free.

pfsense 2.2.6 is FreeBSD 10.1-RELEASE-p25 which has mainly security fixes from the 10-Branch but is missing some other fixes.

https://svnweb.freebsd.org/base?view=revision&revision=277625 should fix it.

Yes, that's what I thought. Although that patch appears to be in the devel branch https://github.com/pfsense/FreeBSD-src/commit/f72184af7f1b19f99893f951a64a22f22ec344ba. I tried a beta build of 2.3 last week and same problem. Are the beta snapshots taken off the devel branch?

Perforado

I guess it is.

Can you ssh into the pfsense and do an "uname -a" on the shell?

fastpilot

@Perforado:

Can you ssh into the pfsense and do an "uname -a" on the shell?

FreeBSD <redacted> 10.2-STABLE FreeBSD 10.2-STABLE #317 58b7eab(devel): Fri Jan 15 04:28:46 CST 2016     root@pfs23-amd64-builder:/usr/home/pfsense/pfsense/tmp/obj/usr/home/pfsense/pfsense/tmp/FreeBSD-src/sys/pfSense  amd64</redacted>

Perforado

https://github.com/pfsense/FreeBSD-src/commit/f72184af7f1b19f99893f951a64a22f22ec344ba#diff-2a75ab8f3cf1e4838de5abd9c14a1870

seems to be in there. If thats the tree the beta is built from.

fastpilot

Yeah, it looks like it is. The commit hash in uname (58b7eab) is from the devel branch.

As I mentioned earlier, this sounds similar to https://forum.pfsense.org/index.php?topic=55433.0, which wasn't actually solved, just worked around by using a different NIC. I've traced the code back from carp_forus() which attemps to grab the lock, but going back to the ixgbe driver it just gets too complicated for me and I haven't managed to find what may be freeing the ifp pointer https://github.com/pfsense/FreeBSD-src/blob/945ed01c4bae06169f63978e43029c04d4abd731/sys/netinet/ip_carp.c#L1126.

fastpilot

I should add that ether_input() does check if ifp isn't a NULL pointer, but maybe there's a race condition here where something else is clearing it. https://github.com/pfsense/FreeBSD-src/blob/5aba7ffcfb97d9b6f4ce464de77b02ad4d7b8ad3/sys/net/if_ethersubr.c#L628.

Perforado

Did you try

hw.pci.enable_msix=0

btw?

fastpilot

@Perforado:

Did you try

hw.pci.enable_msix=0

Yep, that worked! What's the impact of disabling MSI-X though? Would be nice to not have to disable MSI-X and get to the bottom of the bug.

Perforado

MSI-X is an extension to MSI which afaik implements separate capabilty structure, offers more vectors:

https://en.wikipedia.org/wiki/Message_Signaled_Interrupts

Now that it works without MSI-X you could try a different Slot for the ixgbe-card (HP should have a best practice document for that)

And you could try to update the Servers Bios. Maybe MSI-X Setup is somewhat borked.

fastpilot

@Perforado:

Now that it works without MSI-X you could try a different Slot for the ixgbe-card (HP should have a best practice document for that)

And you could try to update the Servers Bios. Maybe MSI-X Setup is somewhat borked.

Server BIOS is up to date, running the latest release from HP which came out last month. I'll see if I can try a different slot. On a somewhat related note, I had to disable x2APIC in the BIOS for the machine to boot. Not sure if that's a BIOS or FreeBSD issue.