Avahi Crash on pfSense 2.7.2

LukeT

Hi,

I've just installed a fresh copy of pfSense 2.7.2 and found that the Avahi package crashes immediately after enabling it in the WebUI, causing pfSense to panic and reboot.

Initially I found this on my main installation after having set up a number of other packages and config, which I run in KVM on an Ubuntu server. I figured I'd try reproing in a bare-bones VM, and the repro is simple...

create a VM and install pfSense 2.7.2 from the ISO
boot it and run through the installation wizard and reboot after installation
in the initial boot settings at the local console, create one VLAN interface (I'm doing this as I only have one physical NIC attached to the VM, so I'm using a VLAN for the LAN interface, giving me access to the pfsen webGUI via an external switch. Not sure if this is relevant for the repro)
browse to the webGUI and click through the initial wizard, leaving all options default.
install the Avahi 2.2_4 package
on the Services -> Avahi tab, check "Enable the Avahi daemon" and click save

As soon as you click save, pfSense crashes. There is panic log spam in the local console, and after rebooting, a crash dump is available via the webGUI. I've attached the files here.

Is Avahi still supported?

Luke

textdump.tar.0
info.0

dennypage

@LukeT said in Avahi Crash on pfSense 2.7.2:

I've just installed a fresh copy of pfSense 2.7.2 and found that the Avahi package crashes immediately after enabling it in the WebUI, causing pfSense to panic and reboot.

Yes, Avahi is still commonly used.

No user space process, Avahi included, should be able to cause a kernel panic. I would look toward issues with the VM infrastructure and/or NIC, perhaps related to Multicast.

LukeT

I've done some further tests. It looks like its a combination of SR-IOV passthrough of my Intel X540 NICs, plus setting a VLAN from within the pfSense VM guest:

If I create a private VM network (regular virtio NIC, no SR-IOV), then there's no crash.
If I add two separate SR-IOV NICs, either both VFs of the same physical NIC, or two entirely separate physical NICs, and avoid creating VLANs in pfSense, then there's also no crash.

I guess this needs to be filed as a bug? Is there any further info I can provide to help fix this?

dennypage

@stephenw10, anything you might offer here?

stephenw10

Hmm, not familiar. Backtrace:

db:0:kdb.enter.default>  bt
Tracing pid 87885 tid 100340 td 0xfffffe006a4e7ac0
kdb_enter() at kdb_enter+0x32/frame 0xfffffe0063881830
vpanic() at vpanic+0x163/frame 0xfffffe0063881960
panic() at panic+0x43/frame 0xfffffe00638819c0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe0063881a20
calltrap() at calltrap+0x8/frame 0xfffffe0063881a20
--- trap 0x9, rip = 0xffffffff80ee0935, rsp = 0xfffffe0063881af0, rbp = 0xfffffe0063881ba0 ---
in_joingroup_locked() at in_joingroup_locked+0x335/frame 0xfffffe0063881ba0
inp_setmoptions() at inp_setmoptions+0x11c6/frame 0xfffffe0063881d30
sosetopt() at sosetopt+0xb4/frame 0xfffffe0063881d80
kern_setsockopt() at kern_setsockopt+0xa5/frame 0xfffffe0063881de0
sys_setsockopt() at sys_setsockopt+0x24/frame 0xfffffe0063881e00
amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe0063881f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0063881f30
--- syscall (105, FreeBSD ELF64, setsockopt), rip = 0x82687dd0a, rsp = 0x8203704f8, rbp = 0x820370550 ---

Panic:

<6>vlan0: changing name to 'ixv0.7'
<6>ixv0: link state changed to DOWN
<6>ixv0: link state changed to UP
arprequest_internal: cannot find matching address


Fatal trap 9: general protection fault while in kernel mode
cpuid = 6; apic id = 06
instruction pointer	= 0x20:0xffffffff80ee0935
stack pointer	        = 0x28:0xfffffe0063881af0
frame pointer	        = 0x28:0xfffffe0063881ba0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 87885 (avahi-daemon)
rdi: fffff80003c2c480 rsi: 0000000000000000 rdx: 00000000000000a0
rcx: 0000000000000020  r8: 0000000000000000  r9: 00000000000000f8
rax: 14d81dff33337b20 rbx: fffffe006a4e7ac0 rbp: fffffe0063881ba0
r10: 0000000000000000 r11: fffffe006a4e7fe0 r12: 0000000000000000
r13: fffffe0063881c04 r14: fffff800797d9800 r15: fffff80003c2c400
trap number		= 9
panic: general protection fault
cpuid = 6
time = 1734052404
KDB: enter: panic

It's interesting that it seems to re-create the interface after boot. Did you resave something there?

I'd have to guess it's some hardware off-loading in the ixv driver causing problem there when it tried to jpin the multicast group.
What hardware offloading is shown as capable or enabled by ifconfig -m ixv0 or for ifconfig -m ixv0.7?