Bug in Broadcom bnxt driver in combination with VLANs

[tmoehle]

Hey guys,

I've ran into a bug after upgrading pfSense to 2.7 on our Dell servers. We're using Broadcom BCM57416 ethernet adapters and as soon as you create multiple VLANs on them, you will receive the following error:

bnxt0: Attempt to re-allocate l2 ctx filter (fid: $somelongnumber)
bnxt1: Attempt to re-allocate l2 ctx filter (fid: $somelongnumber)

It looks like there actually is a bug in the driver: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133

Unfortunately I don't know how to implement this patch myself, so of course I'd be glad if someone could give me a hint how to fix it on my end. But I guess the best way would be if this got fixed for all pfsense users, as for now this firewall server is unusable in production.

Thanks :)

[PalisadesTahoe]

I think I'm running into the same issue. Brand new Dell server with BCM57416. Works fine with single VLAN assigned to the nic, as soon as I add a second VLAN, I get the "Attempt to re-allocate" error on screen and everything breaks. Haven't found a way around it yet.
-S

[stephenw10]

You would need to recompile the driver with the patch and load it as a module. Non-trivial.
It's not included upstream yet even in main: https://github.com/freebsd/freebsd-src/tree/main/sys/dev/bnxt

Steve

[tmoehle]

@stephenw10 said in Bug in Broadcom bnxt driver in combination with VLANs:

You would need to recompile the driver with the patch and load it as a module. Non-trivial.

So, I assume you'd suggest going back to 2.6 for now?

[stephenw10]

Since it looks like that's in VLAN filtering in the driver you could try disabling VLAN hardware off loading on the NIC.

[tmoehle]

@stephenw10
Thank you for that suggestion. The idea was promising, but unfortunately not successful. I tried:

ifconfig bnxt0 -vlanhwtag -vlanhwfilter -vlanhwtso
ifconfig bnxt1 -vlanhwtag -vlanhwfilter -vlanhwtso

But the errors remain :(

[slu]

I'm also interested in a workaround, we plan to use the Broadcom BCM57416 with VLANs.

[stephenw10]

Mmm, worth trying but I did expect to see that on the FreeBSD bug report.

In he short term going back to 2.6 may be the only option.

[Delegator5042]

@stephenw10
Can you point me where to look if I want to try and recompile this for myself?
I'm a beginner but I would like to try to get my NIC to work with more than 1 VLAN

[stephenw10]

As far as I can see there is no validated patch for this yet: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133#c31

[Delegator5042]

@stephenw10
A bummer, but can't be helped! I'll have to find other uses for the NIC in the meantime.
Thanks for the quick response!

[Delegator5042]

@tmoehle Did going back to 2.6 make the VLANs functional?
I tried it for a bit and didn't get anywhere (didn't get the re-allocate error but the interface didn't forward any traffic)
Going back to 2.6 permanently also seems cumbersome since I could not install packages from the package manager until I update to 2.7 again.

[stephenw10]

You can install packages, just set the repo branch to 2.6 (deprecated).

[tmoehle]

@Delegator5042
I did a clean install for pfsense 2.6 and restored the old configuration backup. That went perfectly smooth. After that, as @stephenw10 already suggested, I went to System > Update and changed the desired branch to 2.6, so I could download packages again.

[Delegator5042]

@tmoehle
Encouraging to hear! I will give it a(nother) proper go then.
Silly me just read an old forum posts about the packages requiring an update, which I somehow can't find anymore so I probably didn't read something correct when I was looking for a place to download 2.6

[Delegator5042]

@tmoehle Sorry for prying a lot, but do the VLANs on your NIC really work?
I installed 2.6 and the NIC works without VLANs.
When I create a VLAN and assign it to an interface, No traffic gets through (oddly enough the DHCP does work)
When I set a different network port (non bnxt) with the same VLAN to the same configured interface, it all works (albeit I have to reload firewall rules first).

I am hoping that I am doing something wrong, since DHCP somehow still works on the VLAN with bnxt, but I don't see any issues with the gateway and the firewall rule permits the traffic.

[tmoehle]

@Delegator5042 My VLANs are working perfectly fine, yes. If your DHCP is working within those VLANs and pfsense is your only DHCP server, then chances are you overlook something.

[stephenw10]

This should now be fixed in the next 23.09 snapshots if anyone can test that.

Steve

[tmoehle]

@stephenw10 Will it only go into 23.09 or will there also be an update for CE?

[stephenw10]

It's fixed upstream in FreeBSD so it will be pulled into new CE builds.