PFSense and BCM57810 in LAG not working

DonZalmrol

Hope I'm posting this under the right category but finally upgrading my pfsense from an old Watchguard unit to a custom build low power server with a HPE 530SFP+ (QLogic NetXtreme II BCM57810) 10GB NIC in it.

Before purchasing the card I've looked up if it was supported by BSD and it is.
https://www.freebsd.org/cgi/man.cgi?query=bxe&sektion=4 and BSD uses the BXE driver for it.

Under my /boot/loader.conf I've added the command to load the BXE driver upon boot "if_bxe_load="YES""

kern.cam.boot_delay=10000
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
kern.ipc.nmbjumbo9="524288"
if_bxe_load="YES"
autoboot_delay="3"
hw.usb.no_pf="1"
net.pf.request_maxcount="400000"

After the reboot to enable the driver and use it in PFsense I see it in the kernel log:

Apr 19 13:48:49  kernel   > mem 0x2bf000000-0x2bf7fffff,0x2be800000-0x2beffffff,0x2bf810000-0x2bf81ffff irq 16 at device 0.0 on pci6  
Apr 19 13:48:49  kernel   bxe0: PCI BAR0 [10] memory allocated: 0x2bf000000-0x2bf7fffff (8388608) -> 0xfffffe0219a00000  
Apr 19 13:48:49  kernel   bxe0: PCI BAR2 [18] memory allocated: 0x2be800000-0x2beffffff (8388608) -> 0xfffffe021a200000  
Apr 19 13:48:49  kernel   bxe0: PCI BAR4 [20] memory allocated: 0x2bf810000-0x2bf81ffff (65536) -> 0xfffffe0219988000  
Apr 19 13:48:49  kernel   bxe0: Found 10Gb Fiber media.  
Apr 19 13:48:49  kernel   bxe0: IFMEDIA flags : 20  
Apr 19 13:48:49  kernel   bxe0: Using defaults for TSO: 65518/35/2048  
Apr 19 13:48:49  kernel   bxe0: Ethernet address: xx:xx:xx:xx:xx:xx
Apr 19 13:48:49  kernel   bxe0: MSI-X vectors Requested 3 and Allocated 3  
Apr 19 13:48:49  kernel   bxe1: <QLogic NetXtreme II BCM57810 10GbE (B0) BXE v:1.78.91  
Apr 19 13:48:49  kernel   > mem 0x2be000000-0x2be7fffff,0x2bd800000-0x2bdffffff,0x2bf800000-0x2bf80ffff irq 17 at device 0.1 on pci6  
Apr 19 13:48:49  kernel   bxe1: PCI BAR0 [10] memory allocated: 0x2be000000-0x2be7fffff (8388608) -> 0xfffffe021ac00000  
Apr 19 13:48:49  kernel   bxe1: PCI BAR2 [18] memory allocated: 0x2bd800000-0x2bdffffff (8388608) -> 0xfffffe021b400000  
Apr 19 13:48:49  kernel   bxe1: PCI BAR4 [20] memory allocated: 0x2bf800000-0x2bf80ffff (65536) -> 0xfffffe021aa98000  
Apr 19 13:48:49  kernel   bxe1: Found 10Gb Fiber media.  
Apr 19 13:48:49  kernel   bxe1: IFMEDIA flags : 20  
Apr 19 13:48:49  kernel   bxe1: Using defaults for TSO: 65518/35/2048  
Apr 19 13:48:49  kernel   bxe1: Ethernet address: xx:xx:xx:xx:xx:xx
Apr 19 13:48:49  kernel   bxe1: MSI-X vectors Requested 3 and Allocated 3

It is recognized in my interfaces as BXE0 and BXE1, I've placed both ports in a LAG with LACP as the LAG protocol. Vlans have been applied.

Now comes the strange part, the interfaces are down and stay down. I cannot seem to get the 10gb interfaces activated...
My MikroTik CRS317 switch sees them as "link inactive" and no traffic goes over it.

As a test to see if it was due to the LAG config, I've removed it and added the 10GB interface directly as an individual interface (OPT7), the result is the same. No life in it.

Screenshots:

Any help in this matter is greatly appreciated!

Edit 1: The new custom server build motherboard is an Intel S1200SPL with 8GB DDR4 ECC and (for the moment) an Intel Celeron G3900. 2x120GB Intel SSDs in RAID 1, a HP Broadcom/Qlogic 530SFP+ 2x10GB NIC and a HP Intel Pro/1000 PT Quad Port 4x1GB NIC

Edit 2: The S1200SPL BIOS is running the latest version: S1200SP.86B.03.01.0042.013020190050

DonZalmrol

So hooked up the server to a monitor and saw this output:
output (Small).png

Seems that either PFSense or BSD is loading in the correct driver.

chpalmer

I would also look into whether the Watchguard BIOS for that machine is limiting the use of that card. They are famous for locking things down.

DonZalmrol

Minor correction, I'm moving away from my Watchguard to a new server. That new server is having the issue with the BXE driver, not the otherway around :)

The new server motherboard is an Intel S1200SPL with 8GB DDR4 ECC and (for the moment) an Intel Celeron G3900. The Celeron will be replaced by a low power Intel Xeon.

That BIOS is running the latest version: S1200SP.86B.03.01.0042.013020190050

DonZalmrol

Ok, so after troubleshooting and not finding it, I've caved and purchased an Intel X520-DA2 SFP+ card.
Installed it, booted up and worked immediately after assigning it to the LAG.

So for future readers, abandon hope and purchase Intel NICs...

PS: If one should ask it works out-of-the-box without a LAG as well.
PPS: If someone would ever find the solution, always welcome to share.