Unable to use BCM57810 properly, need new drivers

Gradius

I'm trying to using my BCM57810 (2x SFP+ ports) and pfSense with FTTH again.

It was working perfectly fine on older version, Proxmox v8.0.4, then around 6 months ago, I lost all data (bad HDD), and I didn't care as I wasn't using it at that time (totally busy with work).

Now, it refuses to work installing everything from scratch (and up-to-date). I even tried to compile the driver, so far no luck:
https://github.com/JAMESMTL/snippets/tree/master/bnx2x/proxmox

Yes, the one there for v8.0.4 will NOT load/works.

Under proxmox it "seems" fine, but when I try to use on pfSense v2.8, I got a lot of errors:

[2.8.0-RELEASE][admin@pfSense.home.arpa]/: dmesg | grep -i bxe
bxe0: <QLogic NetXtreme II BCM57810 10GbE (B0) BXE v:1.78.91> mem 0xfa000000-0xfa7fffff,0xfa800000-0xfaffffff,0xfc000000-0xfc00ffff irq 16 at device 0.0 on pci1
bxe0: PCI BAR0 [10] memory allocated: 0xfa000000-0xfa7fffff (8388608) -> 0xfffff800fa000000
bxe0: PCI BAR2 [18] memory allocated: 0xfa800000-0xfaffffff (8388608) -> 0xfffff800fa800000
bxe0: PCI BAR4 [20] memory allocated: 0xfc000000-0xfc00ffff (65536) -> 0xfffff800fc000000
bxe0: ERROR: Invalid SHMEM validity signature: 0x00000008
bxe0: ERROR: Invalid phy config in NVRAM (PHY1=0x00000008 PHY2=0x00000008)
bxe0: Unknown media!
bxe0: IFMEDIA flags : 20
bxe0: Using defaults for TSO: 65518/35/2048
bxe0: ERROR: FW failed to respond!
bxe0: ERROR: MCP response failure, aborting
bxe0: ERROR: Failed to unload previous driver! time_counter 10 rc -1
bxe1: <QLogic NetXtreme II BCM57810 10GbE (B0) BXE v:1.78.91> mem 0xfb000000-0xfb7fffff,0xfb800000-0xfbffffff,0xfc010000-0xfc01ffff irq 17 at device 0.1 on pci1
bxe1: PCI BAR0 [10] memory allocated: 0xfb000000-0xfb7fffff (8388608) -> 0xfffff800fb000000
bxe1: PCI BAR2 [18] memory allocated: 0xfb800000-0xfbffffff (8388608) -> 0xfffff800fb800000
bxe1: PCI BAR4 [20] memory allocated: 0xfc010000-0xfc01ffff (65536) -> 0xfffff800fc010000
bxe1: ERROR: Invalid SHMEM validity signature: 0x00010008
bxe1: ERROR: Invalid phy config in NVRAM (PHY1=0x00010008 PHY2=0x00010008)
bxe1: Unknown media!
bxe1: IFMEDIA flags : 20
bxe1: Using defaults for TSO: 65518/35/2048
bxe1: ERROR: FW failed to respond!
bxe1: ERROR: MCP response failure, aborting
bxe1: ERROR: Failed to unload previous driver! time_counter 10 rc -1
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
vlan0: changing name to 'bxe0.20'
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
bxe0: promiscuous mode enabled
bxe0.20: promiscuous mode enabled
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
bxe0: promiscuous mode disabled
bxe0.20: promiscuous mode disabled
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
bxe0: WARNING: No power capability
bxe0: ERROR: Failed to create dma tag for 'ECORE_ILT' (22)
bxe0: ERROR: ecore_ilt_mem_op ILT_MEMOP_ALLOC failed
bxe0: ERROR: Initialization failed, stack notified driver is NOT running!
bxe0: promiscuous mode enabled
[2.8.0-RELEASE][admin@pfSense.home.arpa]/:

ATM, I'm only using one port (first one) with an ODI DFP-34X-2C2, the other one is disconnected for now.

Keep in mind, I had it working perfectly before data loss with older Proxmox version (v8.0.4), and older pfSense.

Any help in this matter is greatly appreciated!

stephenw10

@Gradius said in Unable to use BCM57810 properly, need new drivers:

https://github.com/JAMESMTL/snippets/tree/master/bnx2x/proxmox

That driver appears to be for Proxmox directly but your pfSense log implies you're passing the NIC hardware through to pfSense so any Proxmox driver would not be used.
Is that what you're doing?

Can I assume what you're attempting to do here is use an incompatible GPON SFP module in the NIC? That seems to be a common use case there.

If so that code was just added to the driver and will be in the next point release:
https://github.com/pfsense/FreeBSD-src/commit/1222560f07264e3beda9bdd2f7116b6dc6f0d204

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

@Gradius said in Unable to use BCM57810 properly, need new drivers:

https://github.com/JAMESMTL/snippets/tree/master/bnx2x/proxmox

That driver appears to be for Proxmox directly but your pfSense log implies you're passing the NIC hardware through to pfSense so any Proxmox driver would not be used.
Is that what you're doing?

Can I assume what you're attempting to do here is use an incompatible GPON SFP module in the NIC? That seems to be a common use case there.

If so that code was just added to the driver and will be in the next point release:
https://github.com/pfsense/FreeBSD-src/commit/1222560f07264e3beda9bdd2f7116b6dc6f0d204

That's correct. I'm directly passing the card by "PCI device" from Proxmox to pfSense.

It worked before (old version). Now it's really good news to see a new driver (patched) is planned!

stephenw10

Hmm, interesting. Hard to see how that driver could ever have made in difference when passed through like that.

If you don't pass it through you could use the modified driver in Proxmox and then present it to pfSense as a virtual NIC until we get the patched driver in pfSense. Except you're saying the modified driver no longer loads in Proxmox....

Gradius

@stephenw10 I downloaded the source from here:
https://github.com/pfsense/FreeBSD-src

Compiled w/o any problem, however, it won't load as the version doesn't match the pfSense v2.8

Drivers:
https://github.com/pfsense/FreeBSD-src/tree/devel-main/sys/dev/bxe

So there you have it, working drivers is there, just need to compile and up-to-date the kernel with it.

stephenw10

It's in 2.8.1-beta if you can test that.

Gradius

@stephenw10

Hi.

I'm running the beta now: RELENG_2_8_1-n256095-47c932dcc0e9: Tue Jul 29 15:43:27 UTC 2025

However, the 1st port is showing me as 10Gbps, when it should be showing as 1Gbps (as configured on NVRAM).

The 1st port (bxe0) is also where my ODI FTTH SFP+ is. My ISP uses PPPoE, hence the use of it:

bxe0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:1e:1e:xx:xx:xx
        inet6 fe80::xxx:xxxx:xxxx:680c%bxe0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

bxe1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
        description: LAN
        options=120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:1e:1e:xx:xx:xx
        inet 192.168.0.254 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::xxx:xxxx:xxxx:680e%bxe1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

bxe0.757: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 00:1e:1e:xx:xx:xx
        inet6 fe80::xxx:xxxx:xxxxx:680c%bxe0.757 prefixlen 64 scopeid 0xc
        groups: vlan
        vlan: 757 vlanproto: 802.1q vlanpcp: 0 parent interface: bxe0
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pppoe0: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        description: WAN
        options=0
        inet 152.172.xxx.xxx --> 10.49.216.3 netmask 0xffffffff
        inet6 fe80::xxxx:xxxx:fe8f:680c%pppoe0 prefixlen 64 scopeid 0xd
        groups: pppoec
        nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>

Finally, on Interfaces > LAN (bxe1, for example), is only showing me 10Gbps under "Speed and Duplex", when I have it fully configured on NVRAM as 1, 2.5 and 10Gbps options, or 35=70 and 36=70 (by using ediag -b10eng).

Thanks.

stephenw10

The speeds shown can often not accurately reflect the link speed for unusual modules. I've never tested that NIC though.

Is it actually linked and passing traffic though?

Did it previously show the correct speeds with a custom module?

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

The speeds shown can often not accurately reflect the link speed for unusual modules. I've never tested that NIC though.

Is it actually linked and passing traffic though?

Yes, is working fine, but I haven't tested 2.5Gbps yet, just 1Gbps. The main problem was with 2.5Gbps. And the firmwares available for ODI won't let me pass PPPoE for some reason, I'll check with the guys on other (ODI) forum. I use the original firmware and works fine, but only at 1Gbps.

Did it previously show the correct speeds with a custom module?

No. However with the old driver (on pfsense v2.5 if I recall), it did show as it should.

stephenw10

Hmm, so that was using a custom kernel module (driver) in pfSense 2.5.X?

The patch code sure looks to set 2.5G as an available speed in the driver but I'd expect the actual presented speeds to be dependent on the SFP module inserted.

Gradius

@stephenw10 Just tested now.

It works, however, you need to unplug and the plug in again (power ON/OFF the SFP).

If you let the SFP there, it will not "wake up".

Then I configured the NIC back to 1Gbps, while the ODI was in 2.5Gbps mode, I got this error:

It "wake up" few seconds after.

I went back to 1Gbps, because it has an upload bug while in 2.5Gbps mode (it gets slow), something from ODI, not the NIC driver.

stephenw10

@Gradius said in Unable to use BCM57810 properly, need new drivers:

It works, however, you need to unplug and the plug in again (power ON/OFF the SFP).

Like if you reboot pfSense you need to reconnect the module? Full power cycle?

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

@Gradius said in Unable to use BCM57810 properly, need new drivers:

It works, however, you need to unplug and the plug in again (power ON/OFF the SFP).

Like if you reboot pfSense you need to reconnect the module? Full power cycle?

No, I just need to plug out the SFP module, and then plug in, so power on & off just the module, then it works.

I just confirmed now, if the NIC is at 1Gbps (NVRAM), I don't need to do that, it only happens at 2.5Gbps.

And right after (that picture), I got 1 warning:
pppoe0: received unexpected PADO

Gradius

More here:

stephenw10

But to get into the situation where you have the reseat the module you reboot pfSense?

Like something during a warm boot puts the NIC/module into an invalid mode.

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

But to get into the situation where you have the reseat the module you reboot pfSense?

Like something during a warm boot puts the NIC/module into an invalid mode.

No, I don't reboot pfSense at all, I just get out the SFP module for a few seconds, and then I insert it again.

If I reboot the entire system, it refuses to bring the SFP up ONLY IF it's in 2.5Gbps mode.

If it's in 1Gbps mode, it works just fine.

Now I made a custom FW (scripts to fix 2.5Gbps speed on ODI module), and when I reboot the entire system, I don't see bxe0 at all, ONLY when I remove the SFP module for a few seconds, is when pfSense is able to recognize it, and even shows the correct speed:

Something must be missing in driver, only when is in 2.5Gbps mode, it won't "internally reset", or something like that, to bring it up that interface on.

stephenw10

Hmm, I'm confused then. What is it that prompts you to re-insert the module if not rebooting?

It looks like it does it at boot from the screenshots.

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

Hmm, I'm confused then. What is it that prompts you to re-insert the module if not rebooting?

It looks like it does it at boot from the screenshots.

I got that "2500 Mbps" AFTER I did removed the SFP and inserted again.

If I let it there, it would never "go up".

stephenw10

But to be clear that's the state it's in after boot? Not after changing the link speed or some other manual change?

There's a good chance something is omitted during the boot process that otherwise gets run when the NIC comes up.

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

But to be clear that's the state it's in after boot? Not after changing the link speed or some other manual change?

There's a good chance something is omitted during the boot process that otherwise gets run when the NIC comes up.

Hi,

Well, I need to enter into ODI cli and configure it manually the speed (I do that on a media converter) via putty (telnet):

flash set LAN_SDS_MODE 6

To check:
cat /proc/lan_sds/lan_sds_cfg
lan_sds_mode = 6(2500BaseX)

So, I perform a reboot, test it again, see that is working, only then I put it back on BCM57810.

On BCM57810's side, I need to have enabled via UEFI, options 56 and 59 to 6:

Under UEFI boot:
ediag -b10eng

dev 1 (port 1 where the ODI will be used)
nvm cfg (enter config)
7 (on menu option, then)
56=6
59=6
save (NVRAM save)
exit

Options are:
5 for 1Gbps
6 for 2.5Gbps
7 for 10Gbps

The only values you only need to setup once are 35 and 36 to 70, because 70 means: 1Gbps, 2.5Gbps and 10Gbps auto mode speeds.

I managed to get a more up-to-date firmware for ODI, I'll try with that after, because the version I use, if I let it in 2.5Gbps mode, and my BCM is in 1Gbps mode, the ODI isn't capable to switch back to 1Gbps automatically. So the interface will never "wakes up".

Also (as mentioned before), there's a bug on ODI part, in 2.5Gbps mode, the upload speed gets really slow, that I will check (if they fixed) with the last firmware release (june 2025). And then, I will check under pfSense boot if it will "wake up", or not, w/o touching ODI.