Unable to use BCM57810 properly, need new drivers

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

@Gradius said in Unable to use BCM57810 properly, need new drivers:

https://github.com/JAMESMTL/snippets/tree/master/bnx2x/proxmox

That driver appears to be for Proxmox directly but your pfSense log implies you're passing the NIC hardware through to pfSense so any Proxmox driver would not be used.
Is that what you're doing?

Can I assume what you're attempting to do here is use an incompatible GPON SFP module in the NIC? That seems to be a common use case there.

If so that code was just added to the driver and will be in the next point release:
https://github.com/pfsense/FreeBSD-src/commit/1222560f07264e3beda9bdd2f7116b6dc6f0d204

That's correct. I'm directly passing the card by "PCI device" from Proxmox to pfSense.

It worked before (old version). Now it's really good news to see a new driver (patched) is planned!

stephenw10

Hmm, interesting. Hard to see how that driver could ever have made in difference when passed through like that.

If you don't pass it through you could use the modified driver in Proxmox and then present it to pfSense as a virtual NIC until we get the patched driver in pfSense. Except you're saying the modified driver no longer loads in Proxmox....

Gradius

@stephenw10 I downloaded the source from here:
https://github.com/pfsense/FreeBSD-src

Compiled w/o any problem, however, it won't load as the version doesn't match the pfSense v2.8

Drivers:
https://github.com/pfsense/FreeBSD-src/tree/devel-main/sys/dev/bxe

So there you have it, working drivers is there, just need to compile and up-to-date the kernel with it.

stephenw10

It's in 2.8.1-beta if you can test that.

Gradius

@stephenw10

Hi.

I'm running the beta now: RELENG_2_8_1-n256095-47c932dcc0e9: Tue Jul 29 15:43:27 UTC 2025

However, the 1st port is showing me as 10Gbps, when it should be showing as 1Gbps (as configured on NVRAM).

The 1st port (bxe0) is also where my ODI FTTH SFP+ is. My ISP uses PPPoE, hence the use of it:

bxe0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
        ether 00:1e:1e:xx:xx:xx
        inet6 fe80::xxx:xxxx:xxxx:680c%bxe0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

bxe1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
        description: LAN
        options=120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:1e:1e:xx:xx:xx
        inet 192.168.0.254 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::xxx:xxxx:xxxx:680e%bxe1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

bxe0.757: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 00:1e:1e:xx:xx:xx
        inet6 fe80::xxx:xxxx:xxxxx:680c%bxe0.757 prefixlen 64 scopeid 0xc
        groups: vlan
        vlan: 757 vlanproto: 802.1q vlanpcp: 0 parent interface: bxe0
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pppoe0: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        description: WAN
        options=0
        inet 152.172.xxx.xxx --> 10.49.216.3 netmask 0xffffffff
        inet6 fe80::xxxx:xxxx:fe8f:680c%pppoe0 prefixlen 64 scopeid 0xd
        groups: pppoec
        nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>

Finally, on Interfaces > LAN (bxe1, for example), is only showing me 10Gbps under "Speed and Duplex", when I have it fully configured on NVRAM as 1, 2.5 and 10Gbps options, or 35=70 and 36=70 (by using ediag -b10eng).

Thanks.

stephenw10

The speeds shown can often not accurately reflect the link speed for unusual modules. I've never tested that NIC though.

Is it actually linked and passing traffic though?

Did it previously show the correct speeds with a custom module?

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

The speeds shown can often not accurately reflect the link speed for unusual modules. I've never tested that NIC though.

Is it actually linked and passing traffic though?

Yes, is working fine, but I haven't tested 2.5Gbps yet, just 1Gbps. The main problem was with 2.5Gbps. And the firmwares available for ODI won't let me pass PPPoE for some reason, I'll check with the guys on other (ODI) forum. I use the original firmware and works fine, but only at 1Gbps.

Did it previously show the correct speeds with a custom module?

No. However with the old driver (on pfsense v2.5 if I recall), it did show as it should.

stephenw10

Hmm, so that was using a custom kernel module (driver) in pfSense 2.5.X?

The patch code sure looks to set 2.5G as an available speed in the driver but I'd expect the actual presented speeds to be dependent on the SFP module inserted.

Gradius

@stephenw10 Just tested now.

It works, however, you need to unplug and the plug in again (power ON/OFF the SFP).

If you let the SFP there, it will not "wake up".

Then I configured the NIC back to 1Gbps, while the ODI was in 2.5Gbps mode, I got this error:

It "wake up" few seconds after.

I went back to 1Gbps, because it has an upload bug while in 2.5Gbps mode (it gets slow), something from ODI, not the NIC driver.

stephenw10

@Gradius said in Unable to use BCM57810 properly, need new drivers:

It works, however, you need to unplug and the plug in again (power ON/OFF the SFP).

Like if you reboot pfSense you need to reconnect the module? Full power cycle?

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

@Gradius said in Unable to use BCM57810 properly, need new drivers:

It works, however, you need to unplug and the plug in again (power ON/OFF the SFP).

Like if you reboot pfSense you need to reconnect the module? Full power cycle?

No, I just need to plug out the SFP module, and then plug in, so power on & off just the module, then it works.

I just confirmed now, if the NIC is at 1Gbps (NVRAM), I don't need to do that, it only happens at 2.5Gbps.

And right after (that picture), I got 1 warning:
pppoe0: received unexpected PADO

Gradius

More here:

stephenw10

But to get into the situation where you have the reseat the module you reboot pfSense?

Like something during a warm boot puts the NIC/module into an invalid mode.

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

But to get into the situation where you have the reseat the module you reboot pfSense?

Like something during a warm boot puts the NIC/module into an invalid mode.

No, I don't reboot pfSense at all, I just get out the SFP module for a few seconds, and then I insert it again.

If I reboot the entire system, it refuses to bring the SFP up ONLY IF it's in 2.5Gbps mode.

If it's in 1Gbps mode, it works just fine.

Now I made a custom FW (scripts to fix 2.5Gbps speed on ODI module), and when I reboot the entire system, I don't see bxe0 at all, ONLY when I remove the SFP module for a few seconds, is when pfSense is able to recognize it, and even shows the correct speed:

Something must be missing in driver, only when is in 2.5Gbps mode, it won't "internally reset", or something like that, to bring it up that interface on.

stephenw10

Hmm, I'm confused then. What is it that prompts you to re-insert the module if not rebooting?

It looks like it does it at boot from the screenshots.

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

Hmm, I'm confused then. What is it that prompts you to re-insert the module if not rebooting?

It looks like it does it at boot from the screenshots.

I got that "2500 Mbps" AFTER I did removed the SFP and inserted again.

If I let it there, it would never "go up".

stephenw10

But to be clear that's the state it's in after boot? Not after changing the link speed or some other manual change?

There's a good chance something is omitted during the boot process that otherwise gets run when the NIC comes up.

Gradius

@stephenw10 said in Unable to use BCM57810 properly, need new drivers:

But to be clear that's the state it's in after boot? Not after changing the link speed or some other manual change?

There's a good chance something is omitted during the boot process that otherwise gets run when the NIC comes up.

Hi,

Well, I need to enter into ODI cli and configure it manually the speed (I do that on a media converter) via putty (telnet):

flash set LAN_SDS_MODE 6

To check:
cat /proc/lan_sds/lan_sds_cfg
lan_sds_mode = 6(2500BaseX)

So, I perform a reboot, test it again, see that is working, only then I put it back on BCM57810.

On BCM57810's side, I need to have enabled via UEFI, options 56 and 59 to 6:

Under UEFI boot:
ediag -b10eng

dev 1 (port 1 where the ODI will be used)
nvm cfg (enter config)
7 (on menu option, then)
56=6
59=6
save (NVRAM save)
exit

Options are:
5 for 1Gbps
6 for 2.5Gbps
7 for 10Gbps

The only values you only need to setup once are 35 and 36 to 70, because 70 means: 1Gbps, 2.5Gbps and 10Gbps auto mode speeds.

I managed to get a more up-to-date firmware for ODI, I'll try with that after, because the version I use, if I let it in 2.5Gbps mode, and my BCM is in 1Gbps mode, the ODI isn't capable to switch back to 1Gbps automatically. So the interface will never "wakes up".

Also (as mentioned before), there's a bug on ODI part, in 2.5Gbps mode, the upload speed gets really slow, that I will check (if they fixed) with the last firmware release (june 2025). And then, I will check under pfSense boot if it will "wake up", or not, w/o touching ODI.

Gradius

Ok, my final test (at least on RC1)...

I'm running the last ODI firmware now, and I'm using:

flash set LAN_SDS_MODE 5 [ lan_sds_mode = 5(HiSGMII MAC) ]

This forces 2.5Gbps all the time.

Exiting from UEFI, and starting POST, once the system goes fully up, the interface (SFP) doesn't want to come up, so I have to remove it, and insert it again, then is able to "power on", and after a while, it gets PPPoE and everything works fine.

If I perform a reboot on Proxmox, the interface is able to come up w/o doing anything more, howerver, I got those errors before it got a public IP from PPPoE:

After is UP, no more errors, or anything.

stephenw10

Ah, interesting so the NIC is passed through to pfSense in Proxmox? I'd forgotten that! So when pfSense is rebooted the NIC may not be reset in the same way it would be running bare metal.