SYS-501S-M server with AOC-STG-B4S

minibr

Hi all,

I would like to ask you for help, or solution with BXE Freebsd driver problem.

We have installed pfsence 2.3.2-p1. We are using Supermicro SYS-501S-M server with AOC-STG-B4S (Broadcom BCM57840S controller – 4xSFP+10Gbps) Server boot up without any problem, but after moving about 2 gigabytes, all SFP moduls goes down with this kind of mystake.

bxeX: ERROR: FW faild to respond!

bxeX: failed to release tx mbufs (28 left)

bxeX: Parity error: HW block parity attention:

[0]: 0x55540000 [1]:0x55555555 [2]:0x00005555 [3]:0xf0000000 [4]:0x00000028

bxeX: ERROR: resource (0xb) in use (status 0xffffffff bit 0x800)

bxeX: ERROR: Recovery flow hasn't properly completed yet, try again
later. If you still see this message after a few retries then power
cycle is required.

bxeX: ERROR: Initialization failed, stack notified driver is NOT running!

Thank you in advance for your answer

stephenw10

Hi Michal,

First check you are not exhausting the mbufs. I would expect to see an error indicating that if you are but not always. Check the monitoring graphs for mbuf levels.
You can also run:

netstat -m

to see the current levels.

Try disabling ALL the network hardware offloading in System > Advanced, Networking. That's probably only 'Hardware Checksum Offloading' the others should be disabled by default,

Try disabling MSI/MSI-X for the bxe driver as described here:
https://www.freebsd.org/cgi/man.cgi?query=bxe&apropos=0&sektion=0&manpath=FreeBSD+10.3-RELEASE+and+Ports&arch=default&format=html

Add the following line to /boot/loader.conf.local (create that file if it doesn't exist):

hw.bxe.interrupt_mode="0"

Those are pretty standard debugging options for NICs that don't behave. That driver appears to have a number of other loader options that might be applicable.

Steve

minibr

Hi Steve,
Thank you very much for your answer. I tried everything possible, but all with the same result. Finally helped technical support from Supermicro. I attach answer:

Anyway it is a good idea to update the NIC’s firmware to the latest version.

You could start with the latest official release, would the issue persist, please use the latest BETA.
STGb4s13.zip (Latest official release)
STGb4s16.zip (Latest BETA release)
Please download the following file from our SFTP:
Filename: STGb4s13.zip and STGb4s16.zip

So far we have only tested the firmware labeled as beta. And everything seems to be working. Next problem is an extreme load on the system. We only NAT otherwise everything at the default settings. When testing 750Mbps TCP Halfduplex, CPU usage to 23 percent all cores.

If it could help someone, I attach firmware. But it seems to be exchanged card for intel chipset for me.

STGb4s13.zip
STGb4s16.zip

stephenw10

The SYS-501S-M doesn't seem to be an actual Supermicro model, maybe the SYS-5019S-M?

What CPU are you running in it?

23% on every core does seem quite a large load. How is that breaking down? Mostly interrupt?

Steve

minibr

I'm sorry for the mistake. It is a model SYS-5019S-M. Complete set is:

Supermicro AOC-STG-B4S
Supermicro SSD-DM032-PHI
Supermicro SYS-5019S-M
INTEL Quad-Core Xeon E3-1220V5 3.0GHZ/8MB/LGA1151/Skylake
Samsung Memory - 8GB DDR4-2133 1Rx4 ECC REG RoHS, M391A1G43DB0-CPB, Supermicro certified.

yes it's 23% only interrupt.

stephenw10

Hmm, that does seem quite high stil but there are a lot of variables. It may be correct for your test method and CPU.

You might check that it's actually running at its rated frequency. We have seem some CPUs that run at their lowest speed unless powerd is enabled.

Steve