BRCM 10G/GbE 2+2P 57800-t rNDC Pfsense problem

JoeriBe

Hi,

I have a problem with this NIC on two Pfsense servers.

The NIC's are shutting down and following message spams the console:

bxe1: WARNING: Interface link is down, ignoring transmit request

Serverhardware is Dell, with NIC type BRCM 10G/GbE 2+2P 57800-t rNDC (new servers)

I have this in the /boot/loader.conf.local


hw.bce.tso_enable=0
kern.cam.boot_delay=10000
kern.ipc.nmbclusters="1048576"
net.inet.tcp.tso=0
hw.pci.enable_msix=0
kern.ipc.nmbjumbop="1048576"
net.isr.bindthreads=0
net.isr.maxthreads=1
kern.random.sys.harvest.ethernet=0
kern.random.sys.harvest.point_to_point=0
kern.random.sys.harvest.interrupt=0
net.isr.defaultqlimit=2048
net.isr.maxqlimit=40960

Any idea what can cause the problem?

Pfsense version:


2.2.4-RELEASE (amd64) 
built on Sat Jul 25 19:57:37 CDT 2015 
FreeBSD 10.1-RELEASE-p15

Serverhardware is Dell, with NIC type BRCM 10G/GbE 2+2P 57800-t rNDC (new servers)

Could it be that this are Broadcom NICs they are perhaps also not supported by pfSense?

Anyway the real failure message is that the link is not up, do you have any problems with the Internet Connection.

cmb

Disable NPAR maybe?
https://lists.freebsd.org/pipermail/freebsd-questions/2015-March/264580.html

JoeriBe

NPAR is disabled on the NIC ports.

Most times after booting the servers, the NIC's are working, but at random times, sometimes, 24 hours or 36 hours the one of the NIC's are starting to post console messages like this:

bxe1: WARNING: Interface link is down, ignoring transmit request

Could also be bxe0.

ifconfig bxe0 down && ifconfig bxe1 down works, wait like 3 minutes and than ifconfig bxe0 up && ifconfig bxe1 up resolves the problem for again like 24 hours.

Could this be some kind of memory issue?

cmb

That's generally the sign of a driver issue, where an ifconfig down && up resolves things for some period of time. Reporting to freebsd-net list with specifics, or filing a FreeBSD PR after replicating with stock 11-CURRENT assuming it's replicable there, would be your best bet. Where it doesn't impact hardware we sell, we don't get much into driver issues.