Problem with Broadcom NICs / enabling more than 3 NICs in pfSense [SOLVED]

uboot66 · Nov 30, 2011, 7:16 AM

I got a box which is driving me nuts …
It's a DELL R210 with 2 integrated Broadcom NICs (BCM5716) and as we need 6 NICs we installed a Quad-Port NetXtreme II (BCM5709) into this server, running pfSense 2.0 (64 bit).

When we do a first configuration with 2 NICs (WAN and LAN) the system is running fine. As soon as we enable more than 3 NICs (assign and then enable them via the web GUI) the system is behaving very strange: you can ping the system through the LAN NIC, you can connect to sshd and web GUI but there is no feedback, neither from sshd nor web GUI.

When I remove the additional NICs via pfSense shell and reboot the system everything is running fine again.
I checked boot messages and can see that we have the following mappings:

bce0: <broadcom netxtreme="" ii="" bcm5709="" 1000base-t="" (c0)="">mem 0xd6000000-0xd7ffffff irq 18 at device 0.0 on pci3
miibus0: <mii bus="">on bce0
brgphy0: <bcm5709c 10="" 100="" 1000basetx="" phy="">PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce0: [ITHREAD]
bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 5Gbps); B/C (5.0.11); Flags (MSI)
bce1: <broadcom netxtreme="" ii="" bcm5709="" 1000base-t="" (c0)="">mem 0xd8000000-0xd9ffffff irq 19 at device 0.1 on pci3
miibus1: <mii bus="">on bce1
brgphy1: <bcm5709c 10="" 100="" 1000basetx="" phy="">PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce1: [ITHREAD]
bce1: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 5Gbps); B/C (5.0.11); Flags (MSI)
pcib4: <pci-pci bridge="">at device 4.0 on pci2
pci4: <pci bus="">on pcib4
bce2: <broadcom netxtreme="" ii="" bcm5709="" 1000base-t="" (c0)="">mem 0xd2000000-0xd3ffffff irq 16 at device 0.0 on pci4
miibus2: <mii bus="">on bce2
brgphy2: <bcm5709c 10="" 100="" 1000basetx="" phy="">PHY 1 on miibus2
brgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce2: [ITHREAD]
bce2: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 5Gbps); B/C (5.0.11); Flags (MSI)
bce3: <broadcom netxtreme="" ii="" bcm5709="" 1000base-t="" (c0)="">mem 0xd4000000-0xd5ffffff irq 17 at device 0.1 on pci4
miibus3: <mii bus="">on bce3
brgphy3: <bcm5709c 10="" 100="" 1000basetx="" phy="">PHY 1 on miibus3
brgphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce3: [ITHREAD]
bce3: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 5Gbps); B/C (5.0.11); Flags (MSI)
ehci0: <intel pch="" usb="" 2.0="" controller="" usb-b="">mem 0xdf0fa000-0xdf0fa3ff irq 22 at device 26.0 on pci0
ehci0: [ITHREAD]
usbus0: EHCI version 1.0
usbus0: <intel pch="" usb="" 2.0="" controller="" usb-b="">on ehci0
pcib5: <acpi pci-pci="" bridge="">at device 28.0 on pci0
pci5: <acpi pci="" bus="">on pcib5
bce4: <broadcom netxtreme="" ii="" bcm5716="" 1000base-t="" (c0)="">mem 0xda000000-0xdbffffff irq 16 at device 0.0 on pci5
miibus4: <mii bus="">on bce4
brgphy4: <bcm5709c 10="" 100="" 1000basetx="" phy="">PHY 1 on miibus4
brgphy4:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bce4: [ITHREAD]
bce4: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Flags (MSI|MFW); MFW (NCSI 2.0.11)
bce5: <broadcom netxtreme="" ii="" bcm5716="" 1000base-t="" (c0)="">mem 0xdc000000-0xddffffff irq 17 at device 0.1 on pci5</broadcom></bcm5709c></mii></broadcom></acpi></acpi></intel></intel></bcm5709c></mii></broadcom></bcm5709c></mii></broadcom></pci></pci-pci></bcm5709c></mii></broadcom></bcm5709c></mii></broadcom>

WAN is on bce4, LAN is on bce3. We tried to add different interfaces (e.g. bce0 and bce1 as OPT1 and OPT2) but the result is always the same.
As soon as I enable these new interfaces in the GUI pfSense is no longer responding. No kernel panic, no logs, no idea …
When I install Linux on this machine everything is running fine, so I assume there are problems with the bce(4) driver within FreeBSD.

Any clue what I am doing wrong ? IRQ problems ?
Please help !

wallabybob · Nov 29, 2011, 7:54 PM

I have a vague recollection of seeing reports of some sort of of system resource exhaustion with "too many" Broadcom NICs.

I suggest you start with 2 NICs enabled, establish a ssh session to the pfSense box then collect the output of pfSense shell command netstat -m, enable a NIC, collect the output of pfSense shell command netstat -m etc and post the results as a reply.

When you have 4 NICs enabled can you still get a response from the system console? If so, what do you see when you type the commands netstat -m and top -S -H?

Then I did a search and turned up http://forum.pfsense.org/index.php/topic,35895.0.html which rather sounds like what you reported.

stherr · Nov 29, 2011, 9:24 PM

I ran into a similar problem with my new Dell R310 server this past weekend (2.0-RELEASE-amd64), except all my NICs are Broadcom (bce). The system itself ran fine and the web interface was responsive at first, but after several minutes the GUI and SSH stopped responding. Pings and console access were fine though.

Yesterday I followed the instructions at http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards (modifying /boot/loader.conf.local) and so far pfSense works perfectly.

uboot66 · Nov 30, 2011, 7:14 AM

Hi,

you were both right, sorry for bothering you with that topic.
I followed the tuning instructions in the post of stherr and now everything is working fine.
It seems to be the kern.ipc.nmbclusters problem which wallabybob mentioned in his post.

My apologies for not finding the forum articles you mentioned (I was looking for "problems with multiple NICs").

Many thanks for your quick help !!!