Fresh install WebConfigurator hangs

Hans-Maier

Hi dogbait,

have the same problem with the MBUF which is at 100%.
And the other thing is that igb1 does not come up with the message:
"igb1: Could not setup receive structures"

I already had a look on a BSD-forum entry from 2010 (four years old)
http://lists.freebsd.org/pipermail/freebsd-stable/2010-October/059534.html

which says that I could modify the value for num_queues in if_igb.c which is really weired.

Is there no solution for this? This makes the whole thing really unusable!

PLEASE HELP me!

Thanx!

Hans-Maier

The fastest solution for me was to enter the BIOS and
LIMIT the CPU kernels to 4!

WTH is this? It works fine now …
leaving me with 4 unused cores!

JeGr

Hi Hans,

as you have igb interfaces, can you try this:

login to pfSense with SSH and edit the file (via vi) "/boot/loader.conf.local" (or open the file via Diagnostic menu). Then give it the following content:


kern.ipc.nmbclusters="655356"
machdep.hyperthreading_allowed="1"
hw.igb.num_queues="4"

and try to reboot. Does it help?

stephenw10

The recommended settings are different from those:

https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_Cards

Do you have reason to use those?

Also: https://forum.pfsense.org/index.php?topic=69486
Steve

JeGr

Because they work for us. ;)

We had initial issues even installing pfSense 2.0.x on our brand new ibm hardware servers and intel/broadcast nics. We couldn't even get it to install nor to recognize the nics in the system because the cpu (8cores) had HT and the number of cores available to the system drew the mbufs into crazyness :)

Those recommendations came from an older redmine bug. See: https://redmine.pfsense.org/issues/1221#note-8

Greets

stephenw10

Ah, yes. However note that in the bug report that setting was tested with a uniprocessor kernel and hyperthreading disabled. The current kernels are all SMP.
If it works for you though then that's really what matters. So you tested these settings with 16 logical CPUs?

Steve

dogbait

OK…so what's the consensus. Our pfSense box has 8 CPU cores, one onboard i354 Intel Network Controller with 4 ports and one PCIe Intel VT Quad Network Controller with 4 ports...

Do we disable 4 cores AND set hw.igb.num_queues to 4 or 1?

stephenw10

You shouldn't need to disable any cores though 8 cores probably won't give you much advantage over 4 in 2.1.X. The new pf in 2.2 will use multiple cores much better.
I would go with JeGr's suggested settings since they have been tested to work with similar cores/ports.

Steve

JeGr

As we re-used some similar hardware, we switched two cpus to other cluster members. So ATM our pair of firewalls are only running a single cpu with 4 cores and HT. So 8 visible to the system at the moment (as PF isn't using SMB now but hopefully will scale much better with FreeBSD 10.x underneath). But our settings are still the same.

So yes we tested originally with 16 visible cores, now have 8 and the system is running good so far. MBUF usage is about 5% (31638/655356) and all of the 8 igb seem fine wether they run single (as igbX interface) or in a LAGG group.

Greets
Jens

dogbait

Thanks guys, appreciate the info.

Just to add, I reduced the available cores in the BIOS from 8 to 4 and the MBUF utilization went from 100% to 50%.