Upgrade from 2.1.5 to 2.2.3 - Configuring interface hangs during boot

montara

Hey all.

Today I decided to upgrade my secondary pfSense firewall and it failed to come up after the reboot. I have attached a screenshot of the console.

The hardware is a Dell R710 with way too much CPU (24 cores) and memory (96GB). My primary firewall is identical hardware and is running 2.1.5 just fine. The wan, lan, and sync interfaces are bce and the other interfaces are VLANs on a 10Gb ix interface. It was previously hanging on the SYNC interface, which was bce3. I booted into single user mode and disabled that interface and now it's hanging on SAN_A, which is a VLAN on ix0. CARP is configured on all of the interfaces except for the SYNC interface.

Any ideas of what I can check? I've done quite a bit of google searching and haven't come up with anything about this particular situation. As you can see, it gets past the interface microcode so it's probably not related to that issue. Any help is greatly appreciated.

fw_interfaces_broken.png_thumb

montara

I have made some progress, but it's still hanging during boot. I booted it up in single user mode and disabled all of the interfaces except for WAN and LAN. Now it gets past the interface configuration step and hangs just after "Configuring CRON…done."

I've attached another screenshot.

pfsense_configuring_cron_hang.png_thumb

cmb

What's your mbuf limit? The new Intel driver in 2.2.x is probably hungrier there, and having the limit way too low for 10G cards would cause what you describe.
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

montara

kern.ipc.nmbclusters was set to 0. I changed it to "1000000" as the document suggested. I have also set kern.ipc.nmbjumbop to the advised value. It was previously unset.

I also set hw.bce.tso_enable=0 and hw.pci.enable_msix=0 for the bce cards.

The box just completed the boot up! Thanks for the suggestions.

cmb

Great. Surprised that worked previously, the default limits generally much too low for 10G cards.

The bce tunables usually aren't necessary unless you're seeing problems of some sort, but shouldn't hurt anything to set them.