New 2.3.3 install with config restore hangs on "setting up interfaces microcode"
-
Background –
We're running a failover pair of pfSense instances currently on 2.1.5. I'm attempting to replace the secondary router (previously an ESXi 5.0 VM) with a new 2.3.3 ESXi 6.5 VM by restoring backed up config. I'm doing this prior to upgrading the primary (on bare metal) to 2.3.3, for safety purposes.
I install fresh install of pfSense 2.3.3 on the VM with a single E1000 virtual NIC. I set up the bare minimum VLAN interface so that I can access the web UI and restore the config. When I restore the config, the VM reboots, and then hangs at "setting up interfaces microcode."
The config specifies 17 interfaces, all VLAN interfaces based on the single NIC (em0). Additionally, there are 33 CARP addresses in the config as well as 2 IP aliases that are not CARP.
I used the same procedure to replace 2 other pfSense routers in another site with no problems. Those also have multiple VLAN interfaces and CARP addresses, but not as many.
Of note may be that if I change the virtual NIC type in the VM to one of the VMXNET types, the boot proceeds normally, but I have no connectivity to or from the VM. In those cases, pfSense still uses the em driver. I didn't necessarily expect that to work, but just tried it out of curiosity.
Any ideas? I'm fine just running on the bare metal primary right now, but I'm wary of even trying to upgrade it because of this issue.
And, for anyone who may ask, the reason I chose to start with a new VM rather than an upgrade on the old one is that the old one didn't have enough disk space.
-
Hmm, I would expect pfSense to detect those interfaces as vmx if that's what ESXi is set to. I would also recommend using those if possible.
However that may not be possible if the primary is using em with real hardware. If it's not you have other problems anyway coming from 2.1.5.
https://doc.pfsense.org/index.php/Redundant_Firewalls_Upgrade_Guide#pfSense_2.2.x_and_pfsyncSteve
-
Hmm, I would expect pfSense to detect those interfaces as vmx if that's what ESXi is set to. I would also recommend using those if possible.
However that may not be possible if the primary is using em with real hardware. If it's not you have other problems anyway coming from 2.1.5.
https://doc.pfsense.org/index.php/Redundant_Firewalls_Upgrade_Guide#pfSense_2.2.x_and_pfsyncSteve
The primary is using lagg0 as the parent for the vlan interfaces. The physical NICs are using the bce driver. I can certainly try a search and replace in the config file on the secondary replacing "em" with "vmx" and see what happens, of course with the virtual NIC set appropriately. Perhaps if I set it prior to installing pfSense and then upload the modified config file. The good news is that with a working primary I have the luxury of time to experiment.
The other significant thing, I suppose, is that most of this config (the 17 interfaces, etc) is cruft left over from when this site was the primary site in the business. Most of the network infrastructure has since moved to another site, and I can prune this config back to the minimum necessary. However, I was hoping to do that after the upgrade. If I have to do it before, I will. The only reason I think the config might be relevant is that when I first googled "setting up interfaces microcode" I was directed to a much older thread in this same forum that referenced the same issue also on ESXi and also with a lot of CARP addresses in the config. Could be a red herring.
-
If that is an issue with ESXi virtual NICs it's not one I'm aware of, it isn't common.
If you're running lagg0 on both nodes that should be OK as noted in the doc there.
Steve
-
If you're running lagg0 on both nodes that should be OK as noted in the doc there.
Steve
I'm not. The physical server uses link aggregation in pfSense. The virtual secondary does not; the link aggregation is left up to the ESXi host.
Thanks for the help.
-
I'm happy to report that starting with a fresh vm with a single VMXNET interface and restoring a backup config with the interface assignments manually modified to reflect the change from the 'em' driver to the 'vmx' driver works fine.
Not sure what it is about my config and 'em' that it isn't happy with, but i have a working secondary on 2.3.3 now!
Matt
-
Also happy to report that an upgrade from 2.1.5 to 2.3.3 on the primary went smoothly.
-
Thanks for the update. :)
Steve
