PfSense does not boot on VMware hardware version 11

ChristianHaag · Apr 24, 2015, 7:43 AM

pfSense was running fine on hardware version 10 of vSphere ESXi 5.5. After an upgrade to vSphere ESXi 6.0 and upgrading to hardware version 11, pfSense does not boot anymore.

It hangs right at the beginning, where the "F1" is displayed and a simulated rotating bar should turn, but does not move. I observed a high CPU load.

Anyone with an idea what to do?

heper · Apr 24, 2015, 10:10 AM

don't have 6 running, don't even have hwversions higher then 9

edit the .vmx file and change it back to virtualHW.version = "10"

ChristianHaag · Apr 24, 2015, 10:23 AM

I just restored the backup I made before upgrading. Indeed, editing the .vmx file is a good idea, even faster, thanks.

Of course it does not hurt running the VM with HW 10 for a while. Maybe my post helps others to not get trapped the same way I did. Or probably someone has an idea what to do to go to HW 11.

johnpoz · Apr 24, 2015, 12:09 PM

Nonsense!!

You have something else going on, runs fine on hardware version 11. Why would it not?

I did the same thing weeks ago when 6 came out.. Upgraded to 6, updated all my vms to 11. No issues with any of the vms. I have also been installing clean versions of pfsense for testing 2.2.1 and now 2.2.2 and they boot and install fine. Have done this prob 10 times over the last couple of weeks. See pftest and pfdhcptest are also both pfsense that boot up just fine on 11.

ChristianHaag · Apr 24, 2015, 12:09 PM

Good to know it's working for you. Thanks for the info. Did you set up the VM recently (and have newer virutal hardware) or did you upgrade it through a couple of vSphere versions? I have for example a E1000 network adapter.

johnpoz · Apr 24, 2015, 12:19 PM

I just updated my post to give details. I have multiple vms of pfsense all running on 11, why would you use e1000 when freebsd 10.1 has native support for vmx3.. But sure I can fire up a vm using that.. Give me one sec.

edit: Here you go pfsense on hardware 11 booting the installer with e1000, will finish the install.. But there is no issues with running pfsense on 11 that I have seen.

edit2: Just added it fully booted and working

ChristianHaag · Apr 24, 2015, 12:14 PM

Thanks a lot. No need to look it up, thanks anyway. Your information is perfect for me to try that route.

ChristianHaag · Apr 28, 2015, 12:38 PM

Thanks again for the advices. Here's what I did from that point:

I cloned the pfSense VM and upgraded the virtual hardware. Result: no boot.
Then I mounted the live CD as boot volume. The purpose of the test was to have a clean boot from a current version using the same VM settings. Result: no boot.
As a cross check I set up a new VM (of course using HW 11) and mounted the live CD as boot volume: Result: boot.

After the virtual hardware upgrade, many entries of the .vmx file were changed. The new VM has a third "version" of the .vmx file. 17 entries are no longer present, 2 are new. The original .vmx file roots in the inital setup of pfSense in December 2012 and I pulled it through the ESXi versions since then.

Instead of figuring out what the "critical" differences in the .vmx file are, I think I am going to set up a new VM and restore the configuration. I'd like to change to 64 bits anyway and maybe that's a good opportunity….

johnpoz · Apr 28, 2015, 1:52 PM

So your saying if I install to hw version ? and then upgrade it to 11 it wont boot? What original hw version, because when I upgraded from 5.5 to 6. I upgraded that previous pfsense to 11 and it booted.

ChristianHaag · Apr 30, 2015, 11:23 AM

Yes, that's exactly what I am saying…

With the first installation of pfSense in 2012 I have used the HP OEM version ESXi 5.1, Oct 2012. Filename was VMware-ESXi-5.1.0-799733-HP-5.32.5.iso. I have upgraded the hardware with each VMware upgrade since then.

The hardware definitions made by the .vmx file with HW 10 compatibility allow to boot from the live CD. I use the live CD for booting to replace a variable by a constant and have comparable results.

After I upgrade this .vmx file to HW 11 compatibility booting from the live CD hangs. What I see on the console is:

F1 pfSense
F6 PXE
Boot: F1
|

And it hangs exactly there.

A newly setup VM with HW 11 compatibility boots without issues.

Here are the .vmx files:

@.vmx:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "10"
svga.present = "TRUE"
vmci0.present = "TRUE"
floppy0.present = "FALSE"
numvcpus = "4"
memSize = "2048"
sched.cpu.units = "mhz"
sched.cpu.latencySensitivity = "normal"
tools.upgrade.policy = "manual"
scsi0.virtualDev = "lsilogic"
scsi0.present = "TRUE"
ide1:0.deviceType = "cdrom-image"
ide1:0.fileName = "/vmfs/volumes/b83a2cb3-4e9da5f9/CD-DVD-ISOs/Temp/pfSense-LiveCD-2.2.2-RELEASE-i386-20150413-2010.iso"
ide1:0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "mypfsense-test.vmdk"
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
scsi0:0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "vpx"
ethernet0.generatedAddress = "00:50:56:b0:ce:9d"
ethernet0.present = "TRUE"
displayName = "mypfsense-test"
annotation = "pfSense Firewall-Gateway"
guestOS = "freebsd"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
tools.syncTime = "FALSE"
tools.guest.desktop.autolock = "FALSE"
messageBus.tunnelEnabled = "FALSE"
uuid.bios = "42 30 61 ca 40 e8 63 48-09 98 a6 e8 7c 1e 61 b3"
vc.uuid = "50 30 a0 6b 5e cc ca 11-a5 a5 27 88 1d 3b f0 e8"
chipset.onlineStandby = "FALSE"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
hpet0.present = "true"
nvram = "mypfsense-test.nvram"
virtualHW.productCompatibility = "hosted"
snapshot.action = "keep"
scsi0.pciSlotNumber = "16"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "35"
replay.supported = "false"
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
ethernet0.generatedAddressOffset = "0"
hostCPUID.0 = "0000000d756e65476c65746e49656e69"
hostCPUID.1 = "000206d70020080017bee3ffbfebfbff"
hostCPUID.80000001 = "0000000000000000000000012c100800"
guestCPUID.0 = "0000000d756e65476c65746e49656e69"
guestCPUID.1 = "000206d700010800969822030fabfbff"
guestCPUID.80000001 = "00000000000000000000000128100800"
userCPUID.0 = "0000000d756e65476c65746e49656e69"
userCPUID.1 = "000206d700200800169822030fabfbff"
userCPUID.80000001 = "00000000000000000000000128100800"
evcCompatibilityMode = "FALSE"
vmotion.checkpointFBSize = "4194304"
softPowerOff = "FALSE"
tools.remindInstall = "FALSE"
monitor.phys_bits_used = "40"
migrate.hostlog = "mypfsense-test-7ce605e0.hlog"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
sched.swap.derivedName = "/vmfs/volumes/51bd00ab-498d5bf2-2a9e-2c768a5419a8/mypfsense-test/mypfsense-test-7d0003a2.vswp"
uuid.location = "56 4d 5f 61 e8 8a bc 7b-84 67 7a f6 40 15 88 65"
replay.filename = ""
scsi0:0.redo = ""
vmci0.id = "2082365875"
cleanShutdown = "TRUE"

@.vmx:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "11"
svga.present = "TRUE"
vmci0.present = "TRUE"
floppy0.present = "FALSE"
numvcpus = "4"
memSize = "2048"
sched.cpu.units = "mhz"
sched.cpu.latencySensitivity = "normal"
tools.upgrade.policy = "manual"
scsi0.virtualDev = "lsilogic"
scsi0.present = "TRUE"
ide1:0.deviceType = "cdrom-image"
ide1:0.fileName = "/vmfs/volumes/b83a2cb3-4e9da5f9/CD-DVD-ISOs/Temp/pfSense-LiveCD-2.2.2-RELEASE-i386-20150413-2010.iso"
ide1:0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "mypfsense-test.vmdk"
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
scsi0:0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "vpx"
ethernet0.generatedAddress = "00:50:56:b0:ce:9f"
ethernet0.present = "TRUE"
displayName = "mypfsense-test"
annotation = "pfSense Firewall-Gateway"
guestOS = "freebsd"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
tools.syncTime = "FALSE"
tools.guest.desktop.autolock = "FALSE"
messageBus.tunnelEnabled = "FALSE"
uuid.bios = "42 30 61 ca 40 e8 63 48-09 98 a6 e8 7c 1e 61 b3"
vc.uuid = "50 30 a0 6b 5e cc ca 11-a5 a5 27 88 1d 3b f0 e8"
chipset.onlineStandby = "FALSE"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
hpet0.present = "true"
nvram = "mypfsense-test.nvram"
virtualHW.productCompatibility = "hosted"
snapshot.action = "keep"
scsi0.pciSlotNumber = "16"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "35"
replay.supported = "false"
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
ethernet0.generatedAddressOffset = "0"
hostCPUID.0 = "0000000d756e65476c65746e49656e69"
hostCPUID.1 = "000206d70020080017bee3ffbfebfbff"
hostCPUID.80000001 = "0000000000000000000000012c100800"
guestCPUID.0 = "0000000d756e65476c65746e49656e69"
guestCPUID.1 = "000206d700010800969822030fabfbff"
guestCPUID.80000001 = "00000000000000000000000128100800"
userCPUID.0 = "0000000d756e65476c65746e49656e69"
userCPUID.1 = "000206d700200800169822030fabfbff"
userCPUID.80000001 = "00000000000000000000000128100800"
evcCompatibilityMode = "FALSE"
vmotion.checkpointFBSize = "4194304"
softPowerOff = "FALSE"
tools.remindInstall = "FALSE"
monitor.phys_bits_used = "42"
migrate.hostlog = "mypfsense-test-7ce605e0.hlog"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
sched.swap.derivedName = "/vmfs/volumes/51bd00ab-498d5bf2-2a9e-2c768a5419a8/mypfsense-test/mypfsense-test-7d0003a2.vswp"
uuid.location = "56 4d 5f 61 e8 8a bc 7b-84 67 7a f6 40 15 88 65"
replay.filename = ""
scsi0:0.redo = ""
vmci0.id = "2082365875"
cleanShutdown = "FALSE"
vmotion.checkpointSVGAPrimarySize = "4194304"

@.vmx:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "11"
nvram = "mypfsense-new-install.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
memSize = "2048"
sched.cpu.units = "mhz"
sched.cpu.affinity = "all"
powerType.powerOff = "soft"
powerType.suspend = "hard"
powerType.reset = "soft"
scsi0.virtualDev = "lsilogic"
scsi0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "mypfsense-new-install.vmdk"
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
scsi0:0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "VM Network"
ethernet0.addressType = "vpx"
ethernet0.generatedAddress = "00:50:56:b0:56:23"
ethernet0.present = "TRUE"
ide1:0.deviceType = "cdrom-image"
ide1:0.fileName = "/vmfs/volumes/b83a2cb3-4e9da5f9/CD-DVD-ISOs/Temp/pfSense-LiveCD-2.2.2-RELEASE-i386-20150413-2010.iso"
ide1:0.present = "TRUE"
floppy0.startConnected = "FALSE"
floppy0.clientDevice = "TRUE"
floppy0.fileName = "vmware-null-remote-floppy"
displayName = "mypfsense-new-install"
guestOS = "freebsd"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
uuid.bios = "42 30 b0 f6 32 d2 59 cd-ec 7c 54 e8 8f 5a f2 45"
vc.uuid = "50 30 1d a6 eb 2a ef 77-cc d9 f6 14 b2 22 e5 f3"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
virtualHW.productCompatibility = "hosted"
sched.swap.derivedName = "/vmfs/volumes/51bd00ab-498d5bf2-2a9e-2c768a5419a8/mypfsense-new-install/mypfsense-new-install-eb477cb0.vswp"
uuid.location = "56 4d 9a 4b 8b 38 94 1a-ed d4 ec e1 e1 ea ea 29"
replay.supported = "FALSE"
replay.filename = ""
migrate.hostlog = "./mypfsense-new-install-eb477cb0.hlog"
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "16"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "33"
vmci0.id = "-1889865147"
monitor.phys_bits_used = "42"
vmotion.checkpointFBSize = "4194304"
vmotion.checkpointSVGAPrimarySize = "4194304"
cleanShutdown = "FALSE"
softPowerOff = "FALSE"

johnpoz · Apr 30, 2015, 12:03 PM

"booting from live CD"

So your not even booting from the installed VM? Your trying to boot off the install CD?

Notice your using the 32bit CD? Why not installing 64bit?

Why does our pfsense vm only have 1 nic? Do you have the native tools installed on this vm that doesn't work when you upgrade it?

ChristianHaag · Apr 30, 2015, 12:18 PM

@johnpoz:

So your not even booting from the installed VM? Your trying to boot off the install CD?

Exactly. I'm doing it a) for better comparison of the results b) because it is a cloned test VM, which should not interfere with the production VMs IP addresses c) because the behavior on the production was the same and d) because at that early stage of booting it should not make any difference.

@johnpoz:

Notice your using the 32bit CD? Why not installing 64bit?

When i started with pfSense in 2009, 32-bit was the recommended. When I moved to a VM in 2012, there was no way to keep the RRD data with an architecture change. As said, this might be an opportunity to change…

@johnpoz:

Why does our pfsense vm only have 1 nic?

I removed the other 3 network adapters after cloning the VM to another host. But the issue is the same on the production machine with 4 adapters.

@johnpoz:

Do you have the native tools installed on this vm that doesn't work when you upgrade it?

The ones that come with pfSense. They are not really working well with the current version. I found another thread about it. But at that early stage of booting it should not make any difference.

johnpoz · Apr 30, 2015, 2:08 PM

that early in your boot shouldn't know. Do you have serial disabled in the bios of the vm. There have been some threads about serial interface causing some really slow boots.

When get a chance will try to duplicate this by bringing up a vm in hw 10 version, and then installing pfsense 32 bit. And then trying to boot the cd after upgrading to hw 11.