pfSense 2.6 will not reboot in Proxmox :(

bmeeks

@mrpete said in pfSense 2.6 will not reboot :(:

@Gertjan
Latest attempt, I notice a console error when attempting with shutdown -r now

I wonder if significant?

/etc/pfSense-rc.shutdown: /etc/backup_voucher.sh: not found

That indicates to me that either you manually, or some package automatically, edited the /etc/pfSense-rc.shutdown shell script and added a call to that missing script. I just checked my SG-5100 box, and there is no such /etc/backup_voucher.sh file present, and my pfSense-rc.shutdown script does not call it either.. Did you by chance do something with captive portal at one time? "Voucher" sounds like it is associated with captive portal.

One thing that script does is scan /usr/local/etc/rc.d/ for *.sh script files to be executed. Perhaps one of those *.sh scripts it finds references the non-existent backup_voucher.sh file.

Here is the section of code from pfSense-rc.shutdown that does that:

# Invoke shutdown scripts if present
scripts=/usr/local/etc/rc.d/shutdown.*.sh
for script in $scripts
do
	if [ -f "$script" -a -x "$script" ]
	then
		$script
	fi
done

stephenw10

Do you have RAM disks enabled?

MrPete

@stephenw10
This is a fresh install of 2.6 downloaded over the weekend onto my two CARP systems. No code edits at all.

It does include a restore of saved configuration (config.xml)
One system auto-installed packages. The other couldn't reach the Internet initially, so I had to manually install.

My package list is:
acme
cron
haproxy-devel
iperf
mtr-nox11
notes
nut
pfblockerng-devel
pimd
shellcmd
sudo

(Plus: manually installed qemu-guest-agent)

My shellcmd list (all earlyshellcmd):

/usr/local/pkg/pfblockerng/pfblockerng.sh aliastables
rm /var/log/nginx/error.log
service qemu-guest-agent start

MrPete

@stephenw10
RAM disks... Ahh: System->Advanced->Misc

Yep, same as before.
40mb /tmp, 200mb /var, backups as default.

Duuuhhh. Of course that means final shutdown log is lost. Will convert to normal for testing...

stephenw10

When you enable ram disks the logs are stored in ram so to avoid losing them they are written out to the drive at shutdown and then rebuilt again at boot. Anything that happens in between those things will be lost. You would still see messages in the Proxmox GUI.
But more importantly it looks like it's trying to backup the captive portal vouchers file and the script is not present. I would expect it to just do nothing at that point and continue to the next script though.
The output you're seeing though looks like it's reached the end of the shutdown scripts, it's unmounted the drives by that point. The next thing it should normally show is Rebooting...

Do you have the Captive Portal enabled?

Are you using ZFS?

Steve

MrPete

@stephenw10

ZFS Yes
Captive Portal: Nope, nothing there. Didn't even know what it was ;)

HOWEVER: searching my config.xml file, I found that CARP Sync for Captive Portal is enabled. (As recommended, I enabled everything.... although now the DHCP forwarder checkboxes are not checked... perhaps because I don't use them?)

stephenw10

Hmm, I did a lot of testing in Proxmox (and I know others did) and never saw this.

Not seeing it now on a 2.6 VM with ramdisks enabled.

I wonder if you have something odd in the VM config...
If you can try cloning the VM and defaulting the config. If that still fails to reboot I would think it has to be something in the VM setup. Not sure what though...

If it's still doing anything when it stalls like that it should report it if you enter ctl+t. It wil tell you what process it's waiting on.

Steve

stephenw10

For reference:
Screenshot from 2022-03-09 23-14-13.png

MrPete

@stephenw10

We have two significant configuration differences.

I'm using the modern UEFI bios
I'm using the q35 rather than i440 CPU. And passing [host] through (on "Processors" line)

Otherwise, nothing exciting.

Either of these could be "it" I suppose.

UEFI is of course quite different.

q35 supports:

PCIe (not just 1997 PCI)
ICH9 chipset
PCIe passthrough -- which I am using to achieve full gigabit performance in a VM.

stephenw10

What CPU are you passing to it?

MrPete

@stephenw10
One is i7-3770 (giving 4 cores; has AES-NI, Active according to pfSense)
The other is i7-4770 (similar but faster)

stephenw10

Hmm, nope still reboots fine here with:
Screenshot from 2022-03-10 01-36-19.png

Maybe you have a non-default option?
Screenshot from 2022-03-10 01-36-40.png

qemu guest agent maybe? Seems most likely...

MrPete

@stephenw10 I definitely use qemu-guest-agent.

Just ran a set of tests:
..
BIOS: reboots fine, shuts down fine. (Screen is 25 lines ;) ). Either i440 or q35

UEFI: does not reboot, either i440 or q35. With i440 I do see "Rebooting..." but it doesn't.

Will check options... I know for sure I am setting UUID -- started out with identical UUID to primary CARP, which caused other trouble, so I set a different one.

I had OS type Linux (auto set for recent). Reset to "Other"... did not help.
Yes I had q-g-a...

stephenw10

Can you disable/remove the qemu agent as a test?

MrPete

@stephenw10

BUMMER. OK, duplicating close to yours, but with the one change I know breaks mine:

i440FX
UEFI (this breaks it for me)
pkg remove qemu-guest-agent**

Still no reboot.

**Note: don't know how it does it, but even though rclocal has enable q-g-e YES,... by disabling in Proxmox, when I looked in pf, it was NOT running. Not trusting that... I completely uninstalled it.

So:

On 2.5.2 I have had zero issues with this aspect of my config. I was running ZFS.in 2.5.2
I reinstalled 2.6 into the exact same VM.
On 2.6, on my host, if I use UEFI then pfSense will not reboot

I will test tomorrow to see if VM running BIOS is a viable workaround for my situation for now.

If you have further testing ideas, I'm all ears. Will do it early AM tomorrow. Now, gotta run. THANK YOU!!!!

stephenw10

What version of Proxmox are you running?

MrPete

@stephenw10

root@pve1:~# pveversion
pve-manager/7.1-10/6ddebafe (running kernel: 5.15.12-1-pve)

stephenw10

Hmm, OK I'm still running 6.4 but I know others here are running 7 and not hitting this.

Well I would be trying to replicate this on a clean install with a basic setup on your host. It's either something in the VM setup of the pfSense config and that would rule one of those out.

Steve

MrPete

@stephenw10 I'm on a close-to-clean install, as this was full reinstall for 2.6. Of course I reloaded my config.xml

I can do that in chunks, perhaps...

Plain install, no config at all. See if even that fails
Install, load config, no packages
Then start making smaller changes if we haven't nailed it.

MrPete

@stephenw10
Several lessons learned so far. This is from installing the 2.6 iso, using as "default" as I could, other than setting UEFI:

See screen grabs below for working almost-default settings.

ISO permissions must be set to 654 (Group executable) or it can't boot.
While your VM uses type "Other", that produced a bad VM for me (EFI partition too small, and would not boot for install.) Setting up VM at least initially as "Linux" (which it picks based on the ISO) produces better settings.
WIth UEFI, un-check "Pre-Enroll Keys" to avoid the Windows-style IOS security check, which will fail.

That's enough for install to work. :)

The good news: This setup does reboot.
The bad news: modifying my plain test install+reconfig setup to exactly match, does NOT reboot.

So on to phase two testing but no time today (during normal hours at least.)
pf2.6 proxmox uefi ok-1.jpg
pf2.6 proxmox uefi ok-2.jpg