pfSense 2.6 will not reboot in Proxmox :(

MrPete

How can I help debug this? All I see is console messages as below. All network traffic stops and disk activity as well.
Then... nothing. The system simply does not reboot.

Should there be a log somewhere of "final moments" before I had hit the switch so to speak?
pfsense 2.6 normal reboot fails.jpg

Gertjan

@mrpete

Tell us more about the device.
A Netgate appliance ?
A bare bone ?
A VM ?

The first can't shut down itself, you have to pull the DC power wire.
A "bare bone" should work, if FreeBSD recognises the 'ACPI' hardware (or whatever is used to signal the build in power supply). Could be a BIOS setting issue, or driver issue.
A VM : works for me. I only tested Hyper-V.

MrPete

@gertjan Are you saying a Netgate Appliance can't reboot?

I'm not trying to shut down. Trying to reboot.

In this case, they are VMs (in Proxmox.)

They easily rebooted under 2.5.2 but no longer.

No matter how I attempt, always the same result.

From pfSense GUI
From pfSense BSD shell (ssh)
From Proxmox GUI via QEMU agent

I see the console messages... but have not yet found a log file containing the log for a reboot.

I assume there is a way to turn on debug-level logging for this.

MrPete

@Gertjan
Latest attempt, I notice a console error when attempting with shutdown -r now

I wonder if significant?

/etc/pfSense-rc.shutdown: /etc/backup_voucher.sh: not found

FSC830

@MrPete A Netgate appliance can reboot and a shutdown, but it can no do a power-off (at least my SG-3100)!
If system is halted via Diagnostics menu the system itself remains powered on, but all services are halted and down.

Regards

bmeeks

@mrpete said in pfSense 2.6 will not reboot :(:

@Gertjan
Latest attempt, I notice a console error when attempting with shutdown -r now

I wonder if significant?

/etc/pfSense-rc.shutdown: /etc/backup_voucher.sh: not found

That indicates to me that either you manually, or some package automatically, edited the /etc/pfSense-rc.shutdown shell script and added a call to that missing script. I just checked my SG-5100 box, and there is no such /etc/backup_voucher.sh file present, and my pfSense-rc.shutdown script does not call it either.. Did you by chance do something with captive portal at one time? "Voucher" sounds like it is associated with captive portal.

One thing that script does is scan /usr/local/etc/rc.d/ for *.sh script files to be executed. Perhaps one of those *.sh scripts it finds references the non-existent backup_voucher.sh file.

Here is the section of code from pfSense-rc.shutdown that does that:

# Invoke shutdown scripts if present
scripts=/usr/local/etc/rc.d/shutdown.*.sh
for script in $scripts
do
	if [ -f "$script" -a -x "$script" ]
	then
		$script
	fi
done

stephenw10

Do you have RAM disks enabled?

MrPete

@stephenw10
This is a fresh install of 2.6 downloaded over the weekend onto my two CARP systems. No code edits at all.

It does include a restore of saved configuration (config.xml)
One system auto-installed packages. The other couldn't reach the Internet initially, so I had to manually install.

My package list is:
acme
cron
haproxy-devel
iperf
mtr-nox11
notes
nut
pfblockerng-devel
pimd
shellcmd
sudo

(Plus: manually installed qemu-guest-agent)

My shellcmd list (all earlyshellcmd):

/usr/local/pkg/pfblockerng/pfblockerng.sh aliastables
rm /var/log/nginx/error.log
service qemu-guest-agent start

MrPete

@stephenw10
RAM disks... Ahh: System->Advanced->Misc

Yep, same as before.
40mb /tmp, 200mb /var, backups as default.

Duuuhhh. Of course that means final shutdown log is lost. Will convert to normal for testing...

stephenw10

When you enable ram disks the logs are stored in ram so to avoid losing them they are written out to the drive at shutdown and then rebuilt again at boot. Anything that happens in between those things will be lost. You would still see messages in the Proxmox GUI.
But more importantly it looks like it's trying to backup the captive portal vouchers file and the script is not present. I would expect it to just do nothing at that point and continue to the next script though.
The output you're seeing though looks like it's reached the end of the shutdown scripts, it's unmounted the drives by that point. The next thing it should normally show is Rebooting...

Do you have the Captive Portal enabled?

Are you using ZFS?

Steve

MrPete

@stephenw10

ZFS Yes
Captive Portal: Nope, nothing there. Didn't even know what it was ;)

HOWEVER: searching my config.xml file, I found that CARP Sync for Captive Portal is enabled. (As recommended, I enabled everything.... although now the DHCP forwarder checkboxes are not checked... perhaps because I don't use them?)

stephenw10

Hmm, I did a lot of testing in Proxmox (and I know others did) and never saw this.

Not seeing it now on a 2.6 VM with ramdisks enabled.

I wonder if you have something odd in the VM config...
If you can try cloning the VM and defaulting the config. If that still fails to reboot I would think it has to be something in the VM setup. Not sure what though...

If it's still doing anything when it stalls like that it should report it if you enter ctl+t. It wil tell you what process it's waiting on.

Steve

stephenw10

For reference:
Screenshot from 2022-03-09 23-14-13.png

MrPete

@stephenw10

We have two significant configuration differences.

I'm using the modern UEFI bios
I'm using the q35 rather than i440 CPU. And passing [host] through (on "Processors" line)

Otherwise, nothing exciting.

Either of these could be "it" I suppose.

UEFI is of course quite different.

q35 supports:

PCIe (not just 1997 PCI)
ICH9 chipset
PCIe passthrough -- which I am using to achieve full gigabit performance in a VM.

stephenw10

What CPU are you passing to it?

MrPete

@stephenw10
One is i7-3770 (giving 4 cores; has AES-NI, Active according to pfSense)
The other is i7-4770 (similar but faster)

stephenw10

Hmm, nope still reboots fine here with:
Screenshot from 2022-03-10 01-36-19.png

Maybe you have a non-default option?
Screenshot from 2022-03-10 01-36-40.png

qemu guest agent maybe? Seems most likely...

MrPete

@stephenw10 I definitely use qemu-guest-agent.

Just ran a set of tests:
..
BIOS: reboots fine, shuts down fine. (Screen is 25 lines ;) ). Either i440 or q35

UEFI: does not reboot, either i440 or q35. With i440 I do see "Rebooting..." but it doesn't.

Will check options... I know for sure I am setting UUID -- started out with identical UUID to primary CARP, which caused other trouble, so I set a different one.

I had OS type Linux (auto set for recent). Reset to "Other"... did not help.
Yes I had q-g-a...

stephenw10

Can you disable/remove the qemu agent as a test?

MrPete

@stephenw10

BUMMER. OK, duplicating close to yours, but with the one change I know breaks mine:

i440FX
UEFI (this breaks it for me)
pkg remove qemu-guest-agent**

Still no reboot.

**Note: don't know how it does it, but even though rclocal has enable q-g-e YES,... by disabling in Proxmox, when I looked in pf, it was NOT running. Not trusting that... I completely uninstalled it.

So:

On 2.5.2 I have had zero issues with this aspect of my config. I was running ZFS.in 2.5.2
I reinstalled 2.6 into the exact same VM.
On 2.6, on my host, if I use UEFI then pfSense will not reboot

I will test tomorrow to see if VM running BIOS is a viable workaround for my situation for now.

If you have further testing ideas, I'm all ears. Will do it early AM tomorrow. Now, gotta run. THANK YOU!!!!