Some packages fail to start after issuing "reboot VM" command on Proxmox
-
@s1nnerman89 said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
To reproduce the bug, simply issue a shutdown/reboot from Proxmox WebUI and those packages should fail to correctly start after pfSense has boot up.
The question is why would anyone shutdown pfSense from Proxmox webgui? If you have to use a browser to access Proxmox's webgui, why not have another tab with pfSense webgui?
Sadly, the issue you're facing is NOT A BUG but, a failure of the network administrator to follow proper protocol in executing a firewall power-off.
-
@NollipfSense Shutdown (rather than Stop) via the Proxmox UI sends an ACPI signal to the guest. If the guest is not configred to respond to the ACPI shutdown signal, then nothing happens. If it is configured so, then one would expect it to shut down cleanly and subsequently boot to a similar state following shutdowns via its own UI or commandline. But perhaps this is not happening here?
-
@darcey said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
Proxmox UI sends an ACPI signal to the guest. I
Are you using Qemu-agent? I have never shutdown pfSense from Proxmox UI and would never consider such action, and I have Qemu-agent installed.
-
@NollipfSense said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
Are you using Qemu-agent? I have never shutdown pfSense from Proxmox UI and would never consider such action, and I have Qemu-agent installed.
No. But then that's not required to shutdown an ACPI aware guest. I would normally reboot pfSense via its own UI. However I also included pfSense guests in PVE backups (shutdown type) and not experienced any problems. This also uses ACPI shutdown (soft-off).
-
@darcey said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
No. But then that's not required to shutdown an ACPI aware guest.
I would recommend installing Qemu-agent despite it been not required...wouldn't hurt. Because my Proxmox box heavily depends on pfSense VM, I would never shut it down without using the pfSense webGUI.
-
The question is why would anyone shutdown pfSense from Proxmox webgui? If you have to use a browser to access Proxmox's webgui, why not have another tab with pfSense webgui?
Of course, shutting down pfSense from its own webui is the most logical option. My problem arises from Proxmox's scheduled backup jobs, which i set to trigger on a bi-weekly basis. Basically, when the job executes, the VM is powered down, a backup of the VM is performed and then the VM is started again. Upon startup, some of my installed packages (avahi and radiusd are the usual culprits) fail to start up. As i said before, this behaviour does not present if shutting down pfSense using its webui. Obviously, knowing there is this bug, i could exclude pfSense's VM from the automated backup jobs and do it manually, but it would have been nice if the process could be automated.
Sadly, the issue you're facing is NOT A BUG but, a failure of the network administrator to follow proper protocol in executing a firewall power-off.
I think that in my initial post i did not specify clearly enough the reason why i was shutting down pfSense using Proxmox's WebUI. This problem arises only due to me wanting to automate pfSense's snapshots/backups. If not for this, i would never trigger its shutdown/reboot using Proxmox's UI, seeing as an "official" package for qemu-guest-agent doesn't even exist but the current implementation was kind of "retrofitted" from the FreeBSD package.
However I also included pfSense guests in PVE backups (shutdown type) and not experienced any problems. This also uses ACPI shutdown (soft-off).
This is interesting, may i ask you which PVE version are you running? I'm currently running version 8.0.3, and i can't remember (or actually didn't notice) if this behaviour happened before the 7.4 -> 8 upgrade.
-
@s1nnerman89 said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
This is interesting, may i ask you which PVE version are you running? I'm currently running version 8.0.3, and i can't remember (or actually didn't notice) if this behaviour happened before the 7.4 -> 8 upgrade.
I'm also running PVE8. I'm currently only doing snapshot backups and probably have not run a shutdown backup since upgrading from PVE 7.4. However I do not run either of the packages you mention. Perhaps it is an issue that has surfaced after the PVE upgrade?
Does the guest actually cleanly shutdown via ACPI, or is the console hung or unclean umount warnings on restart? -
@NollipfSense said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
The question is why would anyone shutdown pfSense from Proxmox webgui?
Because that is how Proxmox node is designed to work. Useful when working on a node or setting up a UPS.
@NollipfSense said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
I have never shutdown pfSense from Proxmox UI
Try it.
When watching the process on the Proxmox console for pfsense it proceeds cleanly.@s1nnerman89 said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
some packages (namely avahi and radiusd) were randomly not starting
I don't run those so can't confirm that behaviour
-
@Patch said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
Useful when working on a node or setting up a UPS.
I pass-through the UPS to pfSense Nut package...
@Patch said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
Try it.
When watching the process on the Proxmox console for pfsense it proceeds cleanly.Okay.
-
Does the guest actually cleanly shutdown via ACPI, or is the console hung or unclean unmount warnings on restart?
Shutdowns are always clean, regardless of being issued with or without qemu-guest-agent package present, the only messages related to the issue are
Jul 24 11:43:27 check_reload_status 425 Starting packages [..] Jul 24 11:43:28 php-fpm 84295 /rc.start_packages: Restarting/Starting all packages. [..] Jul 24 11:43:35 php-fpm 395 /rc.start_packages: Skipping STARTing packages process because previous/another instance is already runningJust tried to replicate the bug in order to post the logfiles and avahi and radiusd have indeed failed to start after the node was powered down by ACPI.
Also, this message appeared after powering down the node thru Proxmox WebUI; if i manually start the failing packages, then reboot/shutdown the node through pfSense's WebUI, on next boot i can see that the OS managed to start each installed packages because log is full of messages like:Starting avahi [..]... Success Starting radiusd [..]... Success -
@s1nnerman89 It's a strange one. I just tried rebooting this pfsense guest via the proxmox UI. It shuts down and boots cleanly with all expected services active. The packages I have installed that provide services are: arpwatch, softflowd, pfblocker. I have acme and System patches too but these don't run daemons AFAIK. Sorry I can't be of more help.
-
@darcey said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
It's a strange one.
It maybe related to the current Proxmox appears to default to only allow 15 not 180 sec for a VM to shut down https://forum.proxmox.com/threads/proxmox-ve-8-0-released.129320/page-15#post-574813
-
@Patch If that was the cause in this case, I would have thought you would see evidence of an unclean guest shutdown. i.e. guest console not showing the final disk sync or fsck inconsistency reported at next boot.
Re the new defualt timeout, does that now mean PVE pulls the plug on the guest if it has not shutdown within 15sec? That seems a bit severe! -
@Patch said in Some packages fail to start after issuing "reboot VM" command on Proxmox:
Try it.
When watching the process on the Proxmox console for pfsense it proceeds cleanly.I actually did, a boot up though using Proxmox...just can't see myself doing a shutdown.
