2.7,1 update seems to break qemu-guest-agent auto-start

crichmon

I'm setting a new proxmox server to replace my existing box. Both have pfSense on them. The older box is running 7.4-17 and was running 2.7.0, and the qemu-guest-agent
auto-started as expected. I installed qemu-guest-agent on the newer box (proxmox 8.0.4) into pfSense 2.7.1, and while it runs when started manually, on reboot, it is not running.
I also updated pfSense on the older box, and discovered that the QGA was not auto-starting now (worked in the past).
I've double checked that the requisite configs are in place on both machines, and that the System Tuneables has virtio_console_load YES set, and GuestAgent is enabled in the VM
options in proxmox.
cat /etc/rc.conf
qemu_guest_agent_enable="YES"
qemu_guest_agent_flags="-d -l /var/log/qemu-ga.log"
I've checked the 2.7.1 release notes, and nothing stuck out as a possible cause.
Checking the older box's system.log bot bootup errors, I found this (post-bootup):
Nov 19 16:41:23 pfSense sudo[27369]: crichmon : TTY=pts/0 ; PWD=/etc ; USER=root ; COMMAND=/usr/bin/vi /etc/rc.conf
Nov 19 16:41:41 pfSense crichmon[47142]: /usr/local/etc/rc.d/qemu-guest-agent: WARNING: failed to start qemu_guest_agent
Nov 19 16:41:45 pfSense sudo[47183]: crichmon : TTY=pts/0 ; PWD=/etc ; USER=root ; COMMAND=/usr/sbin/service qemu-guest-agent start
Doesn't say why the start-up failed, but service qemu-guest-agent start succeeds without issues.
I checked for a similar issue under FreeBSD and didn't find anything, and also checked proxmox's forum. Some related issues, but nothing recent and
specific to pfSense 2.7.1. Did I miss finding the answer, or is this an actual bug?
Thx, Chris

Viper_Rus

I can't confirm. Proxmox 8.0.9 + pfsense CE 2.7.1 - qemu agent starts normally.

crichmon

@Viper_Rus
Sorry... no joy. Updated proxmox to 8.0.9 and still had to restart qemu-ga by hand after reboot.
2.7.1-RELEASE][crichmon@pfSense2.csrhome.home]/var/log: service qemu-guest-agent status
qemu_guest_agent is not running.
[2.7.1-RELEASE][crichmon@pfSense2.csrhome.home]/var/log: sudo service qemu-guest-agent start
Starting qemu_guest_agent.
[2.7.1-RELEASE][crichmon@pfSense2.csrhome.home]/var/log: ps uaxw | grep 45833
root 45833 0.0 0.1 17740 4696 - Is 15:41 0:00.00 /usr/local/bin/qemu-ga -d -l /var/log/qemu-ga.log
crichmon 69769 0.0 0.0 436 264 0 R+ 15:43 0:00.00 grep 45833
[2.7.1-RELEASE][crichmon@pfSense2.csrhome.home]/var/log: sudo service qemu-guest-agent status
qemu_guest_agent is running as pid 45833.

I tried twice including sudo prefixing the status command, and still not running by default on reboot, but
starts fine by hand. Nothing interesting in the boot log.

FWIW, this is a test machine (minimal time investment) for now so I'm open for more risky things to try.

Thx, Chris

Patch

@crichmon said in 2.7,1 update seems to break qemu-guest-agent auto-start:

I'm setting a new proxmox server to replace my existing box

To get the agent to start automatically I have had to do all of the steps listed in
https://forum.netgate.com/post/1116089 and https://forum.netgate.com/post/1098565

Btw it works OK for me using Proxmox v8.0.9 (and many earlier versions with pfsense 2.6, 2.7, 2.7.1

Viper_Rus

@crichmon
@Patch

I also followed these instructions. More specifically, there is an automated script inside that topic.

crichmon

@Viper_Rus
So... I did find on the new box that /usr/local/etc/rc.d/qemu-agent.sh was missing. Once copied from older box,
and actually shutting down the VM, solved the problem on the newer box. So, to follow one of the suggestions,
I renamed /etc/rc.conf to rc.conf.local and got a boot error from devd about a missing rc.conf, but low and behold,
the agent was still started. To solve that, I created another rc.conf that contains just the #!/bin/sh line, and set
permissions, and everything still works as expected.

About the older machine, I'll have to work on that later, since that's our firewall and I'll need a quiet time to fiddle
with. It's more odd in that it was working on pfSense 2.7.0, and broke with update to 2.7.1. I'll update once I
find out more.
thx, Chris

crichmon

@crichmon
So, on the older box, I made two changes at the same time, so don't know which fixed it, but moved the agent config to
rc.conf.local and added the #!/bin/sh to the otherwise empty rc.conf, powered the VM down, and with startup, no complaints
from devd and the agent auto-started. Cased closed, at least for me.
Thx, Chris

bmeeks

@crichmon said in 2.7,1 update seems to break qemu-guest-agent auto-start:

but moved the agent config to
rc.conf.local

Note that on pfSense, at least on previous versions, it was required to put local customizations (such as starting some service or agent at boot) in rc.conf.local, because rc.conf is overwritten by pfSense during every update. Any local entries a user had placed in rc.conf would be lost. But the rc.conf.local file is specifically designed for user customizations and is not touched on upgrades.

crichmon

@bmeeks Hi, yeah, I'm aware (more or less) and is why I moved the config to the .local file.
Both boxes are back in working order.

The semi-confusing part is that there are a multitude of instructions to be found, and I think the actual install
of the qemu-guest-agent package is what says to edit /etc/rc.conf.
Anyhow, I appreciate all the help. Hopefully this thread trends better than the older ones.
Chris

bmeeks

@crichmon said in 2.7,1 update seems to break qemu-guest-agent auto-start:

The semi-confusing part is that there are a multitude of instructions to be found, and I think the actual install
of the qemu-guest-agent package is what says to edit /etc/rc.conf.

That is true for an install on a general FreeBSD or Linux distro. The instructions you see that scroll down as you install the package are simply the default instructions that come with the package from FreeBSD ports. Unfortunately, it is not customized for pfSense.

pfSense is unique. It stores all configuration information that is associated with the most typically edited/customized conf text files in the config.xml file managed by the pfSense code. The configuration parameters in that file are read out during various pfSense operations and physicallly written to the appropriate conf files in /etc/ or /usr/local/etc/ each time a user clicks Save in the GUI. And during those write operations the entire file is recreated with the new content. Hence anything a user had put in there via a shell prompt or other direct editing access is overwritten.

The above is also true for the many packages used on pfSense. The Snort and Suricata packages I maintain work that way. On any other Unix-type system the admin would directly edit the suricata.yaml file to configure Suricata or the snort.conf file to configure Snort. But on pfSense, all of the configuration information lives in config.xml and those filesystem conf files are recreated by the PHP GUI code with each Save operation.

crichmon

@bmeeks Thanks a bunch for the clarifications and details. I'm newish to pfSense and FreeBSD, but not new to UNIX/Linux in general.
This sort of thing (/etc files updating) isn't "usual" for me, but not surprising given how pfSense is set up. I tend to shun GUI's, but
pfSense and proxmox seem really solid, and I like that you can still get to the actual configs in both.

Thx, Chris

bmeeks

@crichmon said in 2.7,1 update seems to break qemu-guest-agent auto-start:

@bmeeks Thanks a bunch for the clarifications and details. I'm newish to pfSense and FreeBSD, but not new to UNIX/Linux in general.
This sort of thing (/etc files updating) isn't "usual" for me, but not surprising given how pfSense is set up. I tend to shun GUI's, but
pfSense and proxmox seem really solid, and I like that you can still get to the actual configs in both.

Thx, Chris

The way pfSense works with regards to recreating/overwriting configuration files on the disk is what allows the single-file config.xml backup process to work. Every configuration parameter needed for pfSense lives in that single config.xml file, so simply backing up and then restoring that single file (along with a reboot after the restore) is all you need to resurrect your firewall to a previous state. You don't have to backup the entire disk file-by-file. The GUI code will read all the configuration parameters out of the config.xml file and recreate/overwrite the necessary conf files on the disk.

crichmon

@bmeeks
Sure, and I 'get it' and not complaining. It helps to have a clear understanding of how things work.
Not trying to rock the boat. As mentioned, I'm doing my best to get with newer (to me) things.

Thx, Chris