pfSense 2.6 problem with zombie processes

gofaizen

@stephenw10 said in pfSense 2.6 problem with zombie processes:

What disks do you have configured there?

Hello....

mount

/dev/ufsid/60618bb1ebc69388 on / (ufs, local, noatime, journaled soft-updates)
devfs on /dev (devfs, local)
tmpfs on /var/run (tmpfs, local)
devfs on /var/dhcpd/dev (devfs, local)

df -hT
Filesystem Type Size Used Avail Capacity Mounted on
/dev/ufsid/60618bb1ebc69388 ufs 5.5G 2.0G 3.1G 39% /
devfs devfs 1.0K 1.0K 0B 100% /dev
tmpfs tmpfs 4.0M 156K 3.8M 4% /var/run
devfs devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev

I have discovered the source of this problem - qemu-guest-agent.
I have ovirt 4.4.10 and I have installed pkg - qemu-guest-agent/
Config of qemu-ga :
cat rc.conf.local
qemu_guest_agent_enable="YES"
qemu_guest_agent_flags="-d -v -m virtio-serial -l /var/log/qemu-ga.log -p /dev/ttyV0.2"

ls -l /dev/vtcon
total 0
lrwxr-xr-x 1 root wheel 10 Apr 21 12:11 com.redhat.spice.0 -> ../ttyV0.3
lrwxr-xr-x 1 root wheel 10 Apr 21 12:11 org.qemu.guest_agent.0 -> ../ttyV0.2
lrwxr-xr-x 1 root wheel 10 Apr 21 12:11 ovirt-guest-agent.0 -> ../ttyV0.1

When I start qemu-guest-agent, it try to get information about filesystems and launch process 'df -hT', which becomes zombie.

I have used this instructions - https://forum.netgate.com/topic/162083/pfsense-vm-on-proxmox-qemu-agent-installation

netblues

@gofaizen No zombie processes here

Running under redhat 8.5 kvm

ls -l /dev/vtcon/
total 0
lrwxr-xr-x  1 root  wheel  10 Mar 19 07:01 com.redhat.spice.0 -> ../ttyV0.1
lrwxr-xr-x  1 root  wheel  10 Mar 19 07:01 org.qemu.guest_agent.0 -> ../ttyV0.2

df -hT

The ovrt-guest-agent.0 -> is missing though.

gofaizen

@netblues Please show config of qemu-guest-login - /etc/rc.conf.local

netblues

@gofaizen

root: cat /etc/rc.conf.local
qemu_guest_agent_enable="YES"
qemu_guest_agent_flags="-d -v -l /var/log/qemu-ga.log"
virtio_console_load="YES"

netblues

@gofaizen said in pfSense 2.6 problem with zombie processes:

qemu_guest_agent_flags="-d -v -m virtio-serial -l /var/log/qemu-ga.log -p /dev/ttyV0.2"

Where did you get this?

gofaizen

@netblues I have the same

gofaizen

@netblues This is command line from linux.
I have tried qemu_guest_agent_flags="-d -v -l /var/log/qemu-ga.log" and qemu_guest_agent_flags="-d -v -m virtio-serial -l /var/log/qemu-ga.log -p /dev/ttyV0.2"
Same effect - zombie process every minute

netblues

@gofaizen said in pfSense 2.6 problem with zombie processes:

@netblues This is command line from linux.
I have tried qemu_guest_agent_flags="-d -v -l /var/log/qemu-ga.log" and qemu_guest_agent_flags="-d -v -m virtio-serial -l /var/log/qemu-ga.log -p /dev/ttyV0.2"
Same effect - zombie process every minute

There is no -p /dev/ttyV0.2 in my config.

stephenw10

Hmm, it's unclear which of those things is a symptom.
Is the widget throwing that error because qemuguest is continually trying to access the disk?
Or is the agent unable to read the disk status because the widget is doping somthing wrong?
Or maybe both are failing because of some other issue.

If you disable the qemu agent do you still see the php error from the widget?

Steve

gofaizen

@stephenw10
Widget throws error in random order.
Error in widget and qemu-guest-agent not in any connect. When I disable qemu-guest-agent widget continues throwing errors in random order. It can work whole day in normal mode, or can throw error every 10 minutes. And it throws error every time after reboot.
qemu-guest-agent make zombie every time after getting data: {"execute":"guest-get-fsinfo"} message from oVirt.

stephenw10

Hmm, well that sounds more like both things are failing because of something else preventing access to the filesystem.

If you run /bin/df -hT manually at the command line repeatedly does it ever fail?

Can we see the output of: /bin/df -hT --libxo=json
Though that too might need to be run until it fails. That's what the widget is trying to do and choking on the output.

Steve

gofaizen

@stephenw10 Yes, I can run df -hT manually. I have checked this in first order.
": [{"name":"/dev/ufsid/60618bb1ebc69388","type":"ufs ","blocks":"5.5G","used":"2.0G","available":"3.1G","used-percent":39,"mounted-on":"/"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/dev"}, {"name":"tmpfs","type":"tmpfs","blocks":"4.0M","used":"152K","available":"3.9M","used-percent":4,"mounted-on":"/var/run"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/var/dhcpd/dev"}]}

stephenw10

But if you run /bin/df -hT manually at the command line repeatedly does it ever fail? Like after 100 tries?
It will likely only fails as often as the widget does. Or if you happen to run it when the qemu agent is trying to access the filesystem also.

The JSON output there seems to be missing the initial terms. I expect it to read like:

[22.05-DEVELOPMENT][admin@plusdev-2.stevew.lan]/root: /bin/df -hT --libxo=json
{"storage-system-information": {"filesystem": [{"name":"/dev/ufsid/626069f74a9f0e6e","type":"ufs  ","blocks":"9.2G","used":"1.5G","available":"7.0G","used-percent":18,"mounted-on":"/"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/dev"}, {"name":"tmpfs","type":"tmpfs","blocks":"4.0M","used":"112K","available":"3.9M","used-percent":3,"mounted-on":"/var/run"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/var/dhcpd/dev"}]}
}

Steve

gofaizen

@stephenw10
{"storage-system-information": {"filesystem": [{"name":"/dev/ufsid/60618bb1ebc69388","type":"ufs ","blocks":"5.5G","used":"2.0G","available":"3.1G","used-percent":39,"mounted-on":"/"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/dev"}, {"name":"tmpfs","type":"tmpfs","blocks":"4.0M","used":"152K","available":"3.9M","used-percent":4,"mounted-on":"/var/run"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/var/dhcpd/dev"}]}
}
Possibly I have copied part of output.

Tried to run /bin/df -hT --libxo=json in loop 1000 times. All times were successful.

#!/bin/sh

for i in seq 1 1000
do
/bin/df -hT --libxo=json
done

stephenw10

Hmm. If you remove the qemu-agent can I assume this goes away?

gofaizen

@stephenw10
When I stopped qemu-ga service zombie processes died.
I have checked it several times. But I need qemu-ga in my oVirt installation.

stephenw10

Hmm, well can you configure it not to query the disk status? I would assume that also solves it.

Can you test qemu-agent in a FreeBSD 12.3 install? Hard to see what but it might be something in base that's changed.

Steve

gofaizen

@stephenw10
I am trying to disable query for file system status in qemu-ga.

gofaizen

@stephenw10
With this parameters -d -v -l /var/log/qemu-ga.log -b "guest-get-fsinfo" qemu-ga does not generate zombies.
-b "guest-get-fsinfo" - means - blacklist guest-get-fsinfo command

stephenw10

Mmm, OK. And does that still provide the info you need?