pfSense 2.6 problem with zombie processes

[gofaizen]

Hello, Dear Sirs !
I have pfSense 2.6 on virtual KVM machine with qemu-guest-agent.
I have upgraded 2.5 to 2.6. Now I have a large problem with zombie process every minute - df -hT :

root 22734 0.0 0.0 0 0 - Z 12:12 0:00.00 <defunct>
root 44031 0.0 0.0 0 0 - Z 12:12 0:00.00 <defunct>
root 70180 0.0 0.0 0 0 - Z 12:12 0:00.00 <defunct>
root 98025 0.0 0.0 0 0 - Z 12:11 0:00.00 <defunct>

procstat -c 98025
98025 df /bin/df -hT

In system log I have this message -
Apr 21 10:50:24 php-fpm 99129 thrown
Apr 21 10:50:24 php-fpm 99129 #4 {main}
Apr 21 10:50:24 php-fpm 99129 #3 /usr/local/www/index.php(428): include('/usr/local/www/...')
Apr 21 10:50:24 php-fpm 99129 #2 /usr/local/www/widgets/widgets/disks.widget.php(65): disks_compose_widget_table(Array)
Apr 21 10:50:24 php-fpm 99129 #1 /usr/local/www/widgets/include/disks.inc(102): disks_compose_widget_tree_body()
Apr 21 10:50:24 php-fpm 99129 #0 /usr/local/www/widgets/include/disks.inc(251): disk_compose_root_filesystem_row()
Apr 21 10:50:24 php-fpm 99129 Stack trace:
Apr 21 10:50:24 php-fpm 99129 /index.php: New alert found: PHP ERROR: Type: 1, File: /usr/local/www/widgets/include/disks.inc, Line: 182, Message: Uncaught Error: Call to a member function getPath() on bool in /usr/local/www/widgets/include/disks.inc:182

Please assist me in solving this problem.

Regards,
Ilia Molchadsky

[stephenw10]

What disks do you have configured there?

[gofaizen]

@stephenw10 said in pfSense 2.6 problem with zombie processes:

What disks do you have configured there?

Hello....

mount

/dev/ufsid/60618bb1ebc69388 on / (ufs, local, noatime, journaled soft-updates)
devfs on /dev (devfs, local)
tmpfs on /var/run (tmpfs, local)
devfs on /var/dhcpd/dev (devfs, local)

df -hT
Filesystem Type Size Used Avail Capacity Mounted on
/dev/ufsid/60618bb1ebc69388 ufs 5.5G 2.0G 3.1G 39% /
devfs devfs 1.0K 1.0K 0B 100% /dev
tmpfs tmpfs 4.0M 156K 3.8M 4% /var/run
devfs devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev

I have discovered the source of this problem - qemu-guest-agent.
I have ovirt 4.4.10 and I have installed pkg - qemu-guest-agent/
Config of qemu-ga :
cat rc.conf.local
qemu_guest_agent_enable="YES"
qemu_guest_agent_flags="-d -v -m virtio-serial -l /var/log/qemu-ga.log -p /dev/ttyV0.2"

ls -l /dev/vtcon
total 0
lrwxr-xr-x 1 root wheel 10 Apr 21 12:11 com.redhat.spice.0 -> ../ttyV0.3
lrwxr-xr-x 1 root wheel 10 Apr 21 12:11 org.qemu.guest_agent.0 -> ../ttyV0.2
lrwxr-xr-x 1 root wheel 10 Apr 21 12:11 ovirt-guest-agent.0 -> ../ttyV0.1

When I start qemu-guest-agent, it try to get information about filesystems and launch process 'df -hT', which becomes zombie.

I have used this instructions - https://forum.netgate.com/topic/162083/pfsense-vm-on-proxmox-qemu-agent-installation

[netblues]

@gofaizen No zombie processes here

Running under redhat 8.5 kvm

ls -l /dev/vtcon/
total 0
lrwxr-xr-x  1 root  wheel  10 Mar 19 07:01 com.redhat.spice.0 -> ../ttyV0.1
lrwxr-xr-x  1 root  wheel  10 Mar 19 07:01 org.qemu.guest_agent.0 -> ../ttyV0.2

df -hT

The ovrt-guest-agent.0 -> is missing though.

[gofaizen]

@netblues Please show config of qemu-guest-login - /etc/rc.conf.local

[netblues]

@gofaizen

root: cat /etc/rc.conf.local
qemu_guest_agent_enable="YES"
qemu_guest_agent_flags="-d -v -l /var/log/qemu-ga.log"
virtio_console_load="YES"

[netblues]

@gofaizen said in pfSense 2.6 problem with zombie processes:

qemu_guest_agent_flags="-d -v -m virtio-serial -l /var/log/qemu-ga.log -p /dev/ttyV0.2"

Where did you get this?

[gofaizen]

@netblues I have the same

[gofaizen]

@netblues This is command line from linux.
I have tried qemu_guest_agent_flags="-d -v -l /var/log/qemu-ga.log" and qemu_guest_agent_flags="-d -v -m virtio-serial -l /var/log/qemu-ga.log -p /dev/ttyV0.2"
Same effect - zombie process every minute

[netblues]

@gofaizen said in pfSense 2.6 problem with zombie processes:

@netblues This is command line from linux.
I have tried qemu_guest_agent_flags="-d -v -l /var/log/qemu-ga.log" and qemu_guest_agent_flags="-d -v -m virtio-serial -l /var/log/qemu-ga.log -p /dev/ttyV0.2"
Same effect - zombie process every minute

There is no -p /dev/ttyV0.2 in my config.

[stephenw10]

Hmm, it's unclear which of those things is a symptom.
Is the widget throwing that error because qemuguest is continually trying to access the disk?
Or is the agent unable to read the disk status because the widget is doping somthing wrong?
Or maybe both are failing because of some other issue.

If you disable the qemu agent do you still see the php error from the widget?

Steve

[gofaizen]

@stephenw10
Widget throws error in random order.
Error in widget and qemu-guest-agent not in any connect. When I disable qemu-guest-agent widget continues throwing errors in random order. It can work whole day in normal mode, or can throw error every 10 minutes. And it throws error every time after reboot.
qemu-guest-agent make zombie every time after getting data: {"execute":"guest-get-fsinfo"} message from oVirt.

[stephenw10]

Hmm, well that sounds more like both things are failing because of something else preventing access to the filesystem.

If you run /bin/df -hT manually at the command line repeatedly does it ever fail?

Can we see the output of: /bin/df -hT --libxo=json
Though that too might need to be run until it fails. That's what the widget is trying to do and choking on the output.

Steve

[gofaizen]

@stephenw10 Yes, I can run df -hT manually. I have checked this in first order.
": [{"name":"/dev/ufsid/60618bb1ebc69388","type":"ufs ","blocks":"5.5G","used":"2.0G","available":"3.1G","used-percent":39,"mounted-on":"/"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/dev"}, {"name":"tmpfs","type":"tmpfs","blocks":"4.0M","used":"152K","available":"3.9M","used-percent":4,"mounted-on":"/var/run"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/var/dhcpd/dev"}]}

[stephenw10]

But if you run /bin/df -hT manually at the command line repeatedly does it ever fail? Like after 100 tries?
It will likely only fails as often as the widget does. Or if you happen to run it when the qemu agent is trying to access the filesystem also.

The JSON output there seems to be missing the initial terms. I expect it to read like:

[22.05-DEVELOPMENT][admin@plusdev-2.stevew.lan]/root: /bin/df -hT --libxo=json
{"storage-system-information": {"filesystem": [{"name":"/dev/ufsid/626069f74a9f0e6e","type":"ufs  ","blocks":"9.2G","used":"1.5G","available":"7.0G","used-percent":18,"mounted-on":"/"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/dev"}, {"name":"tmpfs","type":"tmpfs","blocks":"4.0M","used":"112K","available":"3.9M","used-percent":3,"mounted-on":"/var/run"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/var/dhcpd/dev"}]}
}

Steve

[gofaizen]

@stephenw10
{"storage-system-information": {"filesystem": [{"name":"/dev/ufsid/60618bb1ebc69388","type":"ufs ","blocks":"5.5G","used":"2.0G","available":"3.1G","used-percent":39,"mounted-on":"/"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/dev"}, {"name":"tmpfs","type":"tmpfs","blocks":"4.0M","used":"152K","available":"3.9M","used-percent":4,"mounted-on":"/var/run"}, {"name":"devfs","type":"devfs","blocks":"1.0K","used":"1.0K","available":"0B","used-percent":100,"mounted-on":"/var/dhcpd/dev"}]}
}
Possibly I have copied part of output.

Tried to run /bin/df -hT --libxo=json in loop 1000 times. All times were successful.

#!/bin/sh

for i in seq 1 1000
do
/bin/df -hT --libxo=json
done

[stephenw10]

Hmm. If you remove the qemu-agent can I assume this goes away?

[gofaizen]

@stephenw10
When I stopped qemu-ga service zombie processes died.
I have checked it several times. But I need qemu-ga in my oVirt installation.

[stephenw10]

Hmm, well can you configure it not to query the disk status? I would assume that also solves it.

Can you test qemu-agent in a FreeBSD 12.3 install? Hard to see what but it might be something in base that's changed.

Steve

[gofaizen]

@stephenw10
I am trying to disable query for file system status in qemu-ga.