24.03 crashing (again)

rocketboy001

Yep, it occurred last night.

Luca De Andreis

Another crash running HAProxy 2.9.7

stephenw10

Hmm, same crash again. So it appears that panic is unrelated to that known issue with HAProxy using 100% CPU which should now be fixed.

Luca De Andreis

@stephenw10
Yes I can confirm. HAProxy 2.9.7 never use 100% CPU

Luca De Andreis

@stephenw10

But.
I have another PfSense plus with HAproxy 2.9.7, very little traffic, almost nothing. Well, that PfSense has never presented any problems.
The problem is related not only to the presence of HAProxy 2.9.7 but also to the traffic or use of it.

Luca De Andreis

@stephenw10

or....

is there a correlation between HAProxy 2.9.7 with the VM's virtual CPU ? In my case both VM running in Proxmox 8.2.2 (same version of QEMU, identical).
On the version that has NEVER given problems (and is very low traffic):

And on the version that has crashes:

stephenw10

Hmm, possibly some new instruction that HAProxy is using (or trying to use)?

If it was that expect to see it in some crypto operation but the backtrace doesn't look like that, it's in the network stack.

Is there any difference in the network config of those VMs?

Luca De Andreis

@stephenw10

mmmmm no.
I just checked and the configuration of the two network interfaces going to Pfsense is completely identical.
N.2 virtio type NICs with same configuration running on the same version of QEMU.

stephenw10

Hmm, OK the next step here is probably to enable a full kernel core dump and wait for it to happen again.

Do you have SWAP enabled on that VM? How much?

Luca De Andreis

@stephenw10

OK, if I can help, by installing a new kernel configured for debugging in case of core dump on the production HAProxy server, I'm available.

stephenw10

Ok the first step is to enable a full core dump. Edit the file /etc/pfSense-ddb.conf and add a new kdb.enter.default script line like:

# $FreeBSD$
#
#  This file is read when going to multi-user and its contents piped thru
#  ``ddb'' to define debugging scripts.
#
# see ``man 4 ddb'' and ``man 8 ddb'' for details.
#

script lockinfo=show locks; show alllocks; show lockedvnods
script pfs=bt ; show registers ; show pcpu ; run lockinfo ; acttrace ; ps ; alltrace

# kdb.enter.panic       panic(9) was called.
# script kdb.enter.default=textdump set; capture on; run pfs ; capture off; textdump dump; reset
script kdb.enter.default=bt ; show registers ; dump ; reset

# kdb.enter.witness	witness(4) detected a locking error.
script kdb.enter.witness=run lockinfo

So there I commented out the old line and added: script kdb.enter.default=bt ; show registers ; dump ; reset

Now reboot as that is only read in at boot.

Then check it's present at the CLI with:

[24.08-DEVELOPMENT][root@7100.stevew.lan]/root: sysctl debug.ddb.scripting.scripts
debug.ddb.scripting.scripts: lockinfo=show locks; show alllocks; show lockedvnods
pfs=bt ; show registers ; show pcpu ; run lockinfo ; acttrace ; ps ; alltrace
kdb.enter.default=bt ; show registers ; dump ; reset
kdb.enter.witness=run lockinfo

It will now dump the full vmcore after a panic.
You can check it by manually triggering a panic with: sysctl sysctl debug.kdb.panic=1

At the console you will see something like:

db:0:kdb.enter.default>  dump
Dumping 586 out of 8118 MB:..3%..11%..22%..33%..41%..52%..63%..71%..82%..93%
Dump complete
db:0:kdb.enter.default>  reset
Uptime: 17m8s

The available SWAP space must be larger than the used RAM though. That 7100 is only using 586MB because it's a test box.

marcosm

For reference:
https://redmine.pfsense.org/issues/15618