Haproxy 100% cpu usage

maverick_slo

@stephenw10 why are we discussing this when solution is crystal clear?
Binary has to be updated, end of story.

stephenw10

Yup, raised it internally. But trying to determine if the reported kernel panic is actually related.

Luca De Andreis

For me, yes, it's related.

In my case a PfSense 24.03 VM (with HA Proxy 2.9.1) crashed (and therefore auto rebooted) every 3-5 days.
With a usage load... typical of a reverse proxy in production.

The same VM but with almost no workload, it never crashed apart from an anomalous CPU load.

In my opinion yes, the events are correlated, but only when the reverse proxy is used above a certain threshold.

In this regard I had also opened a ticket via "professional" support, which was closed with the response... "HAProxy is a third-party package, its update is managed in best effort" .... closed.

stephenw10

Hmm, nothing much TAC can do beyond opening the bug report, which they did.

We have devs looking at it now.

Luca De Andreis

@stephenw10

ahh I forgot.

After opening the support ticket (and before downgrading to version 23.09.1) I applied the "disable zero copy forwarding" workaround.

The CPU returned to normal on the VM with HAProxy in production... unfortunately after a few days the crash and related reboot occurred again.

I then rolled back to 23.09.1 and the system has been completely rock solid ever since.

Luca

stephenw10

OK the backend haproxy pkg had been updated to current (2.9.7) and looks good in testing here.

You can upgrade it dircetly like: pkg upgrade haproxy

Or reinstall the pfSense HAProxy package to get it.

If this confirms the fix and doesn't introduce any new show stoppers we'll bump the package version so it appears as an upgrade for all users.

Steve

Luca De Andreis

@stephenw10

... upgrade to 2.9.7 in production and... for now... works fine :)

Thanks
Luca

maverick_slo

Same here, removed workaround as well
All good last 8 hours ..

stephenw10

Great, thanks for testing.

coreybrett

I upgraded last night.

Still having same issue.

maverick_slo

@coreybrett you probably did not restart service after upgrade.
Please restart service so that new binary can be run...

coreybrett

@maverick_slo I am pretty sure I rebooted after running the package update
will do so again and check it in the morning

stephenw10

You should see it in the stats page:

HAProxy version 2.9.7-5742051, released 2024/04/05
Statistics Report for pid 81592

Luca De Andreis

@stephenw10

I can confirm that after this uptime:

HAProxy on a normal production load and without workaround applied... works fine:

coreybrett

Still having trouble with this.

Shell Output - /usr/local/sbin/haproxy -v
HAProxy version 2.9.7-5742051 2024/04/05 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.9.7.html
Running on: FreeBSD 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_03-n256311-e71f834dd81: Fri Apr 19 00:28:14 UTC 2024     root@freebsd:/var/jenkins/workspac