Haproxy 100% cpu usage

[maverick_slo]

Hi.

After I upgraded to pfsense plus 24.03-RELEASE (amd64) I have problems with HAPROXY.
After some random time it just starts to use 100% CPU even though there are virtuallly NO requests and no traffic at all...
In previous versions this did not happen.
Haprox is 2.9.1 can you please upgrade to latest binary which will probably solve the issue?

Bug:
https://github.com/replicatedhq/ekco/pull/214

Workaround:
https://github.com/haproxy/haproxy/issues/2402#issuecomment-1875104101

login-to-view

Thanks!

[DaniloZ]

@maverick_slo

I have submitted a bug report. Thank you!

https://redmine.pfsense.org/issues/15457

[maverick_slo]

Anyone?

Is updating binary really so hard?

[maverick_slo]

Can someone pretty please update binary to latest?
There are many things fixed in latest version.

[rootbg]

Hi,

I'm having the same problem like @maverick_slo after the latest pfSense upgrade I have 100% CPU from haproxy after working couple of days. @DaniloZ is there any news about this issue?

[maverick_slo]

Pleeease? :)

[coreybrett]

I have this issue also. Is there a work-around ?

[stephenw10]

Looks like we lost the screenshot from the first post.

Does this actually cause a kernel panic when you hit it or just uses all the available CPU time?

Steve

[maverick_slo]

@stephenw10
Just cpu usage.
Updated versions resolve the issue
Also workaround is working and if applied cpu is ok.

[maverick_slo]

@coreybrett
Come on dude, read a little it is in the first post I made.

[coreybrett]

Adding...

tune.disable-zero-copy-forwarding

did not fix for me.

[maverick_slo]

@coreybrett
Did you entered it in correct settings tab?

[coreybrett]

login-to-view

login-to-view

[coreybrett]

login-to-view

After a stop/start, the HAP process hit 100% and stops responding to inbound requests.

[Luca De Andreis]

@stephenw10

I can confirm. in my case haproxy generates a kernel panic (occasionally). pfsense 24.03 is stable (more than 30 installations in my case) only with the use of haproxy do kernel panics occur. in this case I went back to the previous version.

Luca

[stephenw10]

Do you also see the high CPU usage?

[Luca De Andreis]

@stephenw10

We have two PfSense with HAProxy (dedicated) in 24.03, on the site with the most traffic (a while ago now), we noticed that the VM would reboot going the crash.
Both VMs were experiencing abnormal CPU load (not 100%).
On the third crash I rolled back to 23.09.1 and it has been working perfectly ever since.
The site with less load at the HAProxy level (I left it only for testing in 24.03) never crashed but still has abnormal CPU load (30-40% without doing practically anything).
The site in production with a fair number of accesses, stayed UP 3-5 days, then crashed. After downgrade to version 23.09.1.... perfect.

All PfSense works in VM and we have about 30 of them, only those with HAProxy gave this problem with 24.03.

[maverick_slo]

@stephenw10 why are we discussing this when solution is crystal clear?
Binary has to be updated, end of story.

[stephenw10]

Yup, raised it internally. But trying to determine if the reported kernel panic is actually related.

[Luca De Andreis]

@stephenw10

For me, yes, it's related.

In my case a PfSense 24.03 VM (with HA Proxy 2.9.1) crashed (and therefore auto rebooted) every 3-5 days.
With a usage load... typical of a reverse proxy in production.

The same VM but with almost no workload, it never crashed apart from an anomalous CPU load.

In my opinion yes, the events are correlated, but only when the reverse proxy is used above a certain threshold.

In this regard I had also opened a ticket via "professional" support, which was closed with the response... "HAProxy is a third-party package, its update is managed in best effort" .... closed.