Haproxy 100% cpu usage

[maverick_slo]

Pleeease? :)

[coreybrett]

I have this issue also. Is there a work-around ?

[stephenw10]

Looks like we lost the screenshot from the first post.

Does this actually cause a kernel panic when you hit it or just uses all the available CPU time?

Steve

[maverick_slo]

@stephenw10
Just cpu usage.
Updated versions resolve the issue
Also workaround is working and if applied cpu is ok.

[maverick_slo]

@coreybrett
Come on dude, read a little it is in the first post I made.

[coreybrett]

Adding...

tune.disable-zero-copy-forwarding

did not fix for me.

[maverick_slo]

@coreybrett
Did you entered it in correct settings tab?

[coreybrett]

[coreybrett]

After a stop/start, the HAP process hit 100% and stops responding to inbound requests.

[Luca De Andreis]

@stephenw10

I can confirm. in my case haproxy generates a kernel panic (occasionally). pfsense 24.03 is stable (more than 30 installations in my case) only with the use of haproxy do kernel panics occur. in this case I went back to the previous version.

Luca

[stephenw10]

Do you also see the high CPU usage?

[Luca De Andreis]

@stephenw10

We have two PfSense with HAProxy (dedicated) in 24.03, on the site with the most traffic (a while ago now), we noticed that the VM would reboot going the crash.
Both VMs were experiencing abnormal CPU load (not 100%).
On the third crash I rolled back to 23.09.1 and it has been working perfectly ever since.
The site with less load at the HAProxy level (I left it only for testing in 24.03) never crashed but still has abnormal CPU load (30-40% without doing practically anything).
The site in production with a fair number of accesses, stayed UP 3-5 days, then crashed. After downgrade to version 23.09.1.... perfect.

All PfSense works in VM and we have about 30 of them, only those with HAProxy gave this problem with 24.03.

[maverick_slo]

@stephenw10 why are we discussing this when solution is crystal clear?
Binary has to be updated, end of story.

[stephenw10]

Yup, raised it internally. But trying to determine if the reported kernel panic is actually related.

[Luca De Andreis]

@stephenw10

For me, yes, it's related.

In my case a PfSense 24.03 VM (with HA Proxy 2.9.1) crashed (and therefore auto rebooted) every 3-5 days.
With a usage load... typical of a reverse proxy in production.

The same VM but with almost no workload, it never crashed apart from an anomalous CPU load.

In my opinion yes, the events are correlated, but only when the reverse proxy is used above a certain threshold.

In this regard I had also opened a ticket via "professional" support, which was closed with the response... "HAProxy is a third-party package, its update is managed in best effort" .... closed.

[stephenw10]

Hmm, nothing much TAC can do beyond opening the bug report, which they did.

We have devs looking at it now.

[Luca De Andreis]

@stephenw10

ahh I forgot.

After opening the support ticket (and before downgrading to version 23.09.1) I applied the "disable zero copy forwarding" workaround.

The CPU returned to normal on the VM with HAProxy in production... unfortunately after a few days the crash and related reboot occurred again.

I then rolled back to 23.09.1 ​​and the system has been completely rock solid ever since.

Luca

[stephenw10]

OK the backend haproxy pkg had been updated to current (2.9.7) and looks good in testing here.

You can upgrade it dircetly like: pkg upgrade haproxy

Or reinstall the pfSense HAProxy package to get it.

If this confirms the fix and doesn't introduce any new show stoppers we'll bump the package version so it appears as an upgrade for all users.

Steve

[Luca De Andreis]

@stephenw10

... upgrade to 2.9.7 in production and... for now... works fine :)

Thanks
Luca

[maverick_slo]

Same here, removed workaround as well
All good last 8 hours ..