HAProxy backend port changes are not applied

safe

In a basic setup with a frontend and a backend, set the port to 80 in the backend. Save and apply the settings. You can now reach the backend service running on port 80. If you on the same backend also have a different service on for example port 81, and you change the backend port to 81, then save and reload, your browser will still see the service on port 80. It doesn't matter if you restart the HAProxy service, stop and start it, it is still serving the service on port 80. If you reboot the whole PFSense box, it switches to port 81.

I found out that the problem is related to the file /tmp/haproxy_server_state. This file shows the old port even though the generated haproxy.cfg file shows the correct port. If I stop the HAProxy service, then delete this file and start the service again, it will serve the correct backend port.

Is this a bug, or is it intentionally?

23.01, with haproxy-devel (haproxy-2.6.6)
Thanks

rpontual

@safe I still see the same problem.

PFSENSE 23.09.1
HAPROXY 2.8.2 (package v 0.63_2)

safe

@rpontual As a workaround, if I remember correctly, you can rename the backend when saving the new port, then rename it back (if you want to), and save again. Then it should work. Easier than deleting the file.

pfpv

Wow! I struggled for hours, came here to report this problem and found this thread. I am on 2.7.2 CE and haproxy-devel 0.63_2. devel and non-devel packages have the same version (strange) but different dependencies. First I thought I made a mistake installing devel but it looks like the non-devel package has the same problem.

My findings are exactly as @safe's. I can only add that after I restart the haproxy service I can see in stats that the "backend" line of my backend shows something like 15s UP but the associated "MyBackendName" line shows something like 8m 23s UP, as it was never killed and restarted.

The workaround I found is to clone the backend, delete the old one and rename the new and double-check the frontend but I will try @safe's workaround that seems a little easier.

Since this hasn't gained attention since Feb. last year, how can report this to the developer?

bbrendon

OMFG this workaround is still valid. I just spent an hour wondering what I was doing wrong.

ix-kilian

Seems to be related to the state file: /tmp/haproxy_server_state

Login via CLI, delete the file and reload haproxy - worked for me.

nick23369

Hi, just in case someone has the same issue still in 2025. I'm using pfsense 2.8 and haproxy 0.63_10 and I got the same problem: changing the backend port, is not taking effect unless you delete the haproxy_server_state ans reload haproxy (at least this worked for me). It would be good if this process could be automated anytime your reload/restart haproxy. Or itmight be thatI'm missing something.

andrew_cb

I have been experiencing this issue for a while now, so it is nice to know the cause!

I updated the redmine and posted screenshots of the issue.
https://redmine.pfsense.org/issues/15274

andrew_cb

HAProxy has a directives load-server-state-from-file that is described as:

This directive points HAProxy to a file where server state from previous
running process has been saved. That way, when starting up, before handling
traffic, the new process can apply old states to servers exactly has if no
reload occurred.

and also the directive server-state-file which is described as:

Before reloading HAProxy, it is possible to save the
servers' current state using the stats command "show servers state". The
output of this command must be written in the file pointed by <file>. When
starting up, before handling traffic, HAProxy will read, load and apply state
for each server found in the file and available in its current running
configuration.

https://docs.haproxy.org/2.9/configuration.html#load-server-state-from-file
https://docs.haproxy.org/2.9/configuration.html#server-state-file

Looking in my /var/etc/haproxy/haproxy.cfg shows the global section contains the directive

server-state-file /tmp/haproxy_server_state

and each backend contains the directive

load-server-state-from-file	global

I think this explains why we are seeing the issue - when HAproxy reloads, it writes the current server state to the file and then loads the states back from the file. So even though the configuration has been changed, HAproxy is still using the server states that existed before making the changes.

This also explains why deleting a backend, saving, and then re-creating the backend is a workaround - when a backend server is deleted, all the HAproxy states for it will be closed and thus do not get saved to the file.

They mystery now is why those directives are being added to the HAproxy configuration. I do not see an GUI option for either directive.

andrew_cb

I found what is adding these directives to the config file!

The behavior is hard-coded in /usr/local/pkg/haproxy/haproxy.inc which is what generates /var/etc/haproxy/haproxy.cfg (Line numbers are from pfSense 25.07-Release)

Line 727
function write_backend($configpath, $fd, $name, $pool, $backendsettings) {
                    ...
    Line 1006  (always executes, not subject to any conditional statements)
    fwrite ($fd, "\tload-server-state-from-file\tglobal\n");

Line 1520
function haproxy_writeconf($configpath) {
                    ...
    Line 1612 (always executes, not subject to any conditional statements)
    fwrite ($fd, "\tserver-state-file /tmp/haproxy_server_state\n");

Line 2498
function haproxy_check_run($reload) {
                    ...
    Lines 2507-2508 (executes if haproxy is running when a reload is requested, which is almost always true)
    if ($reload) {
        if (haproxy_is_running()) {
	    $r = haproxy_socket_command("show servers state");
            file_put_contents("/tmp/haproxy_server_state", $r);
		}

These lines result in the following behavior:

Whenever HAProxy's settings are saved using the GUI, the server-state-file directive is added to the global section of the resulting config file.
Whenever HAProxy's settings are saved using the GUI, the load-server-state-from-file directive is added to each backend of the resulting config file.
Whenever HAProxy reloads, it writes the backend server state to /tmp/haproxy_server_state, and due to the server-state-file directive, it then always reads and applies the previous state data.

There is no GUI text that mentions that this behavior is hard-coded, and there is no way to disabled it other than commenting out the lines (1006, 1612, 2507, 2508) in haproxy.inc that are responsible for adding the directives to the config file.

The GUI option Reload behaviour is responsible for the directive hard_stop_after which is for a different behavior.

I've added these findings to redmine 15274

andrew_cb

The line

fwrite ($fd, "\tload-server-state-from-file\tglobal\n");

was added to write_backend in commit 9f7d258 (Aug 10, 2023):

https://github.com/pfsense/FreeBSD-ports/commit/9f7d258917ece10b6d55435776d2db85370e289c

andrew_cb

Adding

load-server-state-from-file	none

to the Advanced Settings > Backend pass thru section of each backend overrides the behavior and makes backend changes apply immediately when reloading.
I am also using the global GUI setting Force immediate stop of old process on reload. (closes existing connections).

nick23369

@andrew_cb Thank you very much for this, I just tried your proposed solution and it did work! That was driving me crasy! Way simpler than deleting the haproxy_server_state file.

andrew_cb

Update: A fix for this issue will be included in pfSense 25.11.

nick23369

@andrew_cb That’s a good news. Will this fix find his way also on the community edition? I’m asking because I am on pfsense ce 2.8.0

andrew_cb

@nick23369 said in HAProxy backend port changes are not applied:

@andrew_cb That’s a good news. Will this fix find his way also on the community edition? I’m asking because I am on pfsense ce 2.8.0

Eventually it will, probably in CE 2.9, but the date for that is unknown. It will probably be 12-18 months based on the time between CE 2.7.2 (Plus 23.09.1) to 2.8 (Plus 25.07).

andrew_cb

@nick23369 You don't have to wait for the official fix. Adding the directive

load-server-state-from-file	none

to each backend will override the default behavior and makes HAProxy backend changes take effect immediately. This is the easiest - do it one time and it's done.

You can also stop HAProxy, delete run

rm /tmp/haproxy_server_state

from Diagnostic > Command Prompt, and then start HAProxy.

There is no problem with HAProxy, it just takes some extra work to make negate the hardcoded config settings and make backend changes apply immediately without having to reboot pfSense.