HAProxy backend port changes are not applied

rpontual

@safe I still see the same problem.

PFSENSE 23.09.1
HAPROXY 2.8.2 (package v 0.63_2)

safe

@rpontual As a workaround, if I remember correctly, you can rename the backend when saving the new port, then rename it back (if you want to), and save again. Then it should work. Easier than deleting the file.

pfpv

Wow! I struggled for hours, came here to report this problem and found this thread. I am on 2.7.2 CE and haproxy-devel 0.63_2. devel and non-devel packages have the same version (strange) but different dependencies. First I thought I made a mistake installing devel but it looks like the non-devel package has the same problem.

My findings are exactly as @safe's. I can only add that after I restart the haproxy service I can see in stats that the "backend" line of my backend shows something like 15s UP but the associated "MyBackendName" line shows something like 8m 23s UP, as it was never killed and restarted.

The workaround I found is to clone the backend, delete the old one and rename the new and double-check the frontend but I will try @safe's workaround that seems a little easier.

Since this hasn't gained attention since Feb. last year, how can report this to the developer?

bbrendon

OMFG this workaround is still valid. I just spent an hour wondering what I was doing wrong.

ix-kilian

Seems to be related to the state file: /tmp/haproxy_server_state

Login via CLI, delete the file and reload haproxy - worked for me.

nick23369

Hi, just in case someone has the same issue still in 2025. I'm using pfsense 2.8 and haproxy 0.63_10 and I got the same problem: changing the backend port, is not taking effect unless you delete the haproxy_server_state ans reload haproxy (at least this worked for me). It would be good if this process could be automated anytime your reload/restart haproxy. Or itmight be thatI'm missing something.

andrew_cb

I have been experiencing this issue for a while now, so it is nice to know the cause!

I updated the redmine and posted screenshots of the issue.
https://redmine.pfsense.org/issues/15274

andrew_cb

HAProxy has a directives load-server-state-from-file that is described as:

This directive points HAProxy to a file where server state from previous
running process has been saved. That way, when starting up, before handling
traffic, the new process can apply old states to servers exactly has if no
reload occurred.

and also the directive server-state-file which is described as:

Before reloading HAProxy, it is possible to save the
servers' current state using the stats command "show servers state". The
output of this command must be written in the file pointed by <file>. When
starting up, before handling traffic, HAProxy will read, load and apply state
for each server found in the file and available in its current running
configuration.

https://docs.haproxy.org/2.9/configuration.html#load-server-state-from-file
https://docs.haproxy.org/2.9/configuration.html#server-state-file

Looking in my /var/etc/haproxy/haproxy.cfg shows the global section contains the directive

server-state-file /tmp/haproxy_server_state

and each backend contains the directive

load-server-state-from-file	global

I think this explains why we are seeing the issue - when HAproxy reloads, it writes the current server state to the file and then loads the states back from the file. So even though the configuration has been changed, HAproxy is still using the server states that existed before making the changes.

This also explains why deleting a backend, saving, and then re-creating the backend is a workaround - when a backend server is deleted, all the HAproxy states for it will be closed and thus do not get saved to the file.

They mystery now is why those directives are being added to the HAproxy configuration. I do not see an GUI option for either directive.

andrew_cb

I found what is adding these directives to the config file!

The behavior is hard-coded in /usr/local/pkg/haproxy/haproxy.inc which is what generates /var/etc/haproxy/haproxy.cfg (Line numbers are from pfSense 25.07-Release)

Line 727
function write_backend($configpath, $fd, $name, $pool, $backendsettings) {
                    ...
    Line 1006  (always executes, not subject to any conditional statements)
    fwrite ($fd, "\tload-server-state-from-file\tglobal\n");

Line 1520
function haproxy_writeconf($configpath) {
                    ...
    Line 1612 (always executes, not subject to any conditional statements)
    fwrite ($fd, "\tserver-state-file /tmp/haproxy_server_state\n");

Line 2498
function haproxy_check_run($reload) {
                    ...
    Lines 2507-2508 (executes if haproxy is running when a reload is requested, which is almost always true)
    if ($reload) {
        if (haproxy_is_running()) {
	    $r = haproxy_socket_command("show servers state");
            file_put_contents("/tmp/haproxy_server_state", $r);
		}

These lines result in the following behavior:

Whenever HAProxy's settings are saved using the GUI, the server-state-file directive is added to the global section of the resulting config file.
Whenever HAProxy's settings are saved using the GUI, the load-server-state-from-file directive is added to each backend of the resulting config file.
Whenever HAProxy reloads, it writes the backend server state to /tmp/haproxy_server_state, and due to the server-state-file directive, it then always reads and applies the previous state data.

There is no GUI text that mentions that this behavior is hard-coded, and there is no way to disabled it other than commenting out the lines (1006, 1612, 2507, 2508) in haproxy.inc that are responsible for adding the directives to the config file.

The GUI option Reload behaviour is responsible for the directive hard_stop_after which is for a different behavior.

I've added these findings to redmine 15274

andrew_cb

The line

fwrite ($fd, "\tload-server-state-from-file\tglobal\n");

was added to write_backend in commit 9f7d258 (Aug 10, 2023):

https://github.com/pfsense/FreeBSD-ports/commit/9f7d258917ece10b6d55435776d2db85370e289c

andrew_cb

Adding

load-server-state-from-file	none

to the Advanced Settings > Backend pass thru section of each backend overrides the behavior and makes backend changes apply immediately when reloading.
I am also using the global GUI setting Force immediate stop of old process on reload. (closes existing connections).

nick23369

@andrew_cb Thank you very much for this, I just tried your proposed solution and it did work! That was driving me crasy! Way simpler than deleting the haproxy_server_state file.