HAProxy backend port changes are not applied

nick23369

Hi, just in case someone has the same issue still in 2025. I'm using pfsense 2.8 and haproxy 0.63_10 and I got the same problem: changing the backend port, is not taking effect unless you delete the haproxy_server_state ans reload haproxy (at least this worked for me). It would be good if this process could be automated anytime your reload/restart haproxy. Or itmight be thatI'm missing something.

andrew_cb

I have been experiencing this issue for a while now, so it is nice to know the cause!

I updated the redmine and posted screenshots of the issue.
https://redmine.pfsense.org/issues/15274

andrew_cb

HAProxy has a directives load-server-state-from-file that is described as:

This directive points HAProxy to a file where server state from previous
running process has been saved. That way, when starting up, before handling
traffic, the new process can apply old states to servers exactly has if no
reload occurred.

and also the directive server-state-file which is described as:

Before reloading HAProxy, it is possible to save the
servers' current state using the stats command "show servers state". The
output of this command must be written in the file pointed by <file>. When
starting up, before handling traffic, HAProxy will read, load and apply state
for each server found in the file and available in its current running
configuration.

https://docs.haproxy.org/2.9/configuration.html#load-server-state-from-file
https://docs.haproxy.org/2.9/configuration.html#server-state-file

Looking in my /var/etc/haproxy/haproxy.cfg shows the global section contains the directive

server-state-file /tmp/haproxy_server_state

and each backend contains the directive

load-server-state-from-file	global

I think this explains why we are seeing the issue - when HAproxy reloads, it writes the current server state to the file and then loads the states back from the file. So even though the configuration has been changed, HAproxy is still using the server states that existed before making the changes.

This also explains why deleting a backend, saving, and then re-creating the backend is a workaround - when a backend server is deleted, all the HAproxy states for it will be closed and thus do not get saved to the file.

They mystery now is why those directives are being added to the HAproxy configuration. I do not see an GUI option for either directive.

andrew_cb

I found what is adding these directives to the config file!

The behavior is hard-coded in /usr/local/pkg/haproxy/haproxy.inc which is what generates /var/etc/haproxy/haproxy.cfg (Line numbers are from pfSense 25.07-Release)

Line 727
function write_backend($configpath, $fd, $name, $pool, $backendsettings) {
                    ...
    Line 1006  (always executes, not subject to any conditional statements)
    fwrite ($fd, "\tload-server-state-from-file\tglobal\n");

Line 1520
function haproxy_writeconf($configpath) {
                    ...
    Line 1612 (always executes, not subject to any conditional statements)
    fwrite ($fd, "\tserver-state-file /tmp/haproxy_server_state\n");

Line 2498
function haproxy_check_run($reload) {
                    ...
    Lines 2507-2508 (executes if haproxy is running when a reload is requested, which is almost always true)
    if ($reload) {
        if (haproxy_is_running()) {
	    $r = haproxy_socket_command("show servers state");
            file_put_contents("/tmp/haproxy_server_state", $r);
		}

These lines result in the following behavior:

Whenever HAProxy's settings are saved using the GUI, the server-state-file directive is added to the global section of the resulting config file.
Whenever HAProxy's settings are saved using the GUI, the load-server-state-from-file directive is added to each backend of the resulting config file.
Whenever HAProxy reloads, it writes the backend server state to /tmp/haproxy_server_state, and due to the server-state-file directive, it then always reads and applies the previous state data.

There is no GUI text that mentions that this behavior is hard-coded, and there is no way to disabled it other than commenting out the lines (1006, 1612, 2507, 2508) in haproxy.inc that are responsible for adding the directives to the config file.

The GUI option Reload behaviour is responsible for the directive hard_stop_after which is for a different behavior.

I've added these findings to redmine 15274

andrew_cb

The line

fwrite ($fd, "\tload-server-state-from-file\tglobal\n");

was added to write_backend in commit 9f7d258 (Aug 10, 2023):

https://github.com/pfsense/FreeBSD-ports/commit/9f7d258917ece10b6d55435776d2db85370e289c

andrew_cb

Adding

load-server-state-from-file	none

to the Advanced Settings > Backend pass thru section of each backend overrides the behavior and makes backend changes apply immediately when reloading.
I am also using the global GUI setting Force immediate stop of old process on reload. (closes existing connections).

nick23369

@andrew_cb Thank you very much for this, I just tried your proposed solution and it did work! That was driving me crasy! Way simpler than deleting the haproxy_server_state file.

andrew_cb

Update: A fix for this issue will be included in pfSense 25.11.

nick23369

@andrew_cb That’s a good news. Will this fix find his way also on the community edition? I’m asking because I am on pfsense ce 2.8.0

andrew_cb

@nick23369 said in HAProxy backend port changes are not applied:

@andrew_cb That’s a good news. Will this fix find his way also on the community edition? I’m asking because I am on pfsense ce 2.8.0

Eventually it will, probably in CE 2.9, but the date for that is unknown. It will probably be 12-18 months based on the time between CE 2.7.2 (Plus 23.09.1) to 2.8 (Plus 25.07).

andrew_cb

@nick23369 You don't have to wait for the official fix. Adding the directive

load-server-state-from-file	none

to each backend will override the default behavior and makes HAProxy backend changes take effect immediately. This is the easiest - do it one time and it's done.

You can also stop HAProxy, delete run

rm /tmp/haproxy_server_state

from Diagnostic > Command Prompt, and then start HAProxy.

There is no problem with HAProxy, it just takes some extra work to make negate the hardcoded config settings and make backend changes apply immediately without having to reboot pfSense.