Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    HAProxy backend port changes are not applied

    Scheduled Pinned Locked Moved Cache/Proxy
    10 Posts 7 Posters 1.9k Views 8 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S Offline
      safe
      last edited by

      In a basic setup with a frontend and a backend, set the port to 80 in the backend. Save and apply the settings. You can now reach the backend service running on port 80. If you on the same backend also have a different service on for example port 81, and you change the backend port to 81, then save and reload, your browser will still see the service on port 80. It doesn't matter if you restart the HAProxy service, stop and start it, it is still serving the service on port 80. If you reboot the whole PFSense box, it switches to port 81.

      I found out that the problem is related to the file /tmp/haproxy_server_state. This file shows the old port even though the generated haproxy.cfg file shows the correct port. If I stop the HAProxy service, then delete this file and start the service again, it will serve the correct backend port.

      Is this a bug, or is it intentionally?

      23.01, with haproxy-devel (haproxy-2.6.6)
      Thanks

      R P 2 Replies Last reply Reply Quote 3
      • R Offline
        rpontual @safe
        last edited by

        @safe I still see the same problem.

        PFSENSE 23.09.1
        HAPROXY 2.8.2 (package v 0.63_2)

        S 1 Reply Last reply Reply Quote 0
        • S Offline
          safe @rpontual
          last edited by

          @rpontual As a workaround, if I remember correctly, you can rename the backend when saving the new port, then rename it back (if you want to), and save again. Then it should work. Easier than deleting the file.

          1 Reply Last reply Reply Quote 0
          • P Offline
            pfpv @safe
            last edited by

            Wow! I struggled for hours, came here to report this problem and found this thread. I am on 2.7.2 CE and haproxy-devel 0.63_2. devel and non-devel packages have the same version (strange) but different dependencies. First I thought I made a mistake installing devel but it looks like the non-devel package has the same problem.

            My findings are exactly as @safe's. I can only add that after I restart the haproxy service I can see in stats that the "backend" line of my backend shows something like 15s UP but the associated "MyBackendName" line shows something like 8m 23s UP, as it was never killed and restarted.

            The workaround I found is to clone the backend, delete the old one and rename the new and double-check the frontend but I will try @safe's workaround that seems a little easier.

            Since this hasn't gained attention since Feb. last year, how can report this to the developer?

            1 Reply Last reply Reply Quote 0
            • B Offline
              bbrendon
              last edited by

              OMFG this workaround is still valid. I just spent an hour wondering what I was doing wrong.

              1 Reply Last reply Reply Quote 0
              • I Offline
                ix-kilian
                last edited by

                Seems to be related to the state file: /tmp/haproxy_server_state

                Login via CLI, delete the file and reload haproxy - worked for me.

                1 Reply Last reply Reply Quote 0
                • P pfpv referenced this topic on
                • N Offline
                  nick23369
                  last edited by

                  Hi, just in case someone has the same issue still in 2025. I'm using pfsense 2.8 and haproxy 0.63_10 and I got the same problem: changing the backend port, is not taking effect unless you delete the haproxy_server_state ans reload haproxy (at least this worked for me). It would be good if this process could be automated anytime your reload/restart haproxy. Or itmight be thatI'm missing something.

                  1 Reply Last reply Reply Quote 1
                  • A Online
                    andrew_cb
                    last edited by

                    I have been experiencing this issue for a while now, so it is nice to know the cause!

                    I updated the redmine and posted screenshots of the issue.
                    https://redmine.pfsense.org/issues/15274

                    1 Reply Last reply Reply Quote 0
                    • A Online
                      andrew_cb
                      last edited by

                      HAProxy has a directives load-server-state-from-file that is described as:

                      This directive points HAProxy to a file where server state from previous
                      running process has been saved. That way, when starting up, before handling
                      traffic, the new process can apply old states to servers exactly has if no
                      reload occurred.
                      

                      and also the directive server-state-file which is described as:

                      Before reloading HAProxy, it is possible to save the
                      servers' current state using the stats command "show servers state". The
                      output of this command must be written in the file pointed by <file>. When
                      starting up, before handling traffic, HAProxy will read, load and apply state
                      for each server found in the file and available in its current running
                      configuration.
                      

                      https://docs.haproxy.org/2.9/configuration.html#load-server-state-from-file
                      https://docs.haproxy.org/2.9/configuration.html#server-state-file

                      Looking in my /var/etc/haproxy/haproxy.cfg shows the global section contains the directive

                      server-state-file /tmp/haproxy_server_state
                      

                      and each backend contains the directive

                      load-server-state-from-file	global
                      

                      I think this explains why we are seeing the issue - when HAproxy reloads, it writes the current server state to the file and then loads the states back from the file. So even though the configuration has been changed, HAproxy is still using the server states that existed before making the changes.

                      This also explains why deleting a backend, saving, and then re-creating the backend is a workaround - when a backend server is deleted, all the HAproxy states for it will be closed and thus do not get saved to the file.

                      They mystery now is why those directives are being added to the HAproxy configuration. I do not see an GUI option for either directive.

                      1 Reply Last reply Reply Quote 0
                      • A Online
                        andrew_cb
                        last edited by andrew_cb

                        I found what is adding these directives to the config file!

                        The behavior is hard-coded in /usr/local/pkg/haproxy/haproxy.inc which is what generates /var/etc/haproxy/haproxy.cfg

                        Line 727
                        function write_backend($configpath, $fd, $name, $pool, $backendsettings) {
                                            ...
                            Line 1006  (always executes, not subject to any conditional statements)
                            fwrite ($fd, "\tload-server-state-from-file\tglobal\n");
                        
                        Line 1520
                        function haproxy_writeconf($configpath) {
                                            ...
                            Line 1612 (always executes, not subject to any conditional statements)
                            fwrite ($fd, "\tserver-state-file /tmp/haproxy_server_state\n");
                        
                        Line 2498
                        function haproxy_check_run($reload) {
                                            ...
                            Lines 2507-2508 (executes if haproxy is running when a reload is requested, which is almost always true)
                            if ($reload) {
                                if (haproxy_is_running()) {
                        	    $r = haproxy_socket_command("show servers state");
                                    file_put_contents("/tmp/haproxy_server_state", $r);
                        		}
                        

                        These lines result in the following behavior:

                        • Whenever HAProxy's settings are saved using the GUI, the server-state-file directive is added to the global section of the resulting config file.
                        • Whenever HAProxy's settings are saved using the GUI, the load-server-state-from-file directive is added to each backend of the resulting config file.
                        • Whenever HAProxy reloads, it writes the backend server state to /tmp/haproxy_server_state, and due to the server-state-file directive, it then always reads and applies the previous state data.

                        There is no GUI text that mentions that this behavior is hard-coded, and there is no way to disabled it other than commenting out the lines (1006, 1612, 2507, 2508) in haproxy.inc that are responsible for adding the directives to the config file.

                        The GUI option Reload behaviour is responsible for the directive hard_stop_after which is for a different behavior.

                        I've added these findings to the redmine.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.