HAProxy has a directives load-server-state-from-file that is described as:
This directive points HAProxy to a file where server state from previous
running process has been saved. That way, when starting up, before handling
traffic, the new process can apply old states to servers exactly has if no
reload occurred.
and also the directive server-state-file which is described as:
Before reloading HAProxy, it is possible to save the
servers' current state using the stats command "show servers state". The
output of this command must be written in the file pointed by <file>. When
starting up, before handling traffic, HAProxy will read, load and apply state
for each server found in the file and available in its current running
configuration.
https://docs.haproxy.org/2.9/configuration.html#load-server-state-from-file
https://docs.haproxy.org/2.9/configuration.html#server-state-file
Looking in my /var/etc/haproxy/haproxy.cfg shows the global section contains the directive
server-state-file /tmp/haproxy_server_state
and each backend contains the directive
load-server-state-from-file global
I think this explains why we are seeing the issue - when HAproxy reloads, it writes the current server state to the file and then loads the states back from the file. So even though the configuration has been changed, HAproxy is still using the server states that existed before making the changes.
This also explains why deleting a backend, saving, and then re-creating the backend is a workaround - when a backend server is deleted, all the HAproxy states for it will be closed and thus do not get saved to the file.
They mystery now is why those directives are being added to the HAproxy configuration. I do not see an GUI option for either directive.
