SOLVED (user error) Confusing HAproxy
-
First, I have two sets of load balanced servers using HAproxy.
The first set, call them A, work just fine.
The second set, B, does not work.I've stared at the configurations for quite some time, went over everything, it looks fine and is pretty much identical to the first set that is working.
To test, I set up a rule to the HAproxy service and one to the servers directly. Each server is responding when traffic comes in.
I then disable direct and enable the rule to HAproxy and all traffic to both servers stop.
Looking at the HAproxy status in dashboard shows it sees both backend servers.
I've even tried putting one into maint mode but still no traffic flowed to either one.
This setup was working perfectly with the two sets of balanced servers for a long time but something, presumably small changed, causing this. The only change that has been made is the backend servers were changed from centos to rocky Linux with new IPs which were updated in the proxy.
That's confusing and I hope it doesn't mean I have to reboot the firewall due to some state problem.
-
Any existing states would have timed out by now.
Do you see the connections coming into HAProxy? Do you see any error response at the client?
Steve
-
Hi,
I've not dared to keep it on long as there is so much data coming in so have not checked those things yet.
I'll set up a test and do that shortly.
-
Looking at the haproxy stats page, all looks fine, both sets of servers look the same in terms of configuration and haproxy sees all four servers.
A rule bypassing the proxy to either of the B back end servers work fine. Both servers respond.
A rule that enables the proxy leads to the following.
From a client, using curl, I get;
- Failed to connect to www.bbb.com port 443: Host is unreachable
In the states, I see the following with the proxy enabled.
-
The brackets on those states make it look like they are NAT'd. The port forwards still active?
I expect those states to be to HAProxy directly.
All those states show two way traffic though so seeing 'host unreachable' looks like it may not be any of those.
Steve
-
Thank you for pointing that out. I always have too many things going on at once and failed to notice I had not updated the server IP's in the aliases.
I was starring at haproxy way too much and missing the most obvious.Thanks again.
-
No problem. Does it work as expected now without the forwards active?
-
Yes, it's all back to normal now. The backend servers were upgraded to a different OS and their IPs were different so they would not conflict with the live servers.
I thought I looked at the aliases so never thought about it again and figured something was up with the proxy until you commented which caused me to double check.
All good now.
